DEC 06 2018 CHRISTIE KOEHLER

In this blog post, we’ll look at practical public key certificate management in Vault, which uses a dynamic secrets approach.

HashiCorp Vault provides secrets management and protection of sensitive data. It provides a central place to secure, store, and control access to tokens, passwords, certificates, and encryption keys. There are challenges of centralized secrets management, particularly related to applications. Applications don’t keep secrets and secrets are often shared among different applications.

Vault presents an answer to these problems in the form of dynamic secrets. A dynamic secret is generated on demand and is unique to a client. This is opposed to a static secret, which is defined ahead of time and is often shared by different clients. Vault associates each dynamic secret with a lease and automatically destroys the credentials when the lease expires. Static secrets, by contrast, often have much longer lifecycles.

Dynamic secrets allows us to manage intentions (e.g. web server needs database access) instead of a managing credentials (e.g. authentication data provided to web servers requiring database access). This allows us to achieve the same end goal while solving major challenges, including leaky applications, non-repudiation, automatic rotation, and practical revocation. See Why We Need Dynamic Secrets for more details.

Vault applies a dynamic secret approach to public key certificates as well, acting as a signing intermediary to generate short lived certificates. This allows certificates to be generated on-demand, as needed, and rotated automatically.

In this post we’ll look at the most common operator activities involved in certificate management using Vault. We’ll cover:

• Enabling and Configuring PKI engine(s).
• Creating roles and generating certificates.
• Revoking certificates and updating Certificate Revocation Lists (CRLs).
• Integrating with applications.

Enabling and configuring the PKI engine(s)

Vault supports many secrets engines. To use a secrets engine, you enable it at a given path or mount point. After installing and configuring Vault, you’ll need to enable and configure the PKI secrets engine.

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

The default is for engines to be enabled at a path named for their type (e.g. pki for the PKI engine, as seen above). With few exceptions, secrets engines can be enabled at multiple paths by enabling each with a unique name. Enabled secrets engines cannot see each other’s data, even if they are of the same type.

To specify a mount point, use the -path argument when enabling the engine.

For example, to create a second PKI secrets engine to act as your intermediate signing authority, you would do the following, after creating the first engine pki above:

$ vault secrets enable -path=pki-inter pki
Success! Enabled the pki secrets engine at: pki-inter/

To issue commands to this instance of the secrets engine, you’ll need to use the pki-inter prefix rather than pki. For example: vault write pki-inter/config/ca instead of vault write pki/config/ca.

Adjust Global TTL

Often it also makes sense to adjust the global maximum time-to-live (TTL) of tokens and leases for this secrets engine. The default is 30 days which might be too short. Individual roles can restrict this value to be shorter on a per-certificate basis.

Here’s how to adjust the TTL to one year:

$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

You’ll need to do this for each PKI engine you’ve enabled.

Configure Root and/or Intermediate CAs

Each PKI secrets engine must be configured with a CA certificate and associated private key.

There are three methods for accomplishing this:

• generate a self-signed root CA
• generate an intermediate CA (with a Certificate Signing Request, CSR, for signing)
• set a PEM-encoded certificate and private key bundle directly into the backend

You’ll also need to configure a root CA. You can have Vault generate a self-signed root CA or provide the details for your root CA. There are separate end points for each.

To generate a self-signed root CA, use the pki/root/generate endpoint:

$ vault write pki/root/generate/internal \
    common_name=my-website.com \
    ttl=8760h

To generate an intermediate CA, use the /pki/intermediate/generate/ endpoint:

$ vault write pki/intermediate/generate/internal common_name=example.com data=@pem_bundle.json

To set a PEM-encoded certificate and private key bundle, use the pki/config/ca endpoint:

$ vault write pki/config/ca pem_bundle=@pem_bundle.json

Only one CA certificate is allowed per secrets engine. If you want to issue certificates from multiple CAs, mount the PKI secrets engine at multiple mount points with separate CA certificates in each.

A common and recommended pattern is to have one mount act as your root CA and to use this CA only to sign intermediate CA CSRs from other PKI secrets engines. See Build Your Own Certificate Authority (CA) on HashiCorp Learn or our webinar Streamline Certificate Management for further details.

Configure URL values for issue certificate endpoints

For each PKI engine that you enable you’ll need to configure the url values for issuing certificate endpoints and CRL distribution points that will be encoded into issued certificates:

$ vault write pki/config/urls \
    issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
    crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"
Success! Data written to: pki/config/urls

Creating roles and generating certificates

Generating certificates requires you to supply a role. The role definition sets the conditions under which a certificate can be generated.

Use the /pki/roles/:name endpoint to create and update roles:

$ vault write pki/roles/example-dot-com \
    allowed_domains=example.com \
    allow_subdomains=true max_ttl=72h
Success! Data written to: pki/roles/example-dot-com

The roles endpoint accepts a number of parameters and nearly any issuing policy can be accommodated. If a client requests a certificate that is not allowed by the role, the request is denied.

Once a role has been created, you can use it to generate certificates with the pki/issue endpoint:

$ vault write pki/issue/example-dot-com \
    common_name=my.example.com
Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----

A certificate can be renewed at any time by providing issue with the same common name as an existing certificate. The original certificate will continue to be valid through its original time-to-live unless explicitly revoked.

Revoking certificates and rotating CRLs

If a certificate must be revoked, you can easily perform the revocation action which will cause the CRL (Certificate Revocation List) to be regenerated. When the CRL is regenerated, any expired certificates are removed from the CRL.

$ vault write pki/revoke serial_number=<serial_number>

Operators can also trigger rotation of the CRLs directly with the pki/crl/rotate endpoint. Because this end point doesn’t ingest any data, you’ll need to use the vault read command instead of write:

$ vault read pki/crl/rotate
Key        Value
---        -----
success    true

As with all Vault commands, you can also interact with the API directly, as such:

$ curl \
    --header "X-Vault-Token: ..." \
    http://127.0.0.1:8200/v1/pki/crl/rotate

There’s also the Vault UI, available in version 0.10+, as well as Vault Enterprise.

Using the tidy endpoint, operators can optimize the storage backend and CRL by periodically removing certificates that have expired and are past a certain buffer period beyond their expiration time.

$ vault write pki/tidy tidy_cert_store=true tidy_revoked_certs=true

Integrating with applications

Already you can see how generating PKI certificates with Vault saves operators time. A single call to the Vault API replaces the tedious process of generating a private key, generating a CSR, submitting to a CA, and then waiting for a verification and signing process to complete.

To automate the process further, use a template rendering tool such as Consul Template.

Consul Template is a daemon that queries a Consul or Vault cluster and updates any number of specified templates on the file system. Rendering templates requires both a template file and a template configuration. Template files are written in the Go Template format and the configuration files are in HCL. (See Consul Template’s README.md for further documentation.)

When generating PKI certificates with Vault, the certificate, private key, and any intermediate certs are all returned as part of the same API call. Most applications require that this data be placed in separate files on the system.

Here are the templates we can use to retrieve the necessary certificate files and save them locally:

{{- /* /tmp/cert.tpl */ -}}
{{ with secret "pki/issue/example-dot-com" "common_name=my.example.com" }}
{{ .Data.certificate }}{{ end }}

{{- /* /tmp/ca.tpl */ -}}
{{ with secret "pki/issue/example-dot-com" "common_name=my.example.com" }}
{{ .Data.issuing_ca }}{{ end }}

{{- /* /tmp/key.tpl */ -}}
{{ with secret "pki/issue/example-dot-com" "common_name=my.example.com" }}
{{ .Data.private_key }}{{ end }}

The with secret directive queries Vault at the supplied API endpoint (“pki/issue/example-dot-com”) with the given parameters (“common_name=my.example.com”). This is equivalent to the Vault command we showed earlier for generating certificates: vault write pki/issue/example-dot-com.

Consul Template writes the data returned by Vault into .Data.certificate, .Data.issuing_ca, and .Data.private_key, which directly correspond to the data returned by the Vault API:

{
...
  "data": {
    "certificate": "-----BEGIN CERTIFICATE-----",
    "issuing_ca": "-----BEGIN CERTIFICATE-----",
    "private_key": "-----BEGIN RSA PRIVATE KEY-----",
    ...
    },
}

In this example, there are three different input templates. But, when run under the same Consul Template process, they are compressed into a single API call, sharing the resulting data.

Here is an example the corresponding Consul Template configuration:

template {
  source      = "/tmp/cert.tpl"
  destination = "/opt/my-app/ssl/my-app.crt"
}

template {
  source      = "/tmp/ca.tpl"
  destination = "/opt/my-app/ssl/ca.crt"
}

template {
  source      = "/tmp/key.tpl"
  destination = "/opt/my-app/ssl/my-app.key"
}

Use multiple template blocks to define multiple templates. The source directive indicates which source file on disk to use as the input template. The destination directive indicates the path on disk where the source template will render.

Note: Please always consider the security implications of having the contents of a secret in plain-text on disk. If an attacker is able to get access to the file, they will have access to plain-text secrets.

Consul Template will fetch a new secret at half the lease duration of the original secret. For example, the role we created above has a max_ttl=72h. Because we did not specify a ttl when we created our certificate with vault write pki/issue/example-dot-com, it will have a lease of 72 hours. This means Consul Template will renew the secret, which in the case of certificates means generating a new one, every 36 hours. However, because Vault does not support blocking queries, Consul Template will not immediately reload in the event a secret is changed as it does with Consul's key-value store.

For more examples of using Consul template, see the examples directory in the project.

Next Steps

To learn more, see:

• Secrets as a Service: Dynamic Secrets
• Build Your Own Certificate Authority (CA)
• Getting Started: Dynamic Secrets
• PKI Secrets Engine
• Streamline Certificate Management with HashiCorp Vault (video)