Source: Exploring container security: Encrypting Kubernetes secrets with Cloud KMS from Google Cloud
Editor’s note: This post picks up again on our blog post series on container security at Google.
At Google Cloud, we care deeply about protecting your data. That’s why we encrypt data at rest by default, including data in Google Kubernetes Engine (GKE). For Kubernetes secrets—small bits of data your application needs at build or runtime—your threat model might be different, so storage-layer encryption is insufficient. Today, we’re excited to announce in beta GKE application-layer secrets encryption, using the same keys you manage in our hosted Cloud Key Management Service (KMS).
In a default Kubernetes installation, Kubernetes secrets are stored in etcd in plaintext. In GKE, this is managed for you: GKE encrypts these secrets on disk, and monitors this data for insider access. But this might not be enough to protect those secrets from a potentially malicious insider, or a malicious application in your environment.
Kubernetes 1.7 first introduced application-layer encryption of secrets for differential protection of secrets. Using this feature, you can encrypt your secrets locally, with a key also stored in etcd. If you’re running Kubernetes in an environment that doesn’t provide encryption by default, this helps meet security best practices; however, if a malicious intruder gained access to etcd, they would still practically have access to your secrets.
A few releases later in Kubernetes 1.10, envelope encryption of secrets with a KMS provider was introduced, meaning that a local key is used to encrypt the secrets (known as a “data encryption key”), and that the key is itself encrypted with another key not stored in Kubernetes (a “key encryption key”). This model means that you can regularly rotate the key encryption key without having to re-encrypt all the secrets. Furthermore, it means that you can rely on an external root of trust for your secrets in Kubernetes—systems ike Cloud KMS or HashiCorp Vault.
Application-layer secrets encryption is now in beta in GKE, so you can protect secrets with envelope encryption: your secrets are encrypted locally in AES-CBC mode with a local data encryption key, and the data encryption key is encrypted with a key encryption key you manage in Cloud KMS as the root of trust. It’s pretty simple as all the hard work is done for you—all you have to do is choose the key you want to use for a particular cluster.
This approach provides flexibility in your security model, to meet specific requirements you may have:
To enable application-layer secrets encryption for a new cluster, specify the
--database-encryption-key flag as part of cluster creation, with your Cloud KMS key
Note that you must use a Cloud KMS key in the same location as your cluster, and that you need to give a GKE service account account to use your key for these operations.
That’s how easy it is to encrypt your Kubernetes secrets with GKE. For more detail, check out Application-layer Secrets Encryption, or this Cloud KMS hands-on lab1. We also gave a talk on Kubernetes secrets at KubeCon China that provides more color. Now encrypt those secrets!
1. Take this lab for free before February 28, 2019 with the code 1j-shh-441)