Kubernetes External Secrets

Teams at GoDaddy use the AWS managed Kubernetes offering, EKS, to deploy their services. We also use AWS Secrets Manager for storing secrets, like private keys and database passwords. EKS, however, does not provide much support for accessing Secrets Manager. Therefore, teams develop custom solutions for accessing secret data from their EKS clusters. One approach, for example, is to fetch secrets from Secret Manager when the application starts. The result is overlapping efforts for developing these custom solutions and because security is critical in this context, the individual efforts can require substantial engineering time.

This blog post describes a generalized approach and implementation for supporting secret management systems, like AWS Secrets Manager, in Kubernetes. We call this system Kubernetes External Secrets and we have open sourced our initial implementation. Our current solution focuses on making it easy for developers to manage secrets with AWS and deploy them to their EKS clusters, but we think our approach is general enough for other types of Kubernetes clusters and secret management systems.

The rest of this blog post describes the motivation for and design of Kubernetes External Secrets. The README.md has instructions for adding Kubernetes External Secrets to your cluster if you are looking to get started immediately.

Overview

Kubernetes has a built-in object for managing secrets called a Secret. The Secret object is convenient to use: it provides a declarative API that makes it easy for application Pods to access secret data without any special code. One downside of Secret objects is that they do not support storing or retrieving secret data from external secret management systems. It's often beneficial, however, to use Kubernetes with an external service that handles secret management and includes useful features. For example, Secrets Manager integrates with other AWS services, like Lambda functions, includes encryption at rest, and has a useful mechanism for codifying rotation policies.

Kubernetes External Secrets aims to provide the same ease of use as native Secret objects and provide access to secrets stored externally. It does this by adding an ExternalSecret object to the Kubernetes API that allows developers to inject external secrets into a Pod using a declarative API similar to the native Secret one.

Instead of inlining base64 encoded secret data into a Secret object, developers define an ExternalSecret object that specifies a secret management system and properties to load from that system.

For example, instead of using a Secret object to store a database password:

apiVersion: v1
kind: Secret
metadata:
  name: cats-and-dogs
type: Opaque
data:
  password: d29vZgo=

a developer can use an ExternalSecret, specifying the secret management systems as backendType and the properties to access in the data array:

apiVersion: 'kubernetes-client.io/v1'
kind: ExtrenalSecret
metadata:
  name: cats-and-dogs
secretDescriptor:
  backendType: secretsManager
  data:
    - key: cats-and-dogs/mysql-password
      name: password

and access that password from a Pod in the same way they would if they were using a Secret object:

apiVersion: v1
kind: Pod
metadata:
  name: cats-and-dogs
spec:
  containers:
  - name: cats-and-dogs
    image: cats-and-dogs
    env:
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: cats-and-dogs
            key: password

Notice that the ExternalSecret does not contain secret data. It's safe to store in plain text along with your other Kubernetes manifest files for your service.

Design

Kubernetes External Secrets adds the ExternalSecret object to Kubernetes using a CustomResourceDefinition, and adds an ExternalSecret controller we wrote in Node.js that implements the behavior of the object type itself.

The ExternalSecret controller follows a familiar pattern seen in other Kubernetes objects, like Deployments: users declare a desired state in an ExternalSecret object and the controller creates or updates a complementary Secret object to reach that state. The controller monitors ExternalSecret objects, fetches secret data from the specified external secret management system, and automatically creates native Secret objects that hold the secret data. The architecture diagram below illustrates this process.

Using

You can add Kubernetes External Secrets to your cluster with a single command:

kubectl apply -f https://raw.githubusercontent.com/godaddy/kubernetes-external-secrets/master/external-secrets.yml

and begin manipulating ExternalSecret objects like you would any other Kubernetes object type:

kubectl -n cats-and-dogs get es

If you're interested in converting existing Secret objects to ExternalSecret objects, one of the authors of Kubernetes External Secrets wrote a Kubernetes External Secrets CLI that makes migration easy by converting a Secret to a series of AWS CLI commands and a kubectl command that loads your secret data into AWS Secrets Manager and creates a complementary ExternalSecret object.

Other approaches

We experimented with and drew inspiration from other projects that help manage access to secret data from Kubernetes.

cmattoon/aws-ssm uses annotations on Secret objects to identify properties in AWS Parameter Store and populate the Secret object with those properties' values.

kubesec makes it easy to encrypt data before storing it in a Secret object or manifest. At run-time, applications are responsible for decrypting the data before using it.

The Kubernetes Vault Integration project provides Vault auth tokens to Pod objects. Application code running in a Pod can use that token to authenticate with Vault and retrieve secret data.

We explored configuring nodes in our CICD pipeline to inject Secret objects during testing and deployment. One benefit of this approach is that Kubernetes clusters would not need to access the external secret management system directly, which might help mitigate data leaks during an attack. On the other hand, it required the Kubernetes manifest files for a single application to be stored in several locations and complicated application debugging and deployment because developers would need to use a combination of kubectl and specialized CICD tools.

Upcoming improvements

Kubernetes External Secrets supports AWS Secrets Manager and AWS Systems Manager, however, we believe the approach is general enough to support other external secret management systems and would be excited about working with community members to add them.

The Kubernetes External Secrets controller does not have a broad attack surface since it is accessible only via the Kubernetes API. We have worked, however, to tighten security by leveraging RBAC to restrict what the controller can access, using eslint-plugin-security to identify potentially vulnerable code patterns, and doing design reviews at GoDaddy. Nevertheless, we think it is important to continually improve security and take advantage of new features and patterns as they emerge. We plan on continuing to improve security by:

• restricting network access to staunch potential attacks that might try to leak secrets;
• facilitating rotation by triggering restarts when there is an update to an ExternalSecret;
• working through a threat model analysis; and
• adopting upcoming EKS features, like IAM roles for Pods, that will help harden our implementation.

Conclusion

We hope that Kubernetes External Secrets can provide teams outside of GoDaddy with a common way to access secret data in secret management systems. Using Kubernetes External Secrets does not require any application modifications and existing Pod specifications that leverage Secret objects will work with ExternalSecret objects without any changes.

If you want to get started with Kubernetes External Secrets or contribute please visit https://github.com/godaddy/kubernetes-external-secrets.

Acknowledgements

Celia Waggoner and Jarrett Cruger provided feedback on the Kubernetes External Secrets design and contributed to early versions of the implementation.