Teams at GoDaddy use the AWS managed Kubernetes offering, EKS, to deploy their services. We also use AWS Secrets Manager for storing secrets, like private keys and database passwords. EKS, however, does not provide much support for accessing Secrets Manager. Therefore, teams develop custom solutions for accessing secret data from their EKS clusters. One approach, for example, is to fetch secrets from Secret Manager when the application starts. The result is overlapping efforts for developing these custom solutions and because security is critical in this context, the individual efforts can require substantial engineering time.
This blog post describes a generalized approach and implementation for supporting secret management systems, like AWS Secrets Manager, in Kubernetes. We call this system Kubernetes External Secrets and we have open sourced our initial implementation. Our current solution focuses on making it easy for developers to manage secrets with AWS and deploy them to their EKS clusters, but we think our approach is general enough for other types of Kubernetes clusters and secret management systems.
The rest of this blog post describes the motivation for and design of Kubernetes External Secrets. The README.md has instructions for adding Kubernetes External Secrets to your cluster if you are looking to get started immediately.
Kubernetes has a built-in object for managing secrets called a
Secret object is convenient to use: it provides a declarative API that makes it easy for application
Pods to access secret data without any special code. One downside of
Secret objects is that they do not support storing or retrieving secret data from external secret management systems. It's often beneficial, however, to use Kubernetes with an external service that handles secret management and includes useful features. For example, Secrets Manager integrates with other AWS services, like Lambda functions, includes encryption at rest, and has a useful mechanism for codifying rotation policies.
Kubernetes External Secrets aims to provide the same ease of use as native
Secret objects and provide access to secrets stored externally. It does this by adding an
ExternalSecret object to the Kubernetes API that allows developers to inject external secrets into a
Pod using a declarative API similar to the native
Instead of inlining base64 encoded secret data into a
Secret object, developers define an
ExternalSecret object that specifies a secret management system and properties to load from that system.
For example, instead of using a
Secret object to store a database password:
a developer can use an
ExternalSecret, specifying the secret management systems as
backendType and the properties to access in the
- key: cats-and-dogs/mysql-password
and access that password from a
Pod in the same way they would if they were using a
- name: cats-and-dogs
- name: SECRET_PASSWORD
Notice that the
ExternalSecret does not contain secret data. It's safe to store in plain text along with your other Kubernetes manifest files for your service.
Kubernetes External Secrets adds the
ExternalSecret object to Kubernetes using a
CustomResourceDefinition, and adds an
ExternalSecret controller we wrote in Node.js that implements the behavior of the object type itself.
ExternalSecret controller follows a familiar pattern seen in other Kubernetes objects, like
Deployments: users declare a desired state in an
ExternalSecret object and the controller creates or updates a complementary
Secret object to reach that state. The controller monitors
ExternalSecret objects, fetches secret data from the specified external secret management system, and automatically creates native
Secret objects that hold the secret data. The architecture diagram below illustrates this process.
You can add Kubernetes External Secrets to your cluster with a single command:
kubectl apply -f https://raw.githubusercontent.com/godaddy/kubernetes-external-secrets/master/external-secrets.yml
and begin manipulating
ExternalSecret objects like you would any other Kubernetes object type:
kubectl -n cats-and-dogs get es
If you're interested in converting existing
Secret objects to
ExternalSecret objects, one of the authors of Kubernetes External Secrets wrote a Kubernetes External Secrets CLI that makes migration easy by converting a
Secret to a series of AWS CLI commands and a kubectl command that loads your secret data into AWS Secrets Manager and creates a complementary
We experimented with and drew inspiration from other projects that help manage access to secret data from Kubernetes.
cmattoon/aws-ssm uses annotations on
Secret objects to identify properties in AWS Parameter Store and populate the
Secret object with those properties' values.
kubesec makes it easy to encrypt data before storing it in a
Secret object or manifest. At run-time, applications are responsible for decrypting the data before using it.
The Kubernetes Vault Integration project provides Vault auth tokens to
Pod objects. Application code running in a
Pod can use that token to authenticate with Vault and retrieve secret data.
We explored configuring nodes in our CICD pipeline to inject
Secret objects during testing and deployment. One benefit of this approach is that Kubernetes clusters would not need to access the external secret management system directly, which might help mitigate data leaks during an attack. On the other hand, it required the Kubernetes manifest files for a single application to be stored in several locations and complicated application debugging and deployment because developers would need to use a combination of kubectl and specialized CICD tools.
Kubernetes External Secrets supports AWS Secrets Manager and AWS Systems Manager, however, we believe the approach is general enough to support other external secret management systems and would be excited about working with community members to add them.
The Kubernetes External Secrets controller does not have a broad attack surface since it is accessible only via the Kubernetes API. We have worked, however, to tighten security by leveraging RBAC to restrict what the controller can access, using eslint-plugin-security to identify potentially vulnerable code patterns, and doing design reviews at GoDaddy. Nevertheless, we think it is important to continually improve security and take advantage of new features and patterns as they emerge. We plan on continuing to improve security by:
- restricting network access to staunch potential attacks that might try to leak secrets;
- facilitating rotation by triggering restarts when there is an update to an
- working through a threat model analysis; and
- adopting upcoming EKS features, like IAM roles for
Pods, that will help harden our implementation.
We hope that Kubernetes External Secrets can provide teams outside of GoDaddy with a common way to access secret data in secret management systems. Using Kubernetes External Secrets does not require any application modifications and existing
Pod specifications that leverage
Secret objects will work with
ExternalSecret objects without any changes.
If you want to get started with Kubernetes External Secrets or contribute please visit https://github.com/godaddy/kubernetes-external-secrets.