𝗧𝗼𝗽𝗶𝗰 — Security 🔐
𝗦𝗲𝗿𝗶𝗲𝘀 — “Under the hood” ⛏
The Envelope encryption is a vital defense-in-depth security feature that has addressed a crucial need — the protection of secrets that would otherwise be stored in plain text format within the
etcd database. We can use envelope encryption to encrypt the Kubernetes secrets in EKS with our own master key.
Encoding & Encrypting
Encoding (without keys):
Encoding is the process of transforming data into a specific format, often for efficient storage or transmission, without the use of encryption keys.
Encryption (with keys):
Encryption is the method of converting data into a secure and unintelligible form using algorithms and encryption keys, primarily to protect it from unauthorized access
The Core Architecture
Certainly, here's a step-by-step explanation of envelope encrypting secrets in EKS:
User Initiates Secret Creation: The user runs
$kubectl create secret, calling the Kubernetes API server.
Kube API Server Authorization & Validation: The Kubernetes API server checks user authorization and validates the request.
Generating DEK in-memory: The Kubernetes API server generates a Data Encryption Key (DEK) locally in memory, unique for this operation, never saved to disk.
Caching for Performance: The Kubernetes API server caches the DEK in memory for efficiency and to reduce the API hits to AWS KMS.
Secret Encryption: Kube API server encrypts the secret using DEK in-memory.
Kube API Server with KMS: Meanwhile the Kubernetes API server sends the DEK to KMS for further encryption.
KMS & CMK Encryption: KMS employs the Customer Master Key (CMK) to encrypt the DEK securely using a master key.
Returns Encrypted DEK: KMS returns encrypted DEK to the Kubernetes API server.
Envelope Encryption: The envelope contains encrypted DEK with KMS CMK and encrypted secret with DEK.
Envelope Stored in etcd: The envelope will now be stored in
Secret Retrieval from etcd: The pod requests a secret from the Kubernetes API server.
Now two ways can be followed up by Kube API server:
Using Cached DEK:
If the Kubernetes API server has previously cached the encrypted DEK in its memory, it will use this cached DEK to perform decryption.
Cached data is used to optimize performance and reduce the API hits to AWS KMS when the same secret is accessed multiple times.
Retrieving from etcd:
If there is no cached DEK data available or if the cached data has expired, the Kubernetes API server again calls the KMS.
This typically happens when the secret is accessed for the first time after a cache miss or expiration.
In a nutshell, we have ensured the data at rest is encrypted in etcd.
Thanks for reading our blog. Feel free to hit me up for any AWS/DevOps/Open Source-related discussions.
Manoj Kumar — LinkedIn.
Poonam Pawar — LinkedIn