Secure Your Cluster: Centralizing Kubernetes Secrets with ESO & AWS

In the world of cloud-native development, the "secret" to security is often managed poorly. We have all seen it happen: a database password committed to a Git repository, an API key hardcoded into a Dockerfile, or a `.env` file shared insecurely over Slack. As organizations migrate to Kubernetes and adopt GitOps workflows, the paradox of secret management becomes glaringly obvious: we want to define our infrastructure as code, but we cannot commit our actual secrets to code.

For CTOs and Tech Leads, this isn't just a workflow annoyance; it is a critical security vulnerability and a compliance nightmare. Native Kubernetes Secrets, while useful, are merely base64-encoded strings stored in etcd—hardly a vault for your most sensitive data.

At Nohatek, we believe in robust, scalable DevSecOps practices. Today, we are diving deep into the industry-standard solution: centralizing secret management by bridging AWS Secrets Manager with your Kubernetes clusters using the External Secrets Operator (ESO). This approach eliminates hardcoded credentials, automates rotation, and keeps your developers happy.

The Problem: Why Native Kubernetes Secrets Aren't Enough

To understand the solution, we must first acknowledge the limitations of the default Kubernetes behavior. By default, Kubernetes Secrets are stored in the cluster's etcd database. While they can be encrypted at rest, the management lifecycle is often manual and disconnected from the source of truth.

Consider the following challenges:

• Base64 is not Encryption: If a developer has RBAC permissions to view secrets in a namespace, they can easily decode the base64 string.
• Secret Sprawl: When secrets are scattered across different clusters and namespaces without a central repository, rotating a compromised credential becomes a manual, error-prone race against time.
• The GitOps Conflict: If you use ArgoCD or Flux, you want your cluster state to match your Git repository. However, you cannot check secrets into Git. Tools like Sealed Secrets or SOPS offer workarounds, but they still require managing encrypted files within the repo, which can become cumbersome at scale.

The goal is to decouple the storage of the secret from the consumption of the secret.

This is where the External Secrets Operator (ESO) shines. It allows you to keep your secrets in a dedicated vault (like AWS Secrets Manager) and automatically injects them into Kubernetes as native secrets only when needed.

The Architecture: How ESO Bridges the Gap

The External Secrets Operator is a Kubernetes controller that integrates with various secret management systems, including AWS Secrets Manager, HashiCorp Vault, and Azure Key Vault. It operates on a "pull" model.

Here is the high-level workflow:

1. Source of Truth: You create a secret (e.g., a database credential) in AWS Secrets Manager. This is secure, audited, and can be set to rotate automatically.
2. The Operator: ESO runs inside your Kubernetes cluster. It is configured with an IAM role (using IRSA - IAM Roles for Service Accounts) that grants it permission to read specific secrets from AWS.
3. The CRDs: You define a SecretStore (how to connect to AWS) and an ExternalSecret (which secret to fetch).
4. Synchronization: ESO fetches the data from AWS and creates a standard Kubernetes Secret object.
5. Consumption: Your application pods mount the native Kubernetes Secret as an environment variable or volume, exactly as they always have.

The beauty of this architecture is that your application code does not change. Your developers don't need to install AWS SDKs or write code to fetch secrets at runtime. The platform handles the complexity transparently.

Implementation Guide: Setting Up ESO on EKS

Let's look at the practical steps to implement this. We will assume you are running Amazon EKS and have Helm installed.

Step 1: IAM Permissions (IRSA)
First, you need an IAM policy that allows reading secrets. Create a policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": ["arn:aws:secretsmanager:region:account-id:secret:my-secret-*"]
    }
  ]
}

Attach this policy to an IAM Role associated with the ESO ServiceAccount in your cluster.

Step 2: Install External Secrets Operator
Use Helm to install the operator into your cluster:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Step 3: Define the SecretStore
The SecretStore tells ESO how to authenticate with AWS. Since we are using EKS, we use the ServiceAccount authentication:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: backend
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: my-service-account

Step 4: Create the ExternalSecret
Finally, map the AWS secret to a Kubernetes secret. This tells the operator to go fetch prod/db/password from AWS and create a K8s secret named db-creds.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: backend
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-creds # The name of the K8s secret to be created
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/db/password
      property: password

Once applied, running kubectl get secrets -n backend will show your new db-creds secret, ready for use.

Strategic Benefits for the Enterprise

Implementing this architecture provides immediate value beyond just "hiding passwords." It aligns your infrastructure with enterprise-grade security standards.

• Compliance & Auditing: By using AWS Secrets Manager, every access to a secret is logged in AWS CloudTrail. You know exactly when your cluster requested a credential. This is vital for SOC2 and HIPAA compliance.
• Automated Rotation: AWS can automatically rotate database credentials. Because ESO polls for changes (based on the refreshInterval), your Kubernetes secrets are automatically updated without human intervention. You can even trigger a pod restart to pick up the new values.
• Multi-Cloud & Hybrid: While we focused on AWS, ESO is cloud-agnostic. You can use the same operator to pull secrets from Google Secret Manager or Azure Key Vault, providing a unified interface for hybrid cloud strategies.

By removing secrets from your Git repositories and CI/CD pipelines, you significantly reduce the attack surface of your software supply chain.

Hardcoded secrets are a ticking time bomb in modern software development. By centralizing your secret management with AWS Secrets Manager and bridging it to Kubernetes via the External Secrets Operator, you achieve the holy grail of DevSecOps: high security with low developer friction.

At Nohatek, we specialize in building secure, scalable cloud infrastructures. Whether you are looking to harden your existing EKS clusters, migrate to a microservices architecture, or implement a robust GitOps workflow, our team is ready to help you navigate the complexities of modern cloud engineering.

Ready to secure your infrastructure? Contact Nohatek today for a consultation on your cloud security strategy.