External Secrets Operator with Python Apps

Running secrets on Kubernetes with a manager as the source of truth gets you two things that hand-managed Secrets cannot: rotation happens in one place, and the manifest in your repository names a secret rather than containing one. An external secrets operator provides both by reconciling a custom resource into an ordinary Kubernetes Secret on a schedule. What it does not change is how your Python process consumes the result — which is exactly why it is worth adopting, and also why the failure modes are subtle. This page covers both, extending the Kubernetes secrets topic.

Problem 1: values that live in the repository

# ANTI-PATTERN: a real credential, in version control, forever
apiVersion: v1
kind: Secret
metadata: { name: api-credentials }
stringData:
  DATABASE_PASSWORD: hunter2

Once a credential is committed it is in the history of every clone, every fork, every CI cache and every developer’s laptop, and removing it requires rewriting history and rotating the value anyway. Sealed or encrypted manifests improve this — the committed form is not readable — but they keep the distribution problem: the decryption key must reach the environment somehow, and rotation means re-encrypting and re-committing.

An operator inverts the flow. The repository holds a resource that says “fetch the key prod/api/database-password from this store and put it in a Secret named api-credentials under the key database_url”. The value never enters version control, rotation happens in the manager without touching the repository, and the manifest is safe for anyone to read.

Problem 2: hand-editing a reconciled Secret

# ANTI-PATTERN: works for one refresh interval, then vanishes
kubectl edit secret api-credentials          # reverted at the next reconciliation

During an incident somebody changes a value directly, verifies the fix, and moves on. An hour later the operator reconciles, overwrites the object with the manager’s version, and the problem returns with no record of what happened — the change existed only in an object that has since been replaced.

The rule is that a reconciled Secret is read-only. Emergency changes go into the manager, where they persist and are audited. If a genuine break-glass path is needed, implement it explicitly by pausing reconciliation for that resource rather than by racing the controller, and make the pause visible so nobody forgets it is in force. A paused sync that outlives the incident is its own failure: the object stops tracking the manager and nobody notices until a rotation appears not to work.

Problem 3: alerting on pods instead of on sync status

# ANTI-PATTERN: the app is healthy, so nothing tells you the sync has been failing for a day
@app.get("/healthz")
def healthz():
    return {"status": "ok"}         # says nothing about secret freshness

When the manager becomes unreachable, reconciliation fails and the existing Secret keeps its last synced values. Running pods are completely unaffected, which is the correct behaviour and also why the failure is silent. The Secret quietly goes stale, and the discovery moment is a rotation that appears to have no effect, or a new pod scheduled after a value expired.

Alert on the operator’s own status — the resource’s sync condition and the age of the last successful sync — rather than on application health. The two are deliberately decoupled, and monitoring only the application means monitoring the thing that is designed not to notice.

Secure implementation

# externalsecret.yaml — the repository names keys, never values
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
spec:
  refreshInterval: 1h                 # how often the manager is polled
  secretStoreRef:
    name: production-store
    kind: SecretStore
  target:
    name: api-credentials             # the Secret the operator creates and owns
    creationPolicy: Owner             # deleting this resource removes the Secret too
  data:
    - secretKey: database_url         # the file name the pod will see
      remoteRef:
        key: prod/api/database-url    # the key in the manager
    - secretKey: api_token
      remoteRef:
        key: prod/api/token

# deployment.yaml — mounted read-only, one file per key
volumes:
  - name: app-secrets
    secret:
      secretName: api-credentials
      defaultMode: 0400
volumeMounts:
  - name: app-secrets
    mountPath: /etc/secrets
    readOnly: true

# config.py — unchanged by the operator's presence
from pathlib import Path
from pydantic import SecretStr, PostgresDsn, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict

SECRETS_DIR = Path("/etc/secrets")


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir=str(SECRETS_DIR) if SECRETS_DIR.is_dir() else None,
        extra="forbid",
    )

    database_url: PostgresDsn        # from /etc/secrets/database_url
    api_token: SecretStr             # from /etc/secrets/api_token

    @field_validator("api_token", mode="before")
    @classmethod
    def _strip(cls, value: str) -> str:
        # values written through a console or a file often carry a trailing newline
        return value.strip() if isinstance(value, str) else value


settings = Settings()                # a missing key fails the pod's startup, loudly

The secretKey names in the resource are the file names your model will read, so choosing them to match your field names removes an entire layer of mapping. It also makes the resource a readable inventory: anyone can see which credentials this workload consumes and where each comes from in the manager, which is the document you want during a rotation and during an incident.

creationPolicy: Owner is the setting that keeps the system tidy. It makes the operator own the Secret’s lifecycle, so removing the resource removes the object rather than leaving an orphaned Secret holding a live credential in the namespace — which is a surprisingly common way for old credentials to persist long after the workload that used them was deleted.

Choosing a refresh interval

The interval trades reconciliation load and manager cost against how quickly a rotation propagates. A short interval means a rotated value reaches pods sooner; it also multiplies read operations against the manager, which is both billed and rate-limited on every major provider, and multiplies the audit-log volume you will later search.

An hour suits most workloads. A rotation then reaches the Secret within an hour, the mounted files within about another minute, and the running process whenever it re-reads or restarts — which is the term that actually dominates, so shortening the operator’s interval below the process’s re-read cadence buys nothing at all. That relationship is the useful one to reason about: propagation is the sum of the intervals in the chain, and there is no point optimising the fastest link.

Shorten it deliberately for credentials with short lifetimes, and lengthen it for values that change once a year. Where a rotation must take effect immediately, the answer is not a very short interval but an explicit trigger — force a reconciliation as a step in the rotation runbook, then restart the affected workloads. That makes propagation a known-duration action rather than something you wait out.

The operator’s own access to the manager deserves as much thought as everything it distributes. It is a single component that can read every secret every workload consumes, which makes its credential the highest-value one in the environment. Prefer a platform identity — a workload identity binding, an instance role — over a stored key, scope its read access to the paths it actually syncs rather than the whole store, and give it its own audit trail so “who read this secret” distinguishes the operator’s scheduled reads from anything else.

Gotchas and version-specific behaviour

• The operator’s output is an ordinary Secret, so every consumption caveat from the parent topic still applies — environment variables stay frozen at container start.
• creationPolicy: Owner ties the Secret’s lifecycle to the resource; other policies can leave orphaned objects holding live credentials.
• A hand-edited Secret is reverted at the next reconciliation, so emergency changes must go into the manager.
• A failed reconciliation leaves the previous values in place, which protects running pods and hides the failure — alert on sync age.
• Template features that assemble values from several keys are powerful and add a second place where a value can be malformed; keep them simple.
• The operator’s own credentials for the manager are the highest-value secret in the namespace; prefer a platform identity over a stored key.
• Deleting the resource deletes the Secret under an owning policy, which will stop workloads that mount it — check consumers before removing one.

Production parity checklist

• The repository contains key names and store references, never values.
• Each workload has its own resource and its own Secret, rather than sharing one across services.
• secretKey names match the settings model’s field names so no mapping layer is needed.
• Secrets are mounted read-only with a restrictive mode, not injected as environment variables.
• Alerts fire on sync age and sync condition, independently of application health.
• The rotation runbook includes forcing a reconciliation and restarting affected workloads, with the expected propagation window written down.
• The operator authenticates to the manager with a platform identity scoped to the paths it syncs.

Key takeaways

An operator moves the source of truth into the manager without changing anything on the pod side. The repository holds key names, the operator reconciles them into an ordinary Secret, and the Python process reads mounted files through secrets_dir exactly as it would with a hand-managed object. That invariance is what makes adoption cheap and reversible.

The failure modes all come from the reconciliation loop rather than from the application. A hand-edited Secret is silently reverted, so emergency changes belong in the manager; a failed sync leaves the last values in place and is invisible from the application’s health, so alerting must watch the sync age; and propagation time is the sum of the operator’s interval, the kubelet’s file refresh, and the process’s own re-read cadence — which means the rotation runbook, not a shorter interval, is what makes a deadline. See the Kubernetes secrets topic for the delivery mechanics this builds on.