Kubernetes Secrets for Python Workloads

The rotation was completed on Tuesday: the credential was changed in the manager, the Kubernetes Secret was updated, the change was confirmed by reading it back. On Thursday the old credential was disabled, and every Python pod failed at once — because each of them had read the value into an environment variable at container start and had been holding the stale copy ever since. Nothing was misconfigured. The delivery mechanism simply does not do what the team assumed it did. Choosing how secrets reach a pod, and knowing exactly how each choice behaves when the value changes, is the substance of running secrets on Kubernetes, and it sits under enterprise secrets management as the deployment-side counterpart to the store integrations.

There are three delivery paths in practice, and they differ on the two questions that matter: who can read the value, and what happens when it changes. An environment variable sourced from a secretKeyRef is resolved once at container start and frozen. A volume-mounted Secret becomes a file the kubelet refreshes in the background, so the file changes while the process runs. An external operator keeps the manager as the source of truth and writes into Kubernetes on a schedule, which changes where rotation is initiated but not how the pod ultimately consumes the result. Every design decision on this page follows from those behaviours.

The arrival path decides rotation behaviour — the application code barely changes between them.

Where the delivery mechanism fits

The Python side of this is deliberately thin. A validated settings model reads whatever the platform provides, and the only thing Kubernetes changes is which source that model consults. Environment variables need no configuration at all; mounted files need secrets_dir pointed at the mount path; an operator-synced Secret is consumed as one of the other two, because the operator’s output is a Kubernetes Secret. That means the choice between mechanisms is an operational decision about rotation and exposure, not an application-architecture decision — and it can be changed later without rewriting application code, which is a useful property to preserve deliberately.

What the choice does affect is where the value is visible. An environment variable is readable by anything inside the container, appears in /proc/self/environ, is inherited by every subprocess the application spawns, and is frequently dumped by crash handlers and diagnostic endpoints. A file at /etc/secrets/api-token is readable only by processes that open it, is not inherited by subprocesses, and does not appear in any environment listing. Both are readable by anyone who can exec into the pod, so neither protects against an attacker with that access — but the file confines the value to code that asks for it, which meaningfully shrinks the accidental-disclosure surface described on the redaction page.

The third factor is who can read the Secret from the API. A Kubernetes Secret is an object in a namespace, and any principal with get secrets in that namespace can read every value in it, base64 decoded in one command. That includes service accounts you granted broad read access to for unrelated reasons, and it includes anyone with the ability to create a pod in the namespace, because a pod can mount any Secret there. Keeping access scoped is therefore as important as choosing a delivery mechanism, and it is the control most often left at whatever the default installation produced.

Secure implementation

Mount the Secret as files, point the settings model at the mount, and keep the manifest free of values.

# deployment.yaml — the manifest names a secret, it never contains one
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          env:
            - name: ENVIRONMENT
              value: production          # non-secret config is fine inline
          volumeMounts:
            - name: app-secrets
              mountPath: /etc/secrets    # files appear here, one per key
              readOnly: true
      volumes:
        - name: app-secrets
          secret:
            secretName: api-credentials
            defaultMode: 0400            # owner-read only, no group or other access

# config.py — pydantic-settings reads a directory of secret files natively
from pathlib import Path
from pydantic import SecretStr, PostgresDsn
from pydantic_settings import BaseSettings, SettingsConfigDict

SECRETS_DIR = Path("/etc/secrets")


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir=str(SECRETS_DIR) if SECRETS_DIR.is_dir() else None,
        extra="forbid",
    )

    environment: str = "local"
    database_url: PostgresDsn      # file: /etc/secrets/database_url
    api_token: SecretStr           # file: /etc/secrets/api_token


settings = Settings()              # a missing or malformed file fails the pod's startup

Three details make this work in practice. defaultMode: 0400 restricts the files to the container’s user, which matters because a container image that runs as root and then drops privileges will otherwise leave world-readable credentials behind. The conditional secrets_dir lets the same image run locally, where /etc/secrets does not exist and the model falls back to environment variables and a .env file — one image, one schema, no Kubernetes-specific code path. And because the model is constructed at import, a Secret with a missing key or a malformed URL fails the container at startup, so the deployment’s rollout stalls with a crash-looping new pod while the previous version continues serving. That is exactly the failure shape you want: visible, automatic, and non-destructive.

One caveat on file contents: whatever wrote the Secret decides whether the value ends with a newline. A value created with kubectl create secret generic --from-file keeps the file’s trailing newline, and pydantic-settings will hand you a token with \n on the end, which most APIs reject with an unhelpful authentication error. Strip whitespace in a validator on any field sourced from a file, and the class of problem disappears permanently.

The file refreshes on its own; the value your process holds does not. Both facts have to be designed for.

Mechanism reference

Mechanism	Updates without restart	Visible in the container’s environment	Python integration	Best for
`env` + `secretKeyRef`	no	yes, and to subprocesses	nothing to configure	values that never rotate
`envFrom` a whole Secret	no	yes, every key at once	nothing to configure	rarely — it hides what is injected
volume-mounted Secret	file yes, process no	no	`secrets_dir`	most workloads
projected service account token	yes, rotated by Kubernetes	no	read the file per request	authenticating to a manager
operator-synced Secret	as per the mount type used	depends on consumption	unchanged	keeping the manager authoritative

envFrom deserves the warning it gets. Injecting every key of a Secret as environment variables means the pod’s environment changes whenever anyone adds a key to that Secret, which makes extra="forbid" in the settings model fail the deployment for a reason nobody expects. It also makes it impossible to tell from the manifest what the container actually receives. Name the keys you consume explicitly, or mount the Secret and let the model’s schema define the contract.

The immutability option is worth a note alongside the table. Marking a Secret immutable improves API-server performance and prevents accidental edits, which sounds appealing until you remember that it also disables the file refresh that makes mounted Secrets rotatable in place. An immutable Secret must be replaced under a new name and the workload updated to point at it, so choosing immutability is choosing the restart-based rotation story permanently. That is a legitimate choice for values that never change — a signing certificate for a fixed period, say — and a poor one for anything on a rotation schedule.

The projected service-account token row is the one that unlocks the better architecture. A short-lived, automatically-rotated token that identifies the pod to an external system means the pod does not need a long-lived credential for that system at all — it exchanges its identity for a lease. That is the mechanism behind Kubernetes authentication in HashiCorp Vault and behind cloud workload identity, and it changes the rotation problem from “distribute a new secret everywhere” to “the platform already rotates it”.

Deployment parity, step by step

Keep values out of the repository. The manifest references secretName; the value comes from a manager, an operator, or a sealed artefact. A stringData block with a real credential in version control is the failure this whole area exists to prevent.
Mount rather than inject. Use a volume with defaultMode: 0400 so the value stays out of the environment and out of subprocesses.
Point the model at the mount, conditionally. The same image must run locally without /etc/secrets, so detect the directory rather than hard-coding a Kubernetes-only path.
Strip whitespace on file-sourced fields. One validator prevents the trailing-newline class of authentication failure.
Fail startup on missing or malformed values. Construct the settings model at import so a bad Secret crash-loops the new pod and the rollout stalls with the old version still serving.
Decide the rotation story before you need it. Either the application re-reads on a schedule, or rotation triggers a rolling restart. “The file updates automatically” is not a rotation story on its own.

Step 6 is the one teams postpone and then discover under pressure. The two viable answers are genuinely different in cost. A rolling restart is simple, works with every delivery mechanism, and briefly doubles resource usage; it also means rotation and deployment share one mechanism, which is easy to reason about. Re-reading in the process avoids the restart entirely but requires the application to hold credentials behind an accessor rather than a module-level constant, and to handle the window where a client holds an old connection. Pick one deliberately, write it down, and test it — a rotation rehearsal in staging costs an hour and is the only way to find out which of your services actually re-read.

Identity instead of a stored credential

Every mechanism above distributes a stored credential, and every stored credential carries the same lifecycle burden: it must be created, delivered, rotated, revoked and audited. The alternative worth reaching for is to stop storing one. A pod already has an identity — its service account — and Kubernetes can present that identity to an external system as a short-lived, automatically-rotated token projected into the container’s filesystem. Systems that understand this identity can exchange it for a lease: a database credential valid for an hour, a cloud role session valid for a few minutes, an API token scoped to one workload.

The shape of the Python code changes very little. Instead of reading a password from a file at boot, the application reads the projected token at the moment it needs a credential, exchanges it, and caches the resulting lease until shortly before expiry. That is a few dozen lines around whichever client library you use, and the pattern is identical whether the far side is Vault’s Kubernetes auth method, a cloud provider’s workload identity federation, or an internal service that trusts the projected token’s issuer.

What changes a great deal is the operational picture. There is no long-lived secret to rotate, so the rotation rehearsal that step 6 asks for becomes unnecessary for that credential. There is no Secret object to protect, so the namespace-read concern does not apply to it. Revocation is immediate and per-workload: remove the role binding and the exchange stops succeeding, without touching any running pod’s files. And the audit trail improves, because the far side sees a distinct identity per workload rather than a shared credential that several services happen to hold.

The trade-off is a hard dependency on the exchange service being reachable at startup, and a class of failure — expired lease, revoked binding, clock skew — that does not exist with a static file. Both are manageable: fail the pod’s readiness probe rather than crashing when a renewal fails, keep the lease refresh well ahead of expiry, and make sure the failure is legible in logs so an on-call engineer is not debugging an authentication error with no context. For workloads that talk to systems supporting this exchange, it is the strongest available answer, and it makes the rest of this page apply only to the credentials that genuinely have to be stored.

Trading a stored secret for an exchangeable identity removes the rotation problem rather than automating it.

Security boundaries and operational guardrails

Base64 is an encoding, not encryption. Anyone who can read the Secret object has the value; the encoding stops nothing.
Enable encryption at rest on the API server. Without an encryption provider configured, an etcd backup is a plaintext credential archive.
Scope get secrets narrowly. Read access in a namespace is read access to every credential in it, and pod-creation rights are equivalent to read access.
Never envFrom a Secret you do not fully control. A key added elsewhere becomes an environment variable in your process.
Set defaultMode: 0400 and run as a non-root user. Default modes are more permissive than most teams assume.
Keep the manager authoritative. If the Secret is edited by hand as well as synced, the two will diverge and the manual edit will win until the next sync, which is the worst of both.

The last bullet describes a failure mode that is worth recognising early because it looks like a tooling bug rather than a process problem. Once an operator is reconciling a Secret, a hand-edited value survives only until the next sync interval, so somebody debugging an incident makes a change, verifies it works, and finds the problem back an hour later with no record of what happened. The remedy is procedural rather than technical: treat the synced object as read-only, make emergency changes in the manager where they will persist and be audited, and if a break-glass path is genuinely needed, implement it by pausing the operator’s reconciliation explicitly rather than by racing it.

The RBAC point deserves more than a bullet. The permission most often over-granted is not get secrets directly but the ability to create pods in a namespace, because a pod definition may mount any Secret in that namespace and print it. Any principal with pod-creation rights therefore has read access to every secret in the namespace, whatever the Secret-level rules say. The practical consequence is that namespace boundaries are the real secret boundaries: separating workloads that must not read each other’s credentials means separate namespaces, not cleverer role bindings.

Troubleshooting

The rotation had no effect. The value arrives as an environment variable, which is fixed at container start. Confirm with kubectl exec and compare the process environment against the current Secret; the fix is a rolling restart now, and a switch to a mounted file before the next scheduled rotation so the same surprise does not recur.

The file updated but the application still uses the old value. The kubelet refreshed the file; nothing told the process. Either re-read the file behind an accessor, or restart. This is the single most common surprise in this area, and it is easy to mistake for a caching bug in the client library.

Authentication fails with a value that looks correct. Suspect a trailing newline from --from-file, or an accidental double base64 encoding when the Secret was created with data rather than stringData. Both produce credentials that look right in a terminal and are wrong by one character.

The pod cannot start: the volume never mounts. The Secret does not exist in that namespace, or the name is misspelled. A pod waiting on a missing Secret stays in ContainerCreating rather than failing outright, so it is easy to misread as a scheduling problem. kubectl describe pod names the missing object.

A new key in the Secret broke the deployment. envFrom injected it, and extra="forbid" rejected it. This is the model working correctly on input it should never have received; switch to explicit keys or a mount.

A sidecar or init container cannot see the mounted files. Volume mounts are per-container, not per-pod. Each container that needs the values must declare the same volumeMounts entry, and a common mistake is mounting into the application container while an init container that runs migrations gets nothing. The symptom is a migration job failing on a missing database URL while the application starts perfectly, which reads as a code problem and is a manifest problem.

Everything works in one environment and fails in another with the same manifest. Compare the two Secrets key by key rather than comparing the manifests, which are identical by construction. The usual differences are a key present in one and absent in the other, a value created with data in one place and stringData in the other, or a namespace-level default that differs. Since the model uses extra="forbid" and required fields, both a missing key and an unexpected one produce a clear startup error naming the field — read the crash-looping pod’s logs before inspecting anything else.

Secrets differ between two environments that should match. Something was edited by hand in one of them. Reconcile from the manager and, if an operator is in use, check its sync status rather than the Secret — the object is downstream of the real source.

Three routes to the same value — which is why workload separation is a namespace decision.

Frequently asked questions

Do environment variables from a Secret update when the Secret changes?

No. Environment variables are resolved when the container starts and are fixed for the life of the process. Editing the Secret changes nothing until the pod is replaced, which is why a rotation that only updates the Secret appears to have no effect. There is no mechanism by which Kubernetes could change them — a process’s environment is set at exec time and is not writable from outside. If you rely on environment injection, rotation must include a rolling restart of every workload consuming that Secret, and that restart is part of the rotation procedure rather than an optional follow-up.

Are mounted secret files updated automatically?

Yes, for volume-mounted Secrets the kubelet refreshes the files periodically, typically within about a minute. Your application still has to re-read them; the file changing does not change a value your process already loaded into memory. The refresh is also atomic in a helpful way — the mount uses a symlinked directory swap, so a reader never sees a half-written file — which means a periodic re-read is safe to implement naively. What you cannot do is assume that a Secret marked immutable, or one consumed through a subPath mount, will refresh: both opt out of the update behaviour.

Is a Kubernetes Secret encrypted?

Only if the API server is configured with an encryption provider. By default the value is stored base64-encoded in etcd, which is an encoding, not encryption — a backup of etcd is a backup of every credential in readable form. Enabling encryption at rest is an API-server configuration change, unrelated to which secrets tooling you use, and it is the first thing to verify on any environment that holds real credentials. Note that it protects the data at rest only: anyone authorised to read the object through the API still receives the decrypted value, as intended.

Should Python read secrets from files or environment variables in Kubernetes?

Files, in most cases. They can be refreshed without a restart, they do not appear in a process’s environment listing, and pydantic-settings reads a directory of them natively through secrets_dir with no custom code. Environment variables remain reasonable for values that genuinely never rotate and for the non-secret configuration around them. The one place files are inconvenient is a third-party component that only accepts an environment variable; there, inject that single variable explicitly rather than switching the whole workload back to environment delivery.

What does an external secrets operator add over a plain Secret?

It keeps the source of truth in your secret manager and syncs it into Kubernetes on a schedule, so rotation happens in one place and the manifest in your repository names a secret rather than containing one. The operator reconciles continuously, which means a hand-edited Secret is reverted rather than quietly diverging, and it gives you an audit trail in the manager covering every environment at once. What it does not change is how the pod consumes the result — the operator’s output is an ordinary Secret, so the mount-versus-inject decision and its rotation consequences are exactly as described above.

Key takeaways

The invariant this topic enforces is that the delivery mechanism, not the application, determines rotation behaviour — so it must be chosen deliberately and written down. Environment variables freeze at container start and require a restart to change. Mounted files refresh on their own but do not change what a running process already holds. An operator moves the source of truth into a manager without altering either of those facts on the pod side. A team that knows which of these it is using can answer “what happens when we rotate?” in one sentence; a team that does not will find out on the day the old credential is disabled.

The Python side stays small if you let it. Mount the Secret read-only with a restrictive mode, point secrets_dir at it conditionally so the same image still runs on a laptop, strip whitespace on file-sourced fields, and construct the model at import so a bad value crash-loops the new pod instead of serving broken requests. Around that, treat the namespace as the real security boundary, enable encryption at rest, and prefer a short-lived projected identity over a long-lived credential wherever the platform you are authenticating to supports it. The result is a workload where secrets are boring: injected the same way everywhere, validated before traffic arrives, and rotatable on a schedule somebody has actually rehearsed.