Reload Configuration Without Restarting Pods

A mounted Secret refreshes on disk within about a minute of a rotation. The value your process is holding does not change at all, because it was read once at startup into an object that nothing updates. Closing that gap — picking up the new value without a restart — is a small piece of engineering with a few sharp edges: what to do when the new value is invalid, what happens to connections built from the old one, and whether the whole exercise is worth it compared with a rolling restart. This page covers all three, extending the Kubernetes secrets topic.

Problem 1: a module-level constant nothing can update

# ANTI-PATTERN: bound once at import, everywhere, forever
from config import settings

API_TOKEN = settings.api_token.get_secret_value()      # a copy, taken at import

def call():
    return httpx.post(URL, headers={"Authorization": f"Bearer {API_TOKEN}"})

Even with a perfect refresh mechanism updating the settings object, this code never sees a new value: it took a copy at import and every call uses that copy. The same applies to any client constructed at import from a credential, any default argument evaluated once, and any closure that captured the value.

The prerequisite for reload is therefore an accessor, not a constant. Reading settings().api_token at the point of use costs an attribute lookup and means the value can change underneath the caller. It is a small discipline that has to be applied consistently — one leftover module constant is enough to leave a component holding a stale credential and produce a partial failure that is genuinely confusing to diagnose.

Problem 2: swapping in an invalid configuration

# ANTI-PATTERN: the new object replaces the old before anything checks it
def refresh():
    global _settings
    _settings = Settings()        # if this raises, the process is left mid-swap

A refresh runs while the process is healthy and serving. If the new values are malformed — a half-written file, a key removed by mistake, a manager returning an error payload — then constructing the model raises, and depending on where the exception lands you either lose the settings object entirely or keep a partially updated one. Either outcome converts a bad value into a bad process, which is much worse than the situation you started in.

The correct shape validates first and swaps only on success. Construct the candidate, let validation run, and assign to the shared reference only if construction returned. On failure, log and keep serving with what you have: the old credential is presumably still working, since the process is up, so continuing is strictly better than failing. This is the opposite of the startup policy, where an invalid configuration must stop the process — and the asymmetry is deliberate.

Problem 3: expecting existing connections to re-authenticate

# ANTI-PATTERN: the pool is still using the credential it was built with
pool = ConnectionPool(dsn=settings().database_url)     # built once, at startup

Refreshing settings does not touch a connection that has already authenticated. A pooled database connection established with the old password stays open and usable until it is closed, which means a rotation followed by immediate revocation of the old credential produces failures at unpredictable moments as pooled connections are recycled — not at the moment of rotation, and not all at once.

Where a rotation must complete on a deadline, the settings refresh has to be paired with recycling whatever was built from the old value: close idle pooled connections, let in-flight work finish, and let the pool re-establish using the accessor. That is more machinery than a settings swap, and it is the point where “just restart the pods” starts to look attractive for good reasons.

Secure implementation

# config.py — an accessor, a validated swap, and a visible timestamp
import logging
import threading
import time
from pathlib import Path
from pydantic import SecretStr, PostgresDsn
from pydantic_settings import BaseSettings, SettingsConfigDict

logger = logging.getLogger(__name__)
SECRETS_DIR = Path("/etc/secrets")


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir=str(SECRETS_DIR) if SECRETS_DIR.is_dir() else None,
        extra="forbid",
        frozen=True,                        # 1. immutable: refresh replaces, never mutates
    )
    environment: str = "local"
    database_url: PostgresDsn
    api_token: SecretStr


_lock = threading.Lock()
_current = Settings()                       # 2. startup: an invalid value must fail here
_loaded_at = time.time()


def settings() -> Settings:
    # 3. the accessor every call site uses instead of a module constant
    return _current


def refresh() -> bool:
    global _current, _loaded_at
    try:
        candidate = Settings()              # 4. validate BEFORE anything is replaced
    except Exception as exc:                # 5. keep serving with the values we have
        logger.warning("configuration refresh failed, keeping current values: %s", type(exc).__name__)
        return False
    with _lock:                             # 6. the swap itself is a single assignment
        _current, _loaded_at = candidate, time.time()
    logger.info("configuration refreshed")  # 7. never log values, only the event
    return True


def loaded_at() -> float:
    return _loaded_at

# refresher.py — a background loop, started after the event loop exists
import asyncio
from config import refresh


async def refresh_loop(interval_seconds: int = 60) -> None:
    while True:
        await asyncio.sleep(interval_seconds)
        await asyncio.to_thread(refresh)     # 8. file reads off the event loop

The frozen=True model is what makes the swap safe without any coordination on the read path. Because a settings object can never change after construction, a caller that grabbed the old object mid-refresh holds a complete, internally consistent configuration — just a slightly old one. That is exactly the property you want; the alternative, mutating fields on a shared object, means a caller can observe a half-updated configuration where the host is new and the password is old.

Note that the accessor returns the object rather than a copy, and that the lock only guards the assignment. Readers never take the lock, so refresh adds no contention to the request path. This is the standard shape for read-mostly shared state in Python and it stays correct under threads and under async, provided the object really is immutable.

When a rolling restart is the better answer

Live reload is a real feature with real maintenance cost, and for most services a rolling restart triggered by the rotation is simpler and safer. It reuses deployment machinery you already trust, it guarantees that everything built from the old configuration is rebuilt — pools, clients, caches, closures — and it needs no accessor discipline, no background loop and no reasoning about partially-stale components.

Reload earns its place in three situations. When restarts are expensive: a process with a long warm-up, a large in-memory cache, or a slow readiness gate. When connections are long-lived and dropping them is disruptive: streaming consumers, websocket fan-out, long-poll workers. And when credentials rotate faster than you want to redeploy, which is the normal case for short-lived dynamic credentials from a manager, where the lease is measured in minutes.

Outside those cases, spend the effort on making the restart reliable instead: a readiness probe that gates traffic properly, a graceful shutdown that finishes in-flight work, and a rotation runbook that restarts every consumer rather than only the obvious one. That combination handles rotation, deployment and configuration change with one mechanism — and one mechanism that always works is worth more than two that each work sometimes.

Gotchas and version-specific behaviour

• A module-level constant or an import-time client keeps the old value regardless of any refresh; the accessor discipline has to be complete.
• Validate before swapping, and keep the previous object on failure — the opposite of the startup policy, deliberately.
• frozen=True makes the swap safe without locking readers, because no caller can observe a half-updated object.
• Connections and pools built from the old credential keep working until recycled; a deadline-bound rotation needs pool recycling too.
• Do the file read off the event loop in an async service — asyncio.to_thread is enough — since a mounted-file read is blocking I/O.
• Expose a load timestamp and the non-secret values so two pods can be compared during an incident without guesswork.
• Mark which fields are actually reloadable; a value consumed once at startup will not change behaviour no matter how often you refresh it.

Production parity checklist

• No module-level constant or import-time client holds a credential; every read goes through the accessor.
• The refresh validates a candidate and only swaps on success, logging failures without values.
• A health or status endpoint reports the environment, the load timestamp and no secrets.
• The rotation runbook states whether reload or restart is the mechanism, per workload.
• Pools and long-lived clients are recycled where the rotation has a revocation deadline.
• A staging rehearsal has proven that a rotated value actually takes effect within the expected window.

Key takeaways

Reload has three prerequisites and one policy. The prerequisites are an accessor rather than module constants, an immutable model so a swap is atomic from a reader’s point of view, and a background refresh that does its file reads off the request path. The policy is validate-then-swap: on failure keep the current values and log, because the process is working and a bad candidate is not a reason to stop.

What reload does not do is rebuild anything already constructed from the old configuration, so pools and long-lived clients need explicit recycling when a rotation has a revocation deadline. That extra machinery is the reason a rolling restart remains the right default for most services — it handles every case uniformly with tooling you already operate. Build reload where restarts are genuinely expensive or credentials genuinely short-lived, and in either case rehearse a rotation in staging so the propagation window is a measured number rather than an assumption. The delivery mechanics behind all of this are on the Kubernetes secrets topic page.