Zero-downtime secret rotation in Python

The hard part of rotation is not generating a new credential — it is swapping it into a running service without a single failed request. The answer is a refreshing cache plus an overlap window where both credentials work. This page implements it, extending Automated Secret Rotation Patterns.

Everything below is about the moment of the swap, which is where the concurrency lives. A single-threaded script picking up a new credential is trivial; a service with sixteen threads, a connection pool, and traffic arriving throughout has three distinct problems — that the refresh happens once rather than sixteen times, that no request sees a partially updated state, and that connections built from the old credential are retired without interrupting work in progress.

Three concurrency problems at the moment of the swap The refresh must happen once rather than per thread, no request may observe a partial update, and connections built from the old credential must retire without interrupting work. the swap is where all the difficulty is refresh once 16 threads see "expired" 1 should call the store double-checked lock no partial state value and timestamp must change together one tuple, one assignment retire gracefully old pool must drain not be severed dispose(close=False)
Three separate problems, three separate mechanisms — solving one and not the others still produces failed requests.

Problem 1: restart-to-rotate

# ANTI-PATTERN: the only way to pick up a new secret is a redeploy
SECRET = fetch_secret()    # module-level; rotation requires restarting every pod

Tying rotation to a restart means downtime and a manual step. It also means rotation happens on the deployment schedule rather than the security schedule — and since teams reasonably avoid deploying during quiet hours, the rotation that was configured for 3 a.m. either does not happen or happens with a human watching, neither of which is what “automated rotation” was supposed to mean.

The deeper problem is coupling. Once picking up a credential requires a restart, every rotation is a change with deployment risk attached, so rotations get batched, deferred, and eventually skipped. Decoupling the two is what lets rotation become frequent enough to matter, and it is the reason the refreshing cache is worth building even for a service that currently rotates only once a quarter.

Problem 2: a race on refresh

# ANTI-PATTERN: two threads refresh at once, one overwrites the other
if expired:
    global SECRET
    SECRET = fetch_secret()    # no lock: torn reads under concurrency

Without a lock, concurrent requests can see a half-updated value. In CPython the assignment itself is atomic, so a reader will never observe a corrupted string — but that is the least interesting part of the problem, and relying on it invites two real failures.

The first is the stampede: sixteen threads all observe the expired flag, all call the store, and the service issues sixteen identical requests where one would do. Under load that is throttling, added latency, and a spike of API cost at every TTL boundary.

The second is state that spans more than one variable. A cache holding a value and the time it was loaded has two assignments, and a reader between them sees a new value with an old timestamp or vice versa. That is a genuine torn read, and the fix is either a lock or storing both in a single tuple so one assignment updates them together.

Two failures from an unlocked refresh Without a lock every thread calls the store at the TTL boundary, and a cache holding a value and timestamp separately can be observed half-updated. stampede every thread sees "expired" every thread calls the store throttling at each TTL boundary latency spike, avoidable cost torn state value and timestamp are two writes a reader can land between them new value, stale timestamp fix: one tuple, or a lock
Atomic assignment in CPython saves you from corrupted strings and from neither of these.

Secure implementation

# secrets/refresh.py
import threading, time
from pydantic import SecretStr

class RefreshingSecret:
    def __init__(self, fetch, ttl: int = 300):
        self._fetch, self._ttl = fetch, ttl     # ttl < overlap window
        self._value: SecretStr | None = None
        self._at = 0.0
        self._lock = threading.Lock()

    def get(self) -> SecretStr:
        now = time.monotonic()
        if self._value is None or now - self._at > self._ttl:
            with self._lock:                     # one refresh under contention
                if self._value is None or time.monotonic() - self._at > self._ttl:
                    self._value, self._at = self._fetch(), time.monotonic()
        return self._value

def on_rotation(pool, secret: RefreshingSecret) -> None:
    pool.recreate(password=secret.get().get_secret_value())   # graceful pool reload

The double-checked lock guarantees exactly one refresh under load; the TTL stays below the store’s overlap window, so the old credential is still valid while the cache catches up; pools are recreated rather than torn down mid-request.

The second check inside the lock is the part that is easy to omit and essential to keep. Fifteen threads queue on the lock behind the one that is fetching; when they acquire it, the value has already been refreshed, and without the re-check each of them would fetch again in turn — producing the stampede one thread at a time instead of all at once. The re-check makes them observe the fresh value and return immediately.

The tuple assignment self._value, self._at = ... matters for the readers outside the lock. It compiles to a single store of a tuple followed by two unpacking stores, so strictly it is not one atomic update — if that bothers you, store an actual tuple in one attribute and unpack on read. In practice the window is a single bytecode and the consequence is one unnecessary refresh, which is why most implementations accept it; being aware of the trade-off, rather than discovering it in a review, is what matters.

Reading outside the lock on the fast path is the whole reason for this structure. The overwhelming majority of calls find a fresh value and never touch the lock at all, so the cost of thread-safety falls entirely on the rare expiry rather than on every request.

Double-checked locking under contention One thread acquires the lock and fetches while the others queue, and the second check inside the lock means they return the fresh value instead of fetching in turn. 16 threads reach an expired cache at once check 1 · outside all 16 see "expired" lock 1 enters, 15 queue check 2 · inside now fresh — return 1 fetch Without check 2 the fifteen queued threads each fetch in turn — the same stampede, merely serialised.
Omitting the inner check does not remove the stampede; it spreads it out, which is harder to spot and just as costly.

The same pattern under asyncio

A threaded lock in an async service is a mistake with a specific failure mode: threading.Lock blocks the event loop, so while one coroutine waits on the store, every other coroutine on that loop — including ones with nothing to do with secrets — stops running. The async equivalent needs asyncio.Lock, which yields instead of blocking.

# secrets/refresh_async.py
import asyncio, time
from pydantic import SecretStr

class AsyncRefreshingSecret:
    def __init__(self, fetch, ttl: int = 300):
        self._fetch, self._ttl = fetch, ttl      # fetch is an async callable
        self._value: SecretStr | None = None
        self._at = 0.0
        self._lock = asyncio.Lock()

    async def get(self) -> SecretStr:
        if self._value is not None and time.monotonic() - self._at <= self._ttl:
            return self._value                   # fast path: no await, no lock
        async with self._lock:
            if self._value is None or time.monotonic() - self._at > self._ttl:
                self._value, self._at = await self._fetch(), time.monotonic()
            return self._value

The structure is identical and one detail differs importantly: the fast path returns without awaiting anything, so a cache hit does not yield control to the loop. That keeps the common case as cheap as a dictionary lookup, and it means adding this to a hot request path costs effectively nothing.

The fetch callable must be genuinely async. Passing a synchronous boto3 call wrapped in an async signature blocks the loop for the duration of the network round-trip — the exact problem asyncio.Lock was chosen to avoid, reintroduced one line lower. Either use an async-native client such as aioboto3, or run the sync call in a thread executor with asyncio.to_thread.

One caveat worth knowing: an asyncio.Lock is bound to the event loop that created it, so an instance created at import time and used from a loop started later can raise. Creating the lock lazily inside the first get() call, or constructing the whole object inside the application’s startup hook, avoids a failure that only appears in certain test arrangements.

Threading lock versus asyncio lock in an async service A threading lock blocks the entire event loop during a fetch, while an asyncio lock yields so unrelated coroutines continue running. threading.Lock one coroutine waits on the store the loop cannot run anything else unrelated requests stall too latency spike across the service asyncio.Lock the waiting coroutine yields the loop keeps serving only the secret path waits fast path never awaits at all
The wrong lock in an async service turns a single secret refresh into a service-wide latency spike.

Reloading the pool without dropping a request

pool.recreate(...) in the snippet stands in for whatever your driver offers, and the shape of that operation decides whether the swap is truly invisible. Getting it wrong is the most common reason a rotation that “works” still produces a handful of errors in the logs.

# db/pool.py — swap the engine, drain the old one
import threading
from sqlalchemy import create_engine

class ReloadablePool:
    def __init__(self, url_for, secret):
        self._url_for, self._secret = url_for, secret
        self._engine = None
        self._built_from: str | None = None
        self._lock = threading.Lock()

    def engine(self):
        value = self._secret.get().get_secret_value()
        if self._engine is not None and value == self._built_from:
            return self._engine                     # fast path, no lock
        with self._lock:
            if self._engine is None or value != self._built_from:
                old = self._engine
                self._engine = create_engine(self._url_for(value), pool_pre_ping=True)
                self._built_from = value
                if old is not None:
                    old.dispose(close=False)        # drain: finish in-flight work
        return self._engine

dispose(close=False) is the operative call. It detaches the old pool so no new checkouts come from it, while leaving connections currently checked out alone — a query already running completes normally and its connection closes when returned. Plain dispose() closes everything immediately, which aborts in-flight statements and produces exactly the errors this design exists to avoid.

pool_pre_ping=True covers the remaining case. Even after the swap, a connection may have been closed by the server because the credential behind it was revoked; pre-ping tests each connection before handing it out and transparently replaces any that fail. The two together mean neither the swap nor the revocation is visible to application code.

The same fast-path-then-lock structure appears here as in the secret cache, and for the same reason: comparing the credential is cheap and happens on every call, while rebuilding is expensive and happens twice a quarter. Repeating the pattern rather than inventing a second one keeps both of them easy to reason about.

Draining the old pool versus closing it Disposing without closing lets in-flight queries finish on the old pool while new work uses the new one, whereas closing immediately aborts running statements. what happens to the queries already running dispose() connections closed at once running statements aborted a burst of errors at every rotation "rotation works, mostly" dispose(close=False) no new checkouts from the old pool in-flight queries finish normally connections close as they return genuinely zero failed requests
One keyword argument separates "rotation works, apart from a few errors" from rotation nobody can detect.

Gotchas & version-specific behaviour

  • The cache TTL must be shorter than the credential overlap window or requests fail at the cutover.
  • Use double-checked locking so a refresh under load does not stampede the secret store.
  • Recreate connection pools on change; do not close active connections abruptly.
  • time.monotonic() for all timing so clock changes cannot extend the TTL.
  • In an async service use asyncio.Lock and an async fetch — a blocking call inside the lock stalls the whole loop.
  • A fetch that raises inside the lock leaves the old value in place, which is usually the behaviour you want; make sure it is deliberate rather than accidental.

That last point deserves attention because the code above gets it right by accident of structure. If self._fetch() raises, the assignment never happens, the lock is released by the with, and the previous value remains cached with its old timestamp — so the next call retries immediately. That is reasonable behaviour: a transient store failure does not discard a working credential. What it also means is that a persistently failing store produces a retry on every call rather than one per TTL, so adding a short backoff after repeated consecutive failures is worth doing for any service that sees real traffic.

Production parity checklist

  • Rotation needs no restart — the refreshing cache picks up new values.
  • Cache TTL is below the overlap window.
  • Refresh is thread-safe (double-checked lock).
  • Pools reload gracefully on credential change.
  • A staging drill forces rotation and asserts zero failed requests.
  • The refresh path is exercised by a concurrency test, not only by a single-threaded one.

The concurrency test is the item most often skipped and the easiest to write. Point the cache at a fetch function that counts its calls and sleeps briefly, start a dozen threads that all call get() at an expiry boundary, and assert the counter reads exactly one. That single assertion covers the stampede, the double-checked lock, and the re-check inside it — three behaviours that are otherwise verified only by hoping.

The staging drill covers what the unit test cannot. Force a real rotation while a small load generator runs against the service, and assert that the error count over the window is exactly zero — not “low”, zero, because a rotation that produces two errors produces two thousand at a hundred times the traffic. Running it on a schedule catches the changes that silently re-arm the failure: a TTL raised past the overlap window, a dispose() that lost its keyword argument, a pool rebuilt on a timer instead of on change.

Key takeaways

A thread-safe refreshing cache sized under the overlap window turns rotation into a non-event — no restart, no failed request. The concurrency details are what make it hold under load: check outside the lock for a cheap fast path, check again inside it so queued threads do not fetch in turn, and keep the value and its timestamp updated together so no reader sees a mismatched pair.

In an async service the same structure applies with asyncio.Lock and an async fetch, and the fast path must not await — otherwise a mechanism intended to be invisible becomes a per-request yield. Verify all of it with a concurrency test that asserts exactly one fetch under contention, because that is the property that silently degrades and the one no manual check will ever notice. A cache that has quietly started fetching sixteen times per expiry still returns correct values and still passes every functional test — it simply costs sixteen times as much and throttles under load. For the overlap-window mechanics in the store, see Automated Secret Rotation Patterns.

Back to Automated Secret Rotation Patterns