Automated Secret Rotation Patterns

Rotation that only happens after a breach is not rotation. The goal is to make credential replacement routine and invisible: the secret changes on a schedule, and not a single request fails. The mechanism is a dual-credential overlap window — both the old and new credentials work simultaneously while traffic shifts over.

Rotation is the operational backbone of the enterprise secrets section. It applies whether the store is AWS Secrets Manager or Vault.

Why the overlap is the whole design

Strip rotation down and it is one hard requirement: at no instant may a running process hold a credential the backend has already rejected. Everything else — schedules, versioning, reconnection logic — exists to satisfy that.

There are only two ways to satisfy it. Either every process learns the new credential before the old one stops working, which is the overlap approach, or every process is replaced at the moment of change, which is a rolling restart. The second is simpler to reason about and considerably more disruptive: it couples credential rotation to deployment, so rotating a secret means shipping, and shipping becomes something you avoid doing during quiet hours precisely when rotation is scheduled.

The overlap approach decouples them. The store creates the new credential and leaves the old one valid for a defined window; applications pick up the change on their own schedule, bounded by their cache TTL; and the old credential is revoked once the window closes. Nothing restarts, nothing deploys, and the only constraint is arithmetic — the cache TTL must be shorter than the overlap window, with margin.

That inequality is the single most important thing on this page. If the cache can hold a value for fifteen minutes and the overlap lasts ten, then some process will be holding a revoked credential for five minutes, and it will fail. Everything else here is elaboration on keeping those two numbers in the right order.

Secure implementation

# secrets/rotation.py
import time
from pydantic import SecretStr

class RotatingSecret:
    """Serves a secret from cache, refreshing within the overlap window."""

    def __init__(self, fetch, ttl: int = 300):
        self._fetch = fetch          # callable returning the current SecretStr
        self._ttl = ttl              # must be shorter than the overlap window
        self._value: SecretStr | None = None
        self._loaded_at = 0.0

    def get(self) -> SecretStr:
        if self._value is None or time.monotonic() - self._loaded_at > self._ttl:
            self._value = self._fetch()        # picks up the rotated value
            self._loaded_at = time.monotonic()
        return self._value

def reconnect_pool(pool, secret: RotatingSecret) -> None:
    pool.recreate(password=secret.get().get_secret_value())  # rebuild on rotation

The cache TTL is deliberately shorter than the store’s overlap window, so the application always picks up the new credential while the old one is still accepted.

Taking fetch as a callable rather than hard-coding a store is what makes this class reusable and testable. In production it is a closure over a Secrets Manager or Vault call; in a test it is a function returning values from a list, so a test can assert that the second get() after an expiry returns the rotated value and that a get() inside the TTL does not call fetch at all. Nothing about the class knows or cares which store it is talking to, which is also what lets one implementation serve a Vault-backed service and a Secrets Manager-backed one.

reconnect_pool is separate for the same reason the Vault client keeps renewal separate from reconnection: only the caller knows what was built from the credential. A single class trying to own both the secret and every consumer of it needs references to all of them, which is the coupling that makes such code impossible to test and awkward to extend when a second consumer appears.

What the snippet does not show is when reconnect_pool gets called, and that is the decision that matters most in practice. Calling it on a timer wastes work most of the time, since the credential usually has not changed. Calling it when the fetched value differs from the one the pool was built with is precise and cheap — compare the new value against the one you last used, and rebuild only when they differ.

Configuration reference

The gap between the last two rows is the strategic choice underneath all of this. A static secret rotated every sixty days spends most of its life as a credential that would be valid to an attacker for weeks; a dynamic credential leased for an hour is worthless almost immediately. Where a dynamic secrets engine can issue the credential — databases, cloud roles, certificates — that is the better answer, and the rotation machinery on this page becomes the fallback for credentials that come from outside and cannot be minted on demand.

The rotation interval deserves less agonising than it usually gets. A credential rotating reliably every ninety days is in far better shape than one nominally rotating weekly through a process that has been silently failing for months. Pick an interval you can actually sustain, verify it works end to end, and shorten it once the mechanism is proven — shortening is a configuration change, whereas fixing a broken rotation is a project.

The overlap window itself is worth choosing with the same pragmatism. It has to exceed the longest cache TTL among every consumer of that secret, and “every consumer” is the phrase that catches teams out — a web service on five minutes, a worker fleet on fifteen, a nightly batch job that reads once at startup and runs for hours. The batch job is the binding constraint, and an overlap sized for the web service leaves it holding a revoked credential mid-run. Enumerating consumers before setting the window is a five-minute exercise that prevents a class of failure which otherwise only appears on the nights that both the rotation and the batch job run.

Rebuilding consumers when the value changes

The half of rotation that lives in the application is deciding when a consumer must be rebuilt. Refreshing a cached secret is free and frequent; rebuilding a connection pool is expensive and should happen only when the credential actually changed. Comparing values makes that precise.

# secrets/rotation.py — rebuild only on an actual change
import threading
from typing import Callable

class CredentialBinding:
    """Holds an object built from a secret; rebuilds it when the secret changes."""

    def __init__(self, secret: RotatingSecret, build: Callable[[str], object]):
        self._secret, self._build = secret, build
        self._built: object | None = None
        self._built_from: str | None = None
        self._lock = threading.Lock()

    def current(self):
        value = self._secret.get().get_secret_value()
        with self._lock:
            if self._built is None or value != self._built_from:
                old, self._built = self._built, self._build(value)
                self._built_from = value
                self._retire(old)
            return self._built

    def _retire(self, old) -> None:
        if old is not None and hasattr(old, "dispose"):
            old.dispose(close=False)      # let in-flight work finish, then close

Most calls take the fast path: the secret is unchanged, the comparison fails, and the existing object is returned. Only a genuine rotation triggers a rebuild, so the pool is rebuilt on the rotation schedule rather than the cache schedule — which for a sixty-day rotation and a five-minute TTL is a factor of about seventeen thousand. Put differently, a pool that would otherwise be discarded and rebuilt every five minutes now survives for two months, which is the difference between a warm pool and a permanently cold one.

Retiring the old object rather than closing it immediately is the detail that keeps a rotation invisible. dispose(close=False) on a SQLAlchemy engine abandons the pool without severing connections currently checked out, so a query already running finishes normally. Closing outright would abort those queries — turning a smooth credential change into a burst of errors, which is precisely the outcome the overlap window exists to prevent.

Not every consumer needs rebuilding. An HTTP client that sends a bearer token per request only needs the new token on the next call, so binding it to the secret is unnecessary complexity. The distinction is whether the credential is used at connection time or at request time: connection-time credentials need a rebuild, request-time credentials just need a fresh read.

Deployment parity: local to production

1. Local dev — test rotation by forcing a refresh and asserting the app reconnects.
2. CI — a test rotates a fake secret mid-run and verifies no request fails.
3. Staging — trigger a real rotation and watch for reconnection errors before production.
4. Production — rotation runs on schedule; alerts fire if a credential nears expiry un-rotated.

Steps two and three are testing different things and both are necessary. The CI test uses a fake store and verifies the application’s behaviour — that the cache refreshes, the pool rebuilds, and no request fails while it happens. It runs in seconds and catches regressions on the commit that introduces them.

The staging rehearsal uses the real store and verifies the system — that the rotation job has permission to create a credential, that the backend accepts it, that the overlap actually lasts as long as the documentation claims. Those are configuration facts no unit test can reach, and they are exactly what breaks after an unrelated infrastructure change.

Local rotation testing is worth the small effort of making the fetch function injectable. If a developer can force a rotation with a keystroke and watch their service reconnect, the behaviour stops being theoretical, and reconnection bugs are found by whoever wrote the consumer rather than by staging a month later.

Production’s role in this list is different from the other three: it is not a test but a standing observation. The credential-age check, the failure alarm, and the refresh counter run continuously and answer the question “is rotation still working?” without anyone asking. That matters because rotation is the rare mechanism whose absence produces no symptom — a service running on a credential that has not changed in a year behaves identically to one rotating perfectly, right up until an audit or an incident reveals the difference.

Rotating credentials that have no overlap

The overlap window is a property of the backend, not of your code, and some backends do not offer one. A third-party API that issues a single key per account, an SFTP server with one password, a legacy system with no notion of versions — for these, creating a new credential invalidates the old one immediately, and the whole pattern above does not apply.

Three approaches work, in descending order of preference. The best is to construct an overlap where the vendor does not provide one: many APIs allow multiple active keys even if their documentation emphasises one, so creating a second key, migrating, and deleting the first reproduces the overlap manually. Check for this before concluding it is impossible — it is available more often than teams assume, and a vendor that supports multiple keys but does not advertise it will usually confirm so if asked directly.

Where it genuinely is not, a drain and switch keeps disruption bounded. Stop accepting new work, let in-flight requests finish, rotate, and resume. For a queue consumer or a batch job this is nearly free; for a request-serving API it means a short period of shedding, which is acceptable at 3 a.m. on a ninety-day cadence and unacceptable weekly.

The last resort is a rolling restart with the new credential injected — coupling rotation to deployment, which is the thing the overlap pattern exists to avoid, but a legitimate answer when the backend forces it. Make the coupling explicit rather than accidental: a documented “rotating this credential requires a deploy” is manageable, whereas discovering it during an outage is not.

# secrets/rotation.py — bounded drain for a no-overlap credential
import contextlib

@contextlib.contextmanager
def drained(consumer, timeout: float = 30.0):
    """Stop new work, wait for in-flight work, then let the caller rotate."""
    consumer.pause()                       # stop accepting new units of work
    try:
        if not consumer.wait_idle(timeout):
            raise TimeoutError("consumer did not drain; aborting rotation")
        yield                              # caller performs the rotation here
    finally:
        consumer.resume()                  # always resume, even on failure

with drained(worker):
    new_key = vendor.rotate_api_key()      # old key dies the instant this returns
    secret_store.put("vendor/api-key", new_key)

The timeout and the finally are what make this safe to run unattended. A drain that never completes must abort rather than hang — leaving the consumer paused indefinitely is a worse outcome than a failed rotation — and the consumer must resume whether the rotation succeeded or threw. A rotation job that leaves a worker paused after an error causes an outage hours later that nobody connects to the rotation.

Security boundaries & guardrails

• Always provision-new-before-revoke-old; never revoke first.
• Keep the cache TTL strictly below the overlap window.
• Reconnect connection pools when the credential changes; do not leave old connections open.
• Wrap rotating credentials in SecretStr and unwrap only at the driver call.
• Alert on any secret whose age exceeds the rotation interval.
• Alarm on rotation failure as well, since a failed rotation breaks nothing and is therefore silent.

Provision-before-revoke sounds obvious and is worth stating because the tempting order is the other way round. During an incident — a credential believed compromised — the instinct is to revoke immediately, which is correct for a compromise and disastrous as a routine rotation strategy. Keeping the two paths distinct matters: routine rotation always provisions first, while incident response revokes first and accepts the outage, because at that point a brief outage is unambiguously the lesser of the two problems and nobody has to weigh it.

The age alert is the guardrail that catches the quiet failure. A rotation job that stops running breaks nothing today, so nothing pages anybody, and the credential simply stops changing. An alert on “this secret has not changed in more than N days” catches that regardless of why rotation stopped — a broken job, a disabled schedule, a permission removed during a cleanup — and it is the only check that covers all of those causes at once.

The SecretStr guardrail earns its place specifically because rotation multiplies the number of places a credential travels. A static password read once at startup passes through one code path; a rotating one passes through a fetch, a cache, a comparison, a rebuild, and a retirement, each of which is somewhere a debugging print might reasonably be added during an investigation. Masking by default means the investigation stays safe even when someone is moving quickly under pressure, which is exactly when the print gets added.

Reconnection deserves one more warning about ordering. Rebuilding a pool before the new credential is confirmed working leaves the service with connections that cannot authenticate and no way back — so where the store offers a test step, let it run first, and where it does not, verify the new credential with a single connection before replacing the pool. Provision, verify, switch, revoke: the same ordering as the store’s own rotation, applied inside the application.

Knowing rotation works without waiting for it to fail

Rotation is unusual among operational mechanisms in that its failure is silent by default. A deployment that fails goes red; a rotation that fails leaves a perfectly functional service running on an unchanged credential. Four signals, none expensive, turn that silence into something observable.

The first is secret age. Whatever store you use knows when each secret last changed, so a scheduled check that lists secrets and flags any older than its rotation interval catches every cause of stopped rotation at once — a disabled schedule, a failed job, a permission removed during a cleanup, a secret nobody ever enrolled.

# ops/secret_age_check.py — one query, catches every cause of stopped rotation
import datetime as dt
import boto3

MAX_AGE = dt.timedelta(days=90)

def stale_secrets() -> list[str]:
    sm = boto3.client("secretsmanager")
    now = dt.datetime.now(dt.timezone.utc)
    stale = []
    for page in sm.get_paginator("list_secrets").paginate():
        for s in page["SecretList"]:
            changed = s.get("LastChangedDate") or s["CreatedDate"]
            if now - changed > MAX_AGE:
                stale.append(f"{s['Name']} ({(now - changed).days}d)")
    return stale

The second is the rotation failure event itself. Stores emit one — a CloudWatch metric, an EventBridge event, an audit log entry — and wiring it to an alert catches a failure on the day it happens rather than at the age threshold weeks later. Age is the backstop; the failure event is the fast signal.

The third is the application’s own refresh counter. Counting how often each service replaced a cached secret tells you whether consumers are actually picking up rotations: a service whose count is stuck at zero across a rotation is not participating, whatever the store believes. This is the only signal that observes the consumer side, and it is the one that catches a service caching at import time.

The fourth is the staging rehearsal, which is the only signal that tests the whole chain before production does. Force a rotation, run traffic through it, assert no failures. Running it weekly turns rotation from something you hope works into something you know worked seven days ago.

Troubleshooting

• Requests fail during rotation — the overlap window is shorter than the cache TTL; widen the window or shorten the cache. See Zero-Downtime Secret Rotation in Python.
• Old credential still used after rotation — a connection pool was not recreated; reconnect on change.
• Rotation never triggers — the schedule or Lambda is misconfigured; assert rotation in staging.
• Credential expired before rotation — alerting is missing; add an age check.
• Only some instances fail — one deployment has a longer TTL or an older build; check that every consumer of the secret is on the same configuration.

That last symptom is a useful diagnostic in itself. Rotation failures that affect every instance simultaneously point at the store or the overlap window; failures that affect a subset point at the consumers — a worker fleet with a different TTL, a batch job someone wrote separately, a service that caches at import time. Splitting on that question first saves working through the wrong half of the system.

The “old credential still used” symptom has a second, less obvious cause worth ruling out: a consumer built from the secret at import time, where the comparison-and-rebuild logic exists but never runs because nothing calls it after startup. The tell is that the service picks up a rotation correctly after any restart and never otherwise, which looks like a caching problem and is a wiring problem — the binding was created but never consulted on the hot path.

Rotation problems also tend to arrive in pairs with unrelated changes, because the mechanism only exercises itself every few weeks. A dependency upgrade that changed a pool’s disposal semantics, a TTL adjusted during a performance investigation, an IAM cleanup that removed a permission the rotation job needed — none of them fails at the time, and all of them fail at the next rotation. When rotation breaks, the useful question is not “what changed today?” but “what changed since the last successful rotation?”, which is a much wider window than instinct suggests. Keeping a note of when the last successful rotation occurred — which the age check already knows — turns that from a guess into a date you can hand to git log.

Key takeaways

The invariant: provision before revoke, overlap longer than the cache, reconnect pools on change. Done right, rotation is a non-event your users never notice.

Two numbers and one habit carry all of it. The cache TTL must sit comfortably below the overlap window, because that inequality is what guarantees no process holds a revoked credential. Consumers must rebuild when the value changes rather than on a timer, because refreshing a cached secret does nothing for a connection that authenticated with the previous one. And rotation must be rehearsed rather than assumed — in CI against a fake store for the application’s behaviour, in staging against the real store for the configuration — because the failure that costs most is the silent one where rotation quietly stopped running and everyone continued to believe otherwise.

If you are starting from nothing, the order that gets you furthest fastest is: add the secret-age check first, because it tells you the truth about what you have today; then wire the rotation failure alarm, so tomorrow’s breakage is visible tomorrow; then make one credential rotate end to end with a rehearsal in staging; and only then extend the pattern to the rest. Each step is independently useful, and the first two cost an afternoon between them while answering the question most teams cannot answer honestly — which of our credentials are actually rotating?