Rotate RDS credentials with AWS Secrets Manager

AWS Secrets Manager has turnkey rotation for RDS — it provisions a new password, updates the database, and stores the new value. Your Python app’s only job is to pick up the change without a failed query. This page handles the app side, extending AWS Secrets Manager Integration.

What makes this tractable is that rotation is not a cutover. Secrets Manager runs a four-step Lambda — create, set, test, finish — and only the last step moves the AWSCURRENT label. During the whole process the previous credential keeps working, and after it, the old value remains valid as AWSPREVIOUS until the next rotation. That overlap is the entire reason an application with a modest cache can ride through a rotation without noticing.

The four rotation steps and where the label moves Create generates a new password, set applies it to the database, test verifies it works, and only finish moves the AWSCURRENT label to the new version. the old credential works throughout all four steps 1 · createSecret new password → AWSPENDING 2 · setSecret applied in the database 3 · testSecret connect and verify 4 · finishSecret AWSCURRENT moves here clients still read the old value — nothing has changed for them yet clients pick up the new A failure in steps 1–3 aborts the rotation with the old credential untouched.
Only the fourth step is visible to clients — which is why a failed rotation is safe rather than an outage.

Problem 1: caching the password forever

# ANTI-PATTERN: connection built once with a password that will rotate
ENGINE = create_engine(get_secret("prod/rds")["password"])   # stale after rotation

When Secrets Manager rotates the password, every connection in this pool fails. The module-level engine makes it worse than a stale value: the credential is captured at import time, so the only way to pick up a new one is to restart the process — which means enabling rotation on a service written this way converts a security improvement into a scheduled outage.

The tell that this pattern is present is that the service is fine for weeks and fails at a fixed interval matching the rotation schedule. Because a restart fixes it immediately, the natural response is to restart and move on, which resets the clock and defers the diagnosis until the next rotation, by which point whoever restarted it has moved on to something else and the connection between the two incidents is easy to miss entirely.

Problem 2: fetching on every query

# ANTI-PATTERN: an ASM call per query — throttled and slow
def query():
    pw = get_secret("prod/rds")["password"]    # API call per query

This hammers the Secrets Manager API and adds latency to every database call. It is the over-correction from the first problem, and the two together define the range: never re-reading is broken by rotation, always re-reading is broken by rate limits, and the workable answer is a cache with a TTL shorter than the rotation interval.

It is also worth noticing that re-reading the secret does not, by itself, do anything. The password only takes effect when a new connection is opened with it — an existing pooled connection was authenticated when it was created and is unaffected by what the application knows now. That is why the fix has two halves: refresh the credential, and make sure the pool rebuilds connections with it.

Refreshing the secret is only half the fix A refreshed password has no effect on connections already open in the pool, so the pool must also discard and re-establish connections. two things must change, not one the cached secret refreshed when the TTL expires now holds the new password on its own: changes nothing the connection pool existing connections keep the old credential they authenticated with pre-ping discards the dead ones
A pooled connection authenticated once; knowing the new password changes nothing until connections are rebuilt.

Secure implementation

# db/rds.py
import json, time, boto3
from pydantic import SecretStr
from sqlalchemy import create_engine

_client = boto3.client("secretsmanager")
_cache: dict | None = None
_loaded = 0.0
TTL = 300                                   # below the rotation interval

def _creds() -> dict:
    global _cache, _loaded
    if _cache is None or time.monotonic() - _loaded > TTL:
        raw = _client.get_secret_value(SecretId="prod/rds")["SecretString"]
        _cache, _loaded = json.loads(raw), time.monotonic()
    return _cache

def make_engine():
    c = _creds()
    pw = SecretStr(c["password"]).get_secret_value()
    return create_engine(f"postgresql://{c['username']}:{pw}@{c['host']}/{c['dbname']}",
                         pool_pre_ping=True)   # drops dead connections after rotation

The TTL cache picks up the rotated password within the window; pool_pre_ping=True discards connections invalidated by the rotation so SQLAlchemy reconnects with the new credential.

pool_pre_ping=True is the load-bearing option and worth understanding precisely. It issues a cheap liveness check before handing a pooled connection to application code, and discards any connection that fails it. After rotation, connections belonging to the old credential are typically closed by the server, so pre-ping catches them and SQLAlchemy transparently opens a replacement — using the credentials in the engine’s URL, which is precisely why the engine itself must be rebuilt rather than merely the cached secret being refreshed.

The username matters as much as the password. Secrets Manager’s RDS rotation offers a single-user strategy, which changes the password of one user in place, and an alternating-user strategy, which switches between two users. Reading username from the secret rather than hard-coding it is the thing that makes the alternating strategy work at all — and the alternating strategy is what makes rotation genuinely zero-downtime, because the previous user keeps working while clients catch up.

Notice too that the secret provides host and dbname. Taking all connection parameters from the secret rather than combining a secret password with a hard-coded host means a failover that changes the endpoint propagates the same way a rotation does, with no deployment involved.

The one parameter to keep outside the secret is the secret’s own identifier. SecretId="prod/rds" is a reference, not a credential, and it belongs in ordinary configuration where it can be set per environment — hard-coding it in the module means the same code cannot run against staging without an edit. Reading it from an environment variable keeps the module environment-agnostic while the secret continues to supply everything sensitive.

Single-user versus alternating-user rotation Single-user rotation changes one password in place leaving a brief window where old connections fail, while alternating-user rotation switches between two users so the previous one keeps working. single user one user, password changed in place simpler to set up old password stops working at once a stale cache means failed queries alternating users two users, used in turn needs a clone user with the same grants the previous user stays valid genuinely zero-downtime
Alternating users is what removes the hard cutover — and it only works if the client reads the username from the secret.

Proving it works before it matters

Rotation is a scheduled event that runs unattended, often overnight, and its failure mode is a service that cannot reach its database. That combination deserves a rehearsal rather than a hope, and Secrets Manager makes rehearsal easy: rotate_secret triggers the whole cycle on demand.

# tests/staging_rotation_check.py — run against staging, watching for failures
import boto3, time, threading

sm = boto3.client("secretsmanager")
errors: list[Exception] = []
stop = threading.Event()

def hammer(engine_factory):
    while not stop.is_set():
        try:
            with engine_factory().connect() as c:
                c.exec_driver_sql("SELECT 1")
        except Exception as exc:          # any failure during rotation is a finding
            errors.append(exc)
        time.sleep(0.5)

t = threading.Thread(target=hammer, args=(make_engine,), daemon=True)
t.start()
sm.rotate_secret(SecretId="staging/rds")   # force a rotation now
time.sleep(600)                            # cover more than one cache TTL
stop.set(); t.join()
assert not errors, f"{len(errors)} failures during rotation: {errors[:3]}"

The test runs longer than one TTL deliberately. A rotation that completes in seconds while the application still holds a cached credential for another four minutes is exactly the window where a misconfigured cache shows itself, and a shorter test would report success while proving nothing, because the application would still be serving happily from a credential nothing had yet asked it to replace.

Running this in staging on a schedule — weekly, say — catches the regressions that matter: a TTL raised above the rotation interval during a performance investigation, pool_pre_ping disabled by someone reducing connection overhead, or a hard-coded username reintroduced during a refactor. Each is an innocuous-looking change that quietly re-arms the failure this page exists to prevent.

The rotation Lambda’s own permissions are worth verifying at the same time. It needs to reach the database, which in a VPC means correct subnet and security-group configuration, and it needs kms:Decrypt if the secret uses a customer-managed key. A rotation that fails at setSecret leaves AWSCURRENT untouched, so nothing breaks — but rotation silently stops happening, and a credential you believe is rotating monthly is in fact as static as one that was never enrolled in rotation at all — with the added disadvantage that everyone assumes otherwise.

What a staging rotation rehearsal catches Forcing a rotation while querying continuously catches a TTL that exceeds the rotation interval, disabled pre-ping, a hard-coded username, and a rotation Lambda that cannot reach the database. each of these looks harmless in review TTL raised above the rotation interval stale credential after every rotation pool_pre_ping disabled for performance dead connections handed to callers username hard-coded in a refactor alternating-user rotation breaks Lambda cannot reach the database rotation silently stops happening
The last row is the quietest failure: nothing breaks, and the credential you believe rotates monthly never changes.

Rebuilding the engine, not just reading the secret

make_engine() returns a new engine every call, which is correct for illustration and wasteful in practice — building an engine per request discards the pool entirely and defeats the point of pooling. The production shape keeps one engine and replaces it only when the credential behind it changes.

# db/rds.py — one engine, replaced when the credential actually changes
import threading

_engine = None
_engine_key: tuple[str, str] | None = None
_engine_lock = threading.Lock()

def engine():
    c = _creds()                                    # TTL-cached secret
    key = (c["username"], c["password"])            # identity of the credential
    global _engine, _engine_key
    with _engine_lock:
        if _engine is None or key != _engine_key:
            old = _engine
            _engine = create_engine(
                f"postgresql://{c['username']}:{c['password']}@{c['host']}/{c['dbname']}",
                pool_pre_ping=True,
                pool_recycle=1800,
            )
            _engine_key = key
            if old is not None:
                old.dispose(close=False)            # let in-flight queries finish
    return _engine

Keying on the credential rather than on a timer is what makes this efficient. A cache refresh that returns the same password — which is the overwhelmingly common case, since rotation is far less frequent than the TTL — leaves the engine and its warm pool untouched. Only an actual change triggers a rebuild, so the pool is rebuilt on the rotation schedule rather than on the cache schedule.

dispose(close=False) is the detail worth knowing. It abandons the old pool without severing connections that are currently checked out, so a query in flight completes normally and its connection is closed when returned rather than yanked mid-statement. Calling plain dispose() would close everything immediately, which turns a smooth rotation into a burst of errors — the exact outcome all of this is meant to avoid.

pool_recycle is a useful companion setting for a different reason. It bounds how long any connection lives regardless of credentials, which handles the database or an intervening proxy closing idle connections. Setting it below whatever idle timeout sits between the application and the database avoids a separate class of “connection was closed unexpectedly” errors that people often misattribute to rotation.

Rebuilding on credential change rather than on a timer Most cache refreshes return the same password and leave the warm pool intact, while only an actual credential change triggers an engine rebuild. refresh often, rebuild rarely refresh · same refresh · same refresh · same refresh · CHANGED refresh · same pool stays warm — no rebuild, no reconnection cost rebuild once warm again dispose(close=False) abandons the old pool without cutting off queries already running.
Keying the engine on the credential means the pool is rebuilt on the rotation schedule, not the cache schedule.

Gotchas & version-specific behaviour

  • Secrets Manager RDS rotation uses two AWS-managed users alternately — both work during the overlap, so a short TTL never sees a hard cutover.
  • Set the cache TTL below the rotation interval.
  • pool_pre_ping=True (SQLAlchemy) is the cheapest way to drop connections killed by rotation.
  • Grant the rotation Lambda its own role; the app only needs GetSecretValue.
  • Alternating-user rotation requires a second database user with identical grants — create it before enabling rotation, or the first setSecret fails.
  • A rotation failure alarms but does not break anything, because AWSCURRENT only moves after a successful test — so alarm on rotation failure explicitly, or it goes unnoticed.

That last point is the one to wire up before you need it. Because a failed rotation is safe, nothing in the application reports a problem, and the only signal is a CloudWatch metric or an EventBridge event from Secrets Manager. Without an alarm on it, the first indication that rotation has been broken for months is a compliance review — by which time the credential has the same age profile as one that was never rotated at all.

Production parity checklist

  • Rotation is enabled on the RDS secret with an AWS-managed schedule.
  • The app caches credentials with a TTL below the rotation interval.
  • The connection pool drops stale connections (pool_pre_ping).
  • App IAM is scoped to GetSecretValue on the secret ARN.
  • A staging test forces rotation and asserts no failed queries.
  • An alarm fires on rotation failure, since a failed rotation is otherwise invisible.
  • The engine is keyed on the credential so the pool is rebuilt on rotation, not on every cache refresh.

Choosing the rotation interval is the one decision this checklist does not make for you, and it is less fraught than it looks. Anything from seven to ninety days is defensible for an RDS credential; the number matters far less than the fact that rotation demonstrably works, because a credential rotating every ninety days reliably is in much better shape than one nominally rotating weekly with a broken Lambda. Start at a longer interval, verify the rehearsal passes and the alarm is wired, and shorten it afterwards — the shortening is a one-field change once the mechanism is proven.

Key takeaways

Let Secrets Manager rotate the RDS password; on the app side a short-TTL cache plus pool_pre_ping makes the change invisible. The reason such a small amount of client code suffices is the design of rotation itself — four steps, with the label moving only after a successful test, and the previous credential remaining valid afterwards — so the application never faces a hard cutover as long as its cache is shorter than the rotation interval.

Take every connection parameter from the secret, not just the password, so an alternating-user rotation and an endpoint change both propagate the same way. Rehearse it in staging under continuous query load rather than assuming, and alarm on rotation failure, because the one failure mode that costs nothing today — rotation quietly not running — is the one that turns a managed credential back into a permanent one. Everything else on this page is code you write once; that alarm is the only part that has to keep working unattended, which is a good reason to put it in place before the rest. For the general pattern, see Automated Secret Rotation Patterns.

Back to AWS Secrets Manager Integration