Caching AWS secrets in memory securely

Calling get_secret_value on every request burns through Secrets Manager rate limits and adds latency; caching it forever means you keep using a credential after it rotates. The fix is a thread-safe, TTL-bound, in-memory cache. This page extends AWS Secrets Manager Integration.

The TTL is doing two jobs at once, which is why picking it feels harder than it should. Shorter means fewer stale-credential seconds after a rotation; longer means fewer API calls and less exposure to a Secrets Manager hiccup. Both pressures are real, and the resolution is that the upper bound comes from the rotation interval while the lower bound comes from the request rate and process count. Anywhere inside that range is defensible; outside it, one of the two failure modes is guaranteed.

Problem 1: a fetch on every request

# ANTI-PATTERN: hammers the API and gets throttled
def handler(event):
    secret = boto3.client("secretsmanager").get_secret_value(SecretId="prod/db")
    return connect(secret["SecretString"])   # one API call per invocation

Under load this hits ThrottlingException and adds a network round-trip to every request. There is a second, quieter cost in that snippet: boto3.client(...) is constructed inside the handler, so every invocation also builds a new client, resolves credentials, and establishes a new TLS connection. That is often more latency than the API call itself, and it is pure waste — boto3 clients are thread-safe, designed to be built once and shared, and constructing one per invocation buys nothing at all.

The failure mode is characteristic of load-dependent bugs. It passes every test, works fine in staging, and appears only when concurrency crosses the throttle threshold — which is to say, during a traffic peak, when the resulting retries and latency are least affordable.

Problem 2: a cache with no expiry

# ANTI-PATTERN: never refreshes, so rotation breaks the app
_SECRET = boto3.client("secretsmanager").get_secret_value(SecretId="prod/db")  # cached forever

Once the secret rotates, this value is stale and every connection fails until a redeploy. Worse, the failure arrives on the rotation schedule rather than on the deployment schedule, so it lands at whatever hour the rotation was configured for — often overnight, when nobody chose that time deliberately.

Module-level evaluation adds its own problem: the API call happens at import time, which means a Secrets Manager outage prevents the module from importing at all. Test collection, a management command, a --help invocation — all of them now make a network call. Deferring the fetch into a function makes the call happen when it is needed and nowhere else, and it makes the failure recoverable: a function that raises can be retried on the next request, whereas an import that fails takes the whole process with it and leaves the orchestrator restarting into the same error until the outage ends.

Secure implementation

# secrets/cache.py
import json
import threading
import time
import boto3
from pydantic import SecretStr

_client = boto3.client("secretsmanager")
_lock = threading.Lock()
_cache: dict[str, tuple[float, dict]] = {}
TTL = 600                                    # seconds; keep below rotation interval

def get_secret(secret_id: str) -> dict[str, SecretStr]:
    now = time.monotonic()
    with _lock:                              # thread-safe: one fetch under contention
        cached = _cache.get(secret_id)
        if cached and now - cached[0] < TTL:
            return cached[1]
        raw = _client.get_secret_value(SecretId=secret_id)["SecretString"]
        parsed = {k: SecretStr(v) for k, v in json.loads(raw).items()}
        _cache[secret_id] = (now, parsed)
        return parsed

One lock means concurrent requests trigger a single fetch, not a thundering herd. The TTL guarantees the app re-fetches after rotation; SecretStr keeps values out of logs.

The lock placement is the part worth studying, because the obvious alternative is wrong in an interesting way. Checking the cache outside the lock and only locking for the fetch looks like an optimisation — readers never block — but it lets N threads all observe an expired entry and all proceed to fetch, which is the stampede the lock was added to prevent. Holding the lock across the check and the fetch means the first thread fetches while the others wait and then find a fresh entry.

The cost is that every cache hit acquires a lock, which sounds expensive and is not: an uncontended threading.Lock acquisition is a fraction of a microsecond, against an API call measured in tens of milliseconds. If profiling ever shows the lock as a bottleneck, the answer is a per-secret lock rather than a global one, not removing the lock.

SecretStr wrapping happens inside the lock too, which is worth noticing because it means the parsed dictionary stored in the cache is already masked. Nothing downstream has to remember to wrap anything, and a future caller who logs the whole returned dictionary during debugging prints masks rather than credentials. Doing the wrapping at the single point where the value enters the process is the same principle as unwrapping at the single point where it leaves — both put the boundary somewhere a reviewer can find it.

Refreshing ahead of expiry

The implementation above refreshes lazily: the request that arrives after expiry pays for the fetch. That is simple and correct, and it has one visible cost — a latency spike on one request every TTL, which shows up as a periodic bump in the tail of your latency distribution.

Refresh-ahead removes it. Serve the cached value immediately once past a soft threshold, and refresh in the background so the next caller gets a fresh entry with no wait.

# secrets/cache.py — serve stale briefly, refresh in the background
import concurrent.futures

_pool = concurrent.futures.ThreadPoolExecutor(max_workers=1)
_refreshing: set[str] = set()
SOFT_TTL = 400          # start refreshing here
HARD_TTL = 600          # block and fetch here

def get_secret(secret_id: str) -> dict[str, SecretStr]:
    now = time.monotonic()
    with _lock:
        cached = _cache.get(secret_id)
        age = now - cached[0] if cached else None
        if cached and age < SOFT_TTL:
            return cached[1]                       # fresh
        if cached and age < HARD_TTL:
            if secret_id not in _refreshing:       # one background refresh at a time
                _refreshing.add(secret_id)
                _pool.submit(_refresh, secret_id)
            return cached[1]                       # serve slightly stale, no wait
        return _fetch_locked(secret_id)            # too old: block and fetch

The single-worker pool and the _refreshing set together guarantee that at most one background refresh is in flight per secret, so this cannot become its own stampede. HARD_TTL remains the real guarantee — past it, callers block rather than receiving a value that may predate a rotation.

Whether this is worth the extra machinery depends on your latency budget. For a background worker, no: the lazy version is simpler and nobody notices a fetch. For a latency-sensitive API where a hundred-millisecond spike shows in the p99, yes — and the extra code is contained in one function that is straightforward to test with a fake clock.

Gotchas & version-specific behaviour

• Use time.monotonic(), not time.time(), so a clock adjustment cannot extend the TTL.
• The cache lives in memory only — never pickle it to disk or a shared cache.
• In multi-process servers (gunicorn), each worker has its own cache; size the TTL accordingly.
• Set the TTL strictly below the Secrets Manager rotation interval so stale values expire fast.
• Do not put secrets in a shared Redis or Memcached “for efficiency” — that moves a credential into a store with different access controls and usually no encryption at rest.
• A SecretStr in the cache still holds the plaintext in process memory; the mask protects logs and reprs, not a memory dump.

The last point is worth being precise about, because SecretStr is sometimes credited with more than it does. It prevents accidental disclosure through repr(), str(), logging, and most serialisation — which covers the overwhelming majority of real leaks. It does not encrypt anything, and anyone who can read the process’s memory can read the value. Attempting to “scrub” secrets from memory in Python is largely futile in any case: strings are immutable, copies are made freely, and the garbage collector decides when that memory is reused. Spend the effort on the code path instead, where the mask actually prevents things.

Caching across processes and Lambda invocations

The cache above is per-process, which suits a long-running server and behaves differently everywhere else. Knowing which of the three common shapes you are in decides whether the TTL you chose actually means anything.

A threaded server — Gunicorn with sync or threaded workers, Uvicorn with a single process — is the straightforward case. One cache per worker process, so the API call rate is the worker count divided by the TTL. Sixteen workers on a ten-minute TTL is under two calls a minute, which is nowhere near any limit.

A forking server needs one extra thought. If the cache is populated before the fork — by module-level code that runs during application import — every child inherits the same entry and the same expiry timestamp, so all of them expire in the same second and all fetch at once. Populating lazily, after the fork, staggers them naturally because each child’s first request arrives at a different moment.

Lambda is the case where intuition misleads. The container is reused across invocations, so a module-level cache does persist — which is why caching helps at all — but the number of concurrent containers scales with traffic, and each one fetches on its first invocation. A burst that spins up two hundred containers produces two hundred GetSecretValue calls in a few seconds regardless of your TTL, because none of them has a warm cache. The mitigations are the Secrets Manager Lambda extension, which caches outside your process, or provisioned concurrency, which keeps containers warm.

# lambda_handler.py — module scope persists across invocations in a warm container
from secrets.cache import get_secret          # cache lives in the module

def handler(event, context):
    creds = get_secret("prod/db")             # API call only on a cold start
    return do_work(creds)

The rule that generalises across all three is to reason about the number of caches, not the number of requests. API calls per minute is roughly the cache count divided by the TTL in minutes, plus one per cold start. Write that number down before choosing a TTL, and the choice stops being guesswork and becomes arithmetic anyone can check.

Production parity checklist

• TTL is shorter than the rotation interval.
• Cache access is guarded by a lock; no duplicate concurrent fetches.
• Values wrapped in SecretStr, unwrapped only at the driver call.
• IAM scoped to GetSecretValue on the exact ARN.
• No secret is ever written to disk or a shared store.
• The boto3 client is module-level and reused, not constructed per call.

Testing this is easy once the clock is injectable. Pass a now callable into get_secret (defaulting to time.monotonic) and a test can advance time arbitrarily, assert that a second call within the TTL makes no API call, assert that a call past the TTL makes exactly one, and — with a barrier and a handful of threads — assert that concurrent expiry produces exactly one fetch. Three tests, no network, and they cover every behaviour this page argues for.

The shared-store bullet deserves a sentence of justification, because putting secrets in Redis is a suggestion that comes up in most teams eventually and sounds efficient. The objection is not performance, it is that a shared cache has entirely different access controls: the IAM policy that carefully scopes GetSecretValue to one ARN says nothing about who can read a Redis key, and anything with network access to that instance now holds the credential. A per-process cache inherits the process’s isolation for free, which is precisely the property that makes it the right place for a secret.

Key takeaways

A locked, monotonic-based TTL cache eliminates throttling while staying rotation-aware. The two details that make it correct rather than approximately correct are holding the lock across the check as well as the fetch, so an expiry cannot stampede, and using a monotonic clock so a time adjustment cannot silently extend the window. Add refresh-ahead only if a periodic latency spike actually matters to you; add a hard TTL regardless, because it is the bound that keeps a pre-rotation credential from being served indefinitely. And before settling on a number, count the caches your deployment actually creates — a value that is comfortable for sixteen server workers can be badly wrong for a Lambda that scales to hundreds of concurrent containers, where cold starts dominate and the TTL barely participates. Pair it with the rotation patterns so the app picks up new credentials automatically.