AWS Secrets Manager Integration

Hard-coding a database password into a Kubernetes manifest puts it in cluster state, image history, and every backup. AWS Secrets Manager keeps the credential in a managed store, scoped by IAM, and rotates it on a schedule. This page integrates it into a Python service so the secret only ever exists in memory as a SecretStr.

AWS Secrets Manager is the native option in the enterprise secrets section for teams on AWS, feeding credentials into the same pydantic-settings model that validates the rest of the configuration.

The design has three moving parts and each solves a distinct problem. IAM decides who may read a secret, and scoping it per ARN is what stops a compromised service reading everything. The in-memory cache decides how often you call the API, which matters because the API is rate-limited and charged per call. And rotation decides how long a credential stays valid, which is the property that makes a leak self-limiting. Getting one right and the others wrong produces a system that looks managed but behaves like a hard-coded password.

Secure implementation

# secrets/asm.py
import json
import time
import boto3
from pydantic import SecretStr

_client = boto3.client("secretsmanager")
_cache: dict[str, tuple[float, dict]] = {}
TTL_SECONDS = 600  # shorter than the rotation interval


def get_secret(secret_id: str) -> dict[str, SecretStr]:
    now = time.monotonic()
    cached = _cache.get(secret_id)
    if cached and now - cached[0] < TTL_SECONDS:
        return cached[1]                       # serve from in-memory cache

    raw = _client.get_secret_value(SecretId=secret_id)["SecretString"]
    parsed = {k: SecretStr(v) for k, v in json.loads(raw).items()}  # mask every field
    _cache[secret_id] = (now, parsed)
    return parsed


creds = get_secret("prod/db")
password = creds["password"].get_secret_value()  # unwrap only at the point of use

The cache keeps API calls within rate limits; SecretStr keeps the value out of logs; the credential is unwrapped only at the moment it is handed to the database driver.

Wrapping every field rather than a hand-picked few is deliberate. A secret stored as JSON typically holds a username, host, port, and password together, and it is tempting to mask only the password. But the fields grow over time — someone adds a token, a connection string, a client secret — and a masking list maintained by hand falls behind. Masking everything and unwrapping at the point of use costs one get_secret_value() call per non-sensitive field and removes the possibility of the list going stale.

Unwrapping at the call site rather than at fetch time is the other half of that discipline. The moment a plain string is assigned to a variable that lives beyond a line or two, it can end up in a log line, an exception’s repr, or a serialised object. Keeping SecretStr all the way to the driver call means the plain value exists for the duration of one expression.

Configuration reference

The SecretId row hides a decision worth making explicitly: whether to reference secrets by name or by full ARN. Names are readable and portable across accounts, which sounds like an advantage until you realise it is also how a service in a development account can accidentally resolve a production secret name if its role happens to permit it. Full ARNs pin the account and region, so a copy-pasted configuration fails loudly rather than reading the wrong environment’s credential.

VersionStage is the row people meet during their first rotation. Secrets Manager keeps multiple versions and labels them: AWSCURRENT is the active value, AWSPREVIOUS is the one before it, and AWSPENDING exists mid-rotation while a new credential is being created and tested. A client that omits VersionStage gets AWSCURRENT, which is almost always right — the case for specifying it is a rollback, where reading AWSPREVIOUS deliberately is how you get back to the credential that was working.

Writing the IAM policy that actually scopes access

Resource: "*" is the default that gets a service working and the reason a single compromised container can read every credential in the account. Writing the policy properly takes a few extra lines and is the highest-value change in this entire integration.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOwnSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-*",
        "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/stripe-*"
      ]
    },
    {
      "Sid": "DecryptWithOwnKey",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/abcd1234-...",
      "Condition": {
        "StringEquals": {"kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"}
      }
    }
  ]
}

The trailing -* on each ARN is not laziness — Secrets Manager appends a random six-character suffix to every secret’s ARN, so an exact ARN breaks the moment a secret is deleted and recreated. The wildcard covers the suffix without widening the scope to other secrets, because the prefix up to that point is the full path.

The kms:ViaService condition on the second statement is the detail that turns a broad-looking KMS grant into a narrow one. Without it, the role can use that key to decrypt anything encrypted with it, anywhere; with it, the key can only be used through Secrets Manager, so the grant does exactly what its name suggests. Conditions like this are how you keep a necessary permission from becoming a general-purpose one.

Two things are deliberately absent. There is no secretsmanager:ListSecrets, because an application that knows which secret it needs never has to enumerate — and enumerating the account’s secrets is precisely what an attacker does first after gaining a foothold. And there is no PutSecretValue or UpdateSecret: the application reads, the rotation function writes, and separating those two roles means that a compromised application cannot overwrite a stored credential with one the attacker controls and then wait for other services to pick it up.

Deployment parity: local to production

1. Local dev — developers assume a read-only role via SSO and fetch a non-prod secret; no plaintext on disk.
2. CI — the pipeline uses an OIDC role scoped to test secrets only.
3. Staging/Production — the pod’s IAM role grants GetSecretValue on its own secret ARNs; rotation runs on the Secrets Manager schedule.

What all three have in common is that no long-lived AWS credential exists anywhere. SSO issues short-lived session credentials to developers; OIDC lets the CI platform exchange a signed job token for a role with no stored access key; and a pod’s role is delivered by the platform’s identity mechanism. If any step in that list involves an AWS_SECRET_ACCESS_KEY in a variable, the secret store has been placed behind a credential with exactly the properties it was adopted to eliminate.

The OIDC arrangement for CI is worth setting up even for a small pipeline. It replaces a stored access key — which must be rotated by someone, and generally is not — with a trust relationship that grants a role only to jobs from a specific repository and branch. That condition is enforced by AWS rather than by the CI platform, so a compromised pipeline in another repository cannot assume the role.

Local development deserves one more note, because it is where a well-designed setup most often springs a leak. The temptation is to fetch the production secret once, paste it into a .env, and get on with the work — which recreates on a laptop exactly the plaintext-on-disk situation the store was adopted to remove, with no audit trail and no expiry. Giving developers SSO access to a non-production secret with realistic-looking values costs nothing and removes the incentive entirely. If someone genuinely needs a production value to debug, that should be a deliberate, audited read rather than a file that stays on the machine for months.

The same principle applies to the shape of the secret. Storing a whole connection string means anyone who can read it for one purpose reads all of it; storing discrete fields lets a role that only needs the hostname get a secret containing just the hostname. Splitting is not always worth it, but it is worth considering for the credentials with the widest set of readers. The cost is more secrets to manage and more API calls to make; the benefit is that each reader’s IAM policy can name exactly what that reader needs, which is impossible when everything lives in one JSON blob behind a single ARN.

Feeding the secret into a settings model

The get_secret function returns a dictionary, which is fine for one credential and unpleasant once a service has several. Wiring Secrets Manager into pydantic-settings as a source means the secret’s fields arrive as typed, validated attributes alongside every other configuration value, and a missing field fails at startup rather than with a KeyError on first use.

# config/settings.py
from typing import Any
from pydantic import SecretStr
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict
from secrets.asm import get_secret

class SecretsManagerSource(PydanticBaseSettingsSource):
    """Reads one JSON secret and exposes its keys as fields."""

    def __init__(self, settings_cls, secret_id: str):
        super().__init__(settings_cls)
        self._data = get_secret(secret_id)          # already SecretStr-wrapped

    def get_field_value(self, field, field_name: str) -> tuple[Any, str, bool]:
        return self._data.get(field_name), field_name, False

    def __call__(self) -> dict[str, Any]:
        return {k: v for k, v in self._data.items() if v is not None}


class Settings(BaseSettings):
    model_config = SettingsConfigDict(extra="ignore")
    username: str
    password: SecretStr
    host: str
    port: int = 5432

    @classmethod
    def settings_customise_sources(cls, settings_cls, init_settings,
                                   env_settings, dotenv_settings, file_secret_settings):
        # env vars still win, so a local override works without touching AWS
        return (init_settings, env_settings,
                SecretsManagerSource(settings_cls, "prod/db"), file_secret_settings)

The source order is the important line. Placing the Secrets Manager source below environment variables preserves the precedence rule that runs through the whole configuration layer — an injected value always wins — which means a developer can override a single field locally without any AWS access, and an incident responder can pin a value without editing the stored secret.

It also makes types real. port: int = 5432 means the value arrives as an integer whatever the JSON contained, password: SecretStr is masked automatically, and a secret missing host fails during Settings() with a message naming the field. Compare that with creds["host"] failing somewhere deep in a connection factory, and the difference is a startup error against a runtime one.

The one thing to watch is that constructing Settings() now makes a network call, so it must not happen at import time in a context that cannot tolerate one — a test collection run, for instance. Constructing it in an application factory, once, and passing it explicitly keeps the network call where you can see it, and it makes the object trivially replaceable in tests — a test constructs Settings(username="u", password="p", host="h") with init arguments, which sit above the Secrets Manager source in the order, so nothing reaches AWS at all. That is the same property that makes init arguments valuable for pinning any field: the override is scoped to the object rather than to the process, and it cannot leak into an unrelated test.

Security boundaries & guardrails

• IAM policy lists explicit secret ARNs — never Resource: "*".
• Cache in memory only; never write the fetched secret to disk.
• Wrap every field in SecretStr and unwrap at the call site only.
• Set the cache TTL below the rotation interval so stale credentials expire quickly.
• Enable CloudTrail on GetSecretValue to audit access.
• Attach a resource policy to high-value secrets so access requires both an IAM grant and the secret’s own consent.

The resource-policy item is the one that scales best in a large account. An IAM policy can be widened by anyone with permission to edit roles, and in a big organisation that is a lot of people; a resource policy on the secret itself is edited by whoever owns the secret. Requiring both means widening access to a production credential takes a change in two places controlled by two teams, which is a meaningful barrier compared with a single over-broad policy nobody reviewed.

CloudTrail deserves more than a checkbox. GetSecretValue events record the principal, time, and secret, which turns “who could have read this?” into “who did read this, and when?” — and an alert on reads by an unexpected principal is one of the few detections in this area that catches a compromise in progress rather than afterwards. Access patterns are highly regular: a service reads its own secret every TTL and nothing else, so an anomaly stands out clearly.

Two alarms are worth building on that regularity. The first fires on a GetSecretValue by any principal outside a known allow-list — the service’s role, the rotation function’s role, and a break-glass role used by humans. Because the expected set is tiny and changes rarely, that alarm is quiet by default and meaningful when it fires. The second fires on a sudden increase in read volume for a secret, which is what exfiltration-by-enumeration looks like from CloudTrail’s perspective and what a caching bug looks like too — either is worth knowing about.

The break-glass role is the piece people leave out and then improvise during an incident. Define it in advance: a role a human can assume with strong authentication, granting read access to production secrets, that always alarms when used. Having it defined means nobody has to widen an application’s policy at 3 a.m. to see a value, which is how over-broad permissions get created in a hurry and then stay in place for years because nobody remembers they were temporary. A break-glass role that alarms loudly is the safe version of the same capability.

Surviving a Secrets Manager outage

Once a service cannot start without reading a secret, the secret store is on its critical path. That is a reasonable trade — the alternative is a credential sitting somewhere less protected — but it deserves a deliberate design rather than the behaviour you happen to get from an unconfigured boto3 client.

boto3’s defaults are not tuned for this. The standard retry mode makes a small number of attempts with a connect timeout measured in tens of seconds, so a transient API problem can stall a startup for minutes while the orchestrator’s readiness probe fails and the pod is killed and restarted into the same stall. Configuring the client explicitly turns that into a bounded, predictable failure.

# secrets/asm.py — bounded, jittered retries with explicit timeouts
import boto3
from botocore.config import Config

_config = Config(
    connect_timeout=3,
    read_timeout=5,
    retries={"max_attempts": 4, "mode": "adaptive"},   # adaptive adds backoff + jitter
)
_client = boto3.client("secretsmanager", config=_config)

mode="adaptive" is the one worth knowing about: it applies client-side rate limiting in response to throttling, which is exactly the behaviour you want when a fleet restarts simultaneously after a deployment and every instance reaches for the same secret at once. The standard mode retries; the adaptive mode retries and slows down, which is what stops a fleet from turning a brief throttle into a sustained one.

The complementary decision is what a running service does when a refresh fails. Serving from a stale cache past its TTL is usually better than failing — the credential is probably still valid, since rotation is infrequent relative to a brief API outage — so a refresh failure should log a warning and extend the cached value rather than raise. Failing only when the cached value is genuinely too old to trust turns a Secrets Manager outage into a logged degradation instead of an incident.

def get_secret(secret_id: str) -> dict[str, SecretStr]:
    now = time.monotonic()
    cached = _cache.get(secret_id)
    if cached and now - cached[0] < TTL_SECONDS:
        return cached[1]
    try:
        raw = _client.get_secret_value(SecretId=secret_id)["SecretString"]
    except Exception:
        if cached and now - cached[0] < TTL_SECONDS * 4:   # grace period
            log.warning("secrets: refresh failed; serving cached value")
            return cached[1]
        raise                                              # too stale to trust
    ...

The grace multiplier is a judgement call and should be set against the rotation interval, not against the TTL. If secrets rotate daily and the TTL is ten minutes, a forty-minute grace period is far inside the safe window; if rotation is hourly, it is not. Writing the relationship down in a comment next to the constant saves the next person from having to reconstruct the reasoning.

Troubleshooting

• AccessDeniedException — the role lacks GetSecretValue on that ARN; scope the policy to it.
• Stale credential after rotation — the cache TTL is longer than the rotation interval; shorten it. See Caching AWS Secrets in Memory.
• Throttling / ThrottlingException — too many GetSecretValue calls; the in-memory cache is missing or disabled.
• Secret value in logs — a field is a plain string; wrap in SecretStr.
• DecryptionFailure — the role can read the secret but not use its KMS key; grant kms:Decrypt on the customer-managed key.

That last one catches nearly everyone who moves from the default AWS-managed key to a customer-managed one. The IAM policy grants GetSecretValue and looks complete, but decryption is a separate authorisation against the KMS key, so the call fails with an error that says nothing about IAM. Two grants are needed — one on the secret, one on the key — and if the key lives in another account, its key policy must permit the role as well.

Cross-account access has a third requirement that catches teams building a shared secrets account. Reading a secret from another account needs the IAM grant on the role, a resource policy on the secret naming the calling principal, and a KMS key policy allowing that principal — and a customer-managed key is mandatory, because the AWS-managed key cannot be shared across accounts at all. When a cross-account read fails, working through those three in order is faster than guessing, and the error message rarely distinguishes them.

The AccessDeniedException message itself is worth reading carefully rather than skimming. It names the principal, the action, and the resource, which usually makes the mismatch obvious — a role name that is not the one you expected means the pod is running with the wrong service account, and a resource ARN with a different region or account means the secret reference itself is wrong, not the policy.

Throttling has a characteristic shape worth recognising too: it appears under load rather than at deploy time, and it appears fleet-wide at once. That is the signature of a cache that is per-request rather than per-process — for example, a get_secret call inside a request handler that constructs a new client each time — so the request rate and the API call rate are the same number.

The module-level _cache in the implementation above is per-process, which is the right granularity for most deployments and worth understanding precisely. Each worker process in a Gunicorn or Uvicorn deployment keeps its own copy, so the API call rate is the process count divided by the TTL — thirty-two processes on a ten-minute TTL is roughly three calls a minute, comfortably within limits. It also means a fork-based server must not populate the cache before forking if the secret might rotate, since every child would inherit the same expiry timestamp and refresh in lockstep.

The other consequence of a per-process cache is that a secret rotated now takes up to one TTL to reach every process, and different processes will briefly hold different credentials. That is harmless as long as both versions are valid, which is exactly what the AWSCURRENT/AWSPREVIOUS overlap guarantees — and it is the reason that overlap exists rather than an accident of the API design.

Key takeaways

The invariant: the credential lives in Secrets Manager, reaches the app only as a TTL-cached SecretStr, and is accessed through an IAM role scoped to its exact ARN. Rotation is the store’s job, not a redeploy.

Two numbers determine whether that invariant holds in practice. The cache TTL must be comfortably shorter than the rotation interval, or the application will keep using a credential the store has already replaced — and the overlap between AWSCURRENT and AWSPREVIOUS is what saves you when they are close rather than a licence to ignore the relationship. And the IAM policy’s Resource list must name the exact ARNs the service uses, because that list is the only thing standing between a compromised service and every other secret in the account. Get those two right, wrap everything in SecretStr, and the remaining work is operational: watch CloudTrail for unexpected principals, and keep the KMS grant in step with the secret grant.

Everything else on this page is refinement worth adding as the service matters more: the settings-model source so secrets arrive typed and validated, the bounded retry configuration and grace period so a store outage degrades rather than fails, and the resource policy plus break-glass role so widening production access is a deliberate act somebody reviews. None of those is required to get a working integration, and each removes a specific failure that a working integration will eventually meet.