Load pydantic settings from AWS Parameter Store

pydantic-settings reads environment variables and .env files out of the box, but not AWS Parameter Store. The clean integration is a single custom settings source, not a pile of boto3 calls scattered through your app. This page builds it, extending Settings from AWS Parameter Store.

This page focuses on the fetch mechanics — the two boto3 details that make the difference between an integration that works at scale and one that is slow and insecure. The first is fetching in bulk: pulling a whole path prefix in one paginated get_parameters_by_path call rather than one get_parameter per field. The second is decryption: passing WithDecryption=True so SecureString parameters come back as plaintext rather than ciphertext. Get both right inside a custom settings source, and the parameters flow into the same validated model as everything else; get either wrong, and you have either a slow N-call fetch or a field holding an unusable encrypted blob.

Both details live inside the custom settings source, which is the right home for them. Rather than teaching every module that needs a parameter how to call SSM, decrypt, and coerce the result, you write that logic once in a PydanticBaseSettingsSource subclass and register it with settings_customise_sources. The source’s __call__ method does the batch fetch and returns a plain dict; the model does the validation, coercion, and secret masking. That separation is what keeps the AWS-specific code contained and the rest of the application oblivious to where its configuration came from — a module reads settings.database_url whether the value arrived from an environment variable or a decrypted SSM SecureString.

Problem 1: fetching parameters one by one

# ANTI-PATTERN: N API calls, no validation, scattered everywhere
db = boto3.client("ssm").get_parameter(Name="/myapp/database_url")["Parameter"]["Value"]
key = boto3.client("ssm").get_parameter(Name="/myapp/api_key")["Parameter"]["Value"]

Per-parameter calls are slow, untyped, and bypass the settings model entirely. Every get_parameter is a round trip to the SSM API, so a service with a dozen parameters makes a dozen calls at startup, and under any load that reconstructs settings, those calls multiply toward SSM’s rate limit. They are also scattered — each call lives wherever the value happens to be needed — so the value is untyped at each site and there is no single place that knows the full set of parameters the service reads. It is the boto3 equivalent of os.getenv sprinkled through the codebase, with the added and worse cost of a real network round trip on each and every read.

The fix is one batch call: get_parameters_by_path fetches every parameter under a prefix in a single (paginated) request, so a dozen parameters cost one API call instead of a dozen. Doing that inside a settings source also solves the typing and scattering problems at the same time — the batch result becomes a dict the model validates, so the parameters are typed, validated, and inventoried in one place.

The performance difference is not academic. SSM enforces a per-account request rate, and a service that makes one API call per parameter is spending that budget wastefully — a fleet of pods each making a dozen calls at startup can, during a deploy that cycles many pods at once, approach the throttling limit and start failing to read configuration for reasons that have nothing to do with the configuration itself. Collapsing those dozen calls into one batch fetch cuts the request count by an order of magnitude, and combined with a startup-time construction where each pod fetches once at boot rather than per request, it keeps SSM usage comfortably within the account’s limits even under aggressive rollouts that cycle many pods simultaneously. The batch call is both the faster and the more resilient choice.

Problem 2: forgetting decryption

# ANTI-PATTERN: SecureString returned still encrypted
ssm.get_parameters_by_path(Path="/myapp/")   # WithDecryption defaults to False

Without WithDecryption=True, SecureString parameters come back as ciphertext. WithDecryption defaults to False, so a call that omits it silently returns the encrypted KMS blob for every SecureString, and your database_url field receives an unusable, opaque string of ciphertext that no driver can parse. If that field has a validator it fails with a confusing error about a value that looks nothing like a URL; if it does not, the bad ciphertext value flows downstream unchecked and fails much later at connection time with an error that points at the driver, not the fetch. Either way the cause — a single missing boolean flag on the fetch call — is far from obvious to whoever is debugging the resulting error. Passing WithDecryption=True fixes it, but it also requires the reading identity to hold kms:Decrypt on the parameter’s key, so the flag and the IAM permission travel together — enable one without the other and the call fails with AccessDenied instead of returning ciphertext, which at least fails loudly rather than silently. Because WithDecryption defaults to off, the safe habit is to set it explicitly to True in every get_parameters_by_path call that might touch a SecureString, and to grant kms:Decrypt on exactly the key those parameters use — no broader. The two together are what turn an encrypted-at-rest secret into a usable plaintext value that the model can then re-protect as a SecretStr.

Secure implementation

# config/ssm_source.py
import boto3
from pydantic import SecretStr
from pydantic_settings import (
    BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict,
)

class SSMSource(PydanticBaseSettingsSource):
    def __call__(self) -> dict[str, object]:
        ssm = boto3.client("ssm")
        values: dict[str, object] = {}
        for page in ssm.get_paginator("get_parameters_by_path").paginate(
            Path="/myapp/", Recursive=True, WithDecryption=True,   # decrypt SecureStrings
        ):
            for p in page["Parameters"]:
                values[p["Name"].rsplit("/", 1)[-1].lower()] = p["Value"]
        return values

    def get_field_value(self, field, field_name):
        return None, field_name, False

class Settings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")
    database_url: str
    api_key: SecretStr

    @classmethod
    def settings_customise_sources(cls, settings_cls, init_settings,
                                   env_settings, dotenv_settings, file_secret_settings):
        return (init_settings, env_settings, SSMSource(settings_cls))  # env still wins

One paginated call fetches the whole prefix, decryption is explicit, and the model validates the result. Environment variables remain above SSM so local overrides still work.

Three details in the fetch matter. Recursive=True descends into sub-paths, so a nested layout like /myapp/cache/host is picked up rather than only the top-level parameters directly under the prefix. The get_paginator(...).paginate(...) loop is essential because get_parameters_by_path returns results a page at a time — a service with more than ten parameters would silently miss everything past the first page if you called the API directly instead of paginating through every page of results. And the name-to-field mapping — p["Name"].rsplit("/", 1)[-1].lower() — strips each parameter down to its last path segment and lower-cases it so a parameter at /myapp/database_url correctly populates the model’s database_url field. Those three lines are the whole difference between a source that fetches everything correctly and one that quietly drops parameters or mis-maps them.

The get_field_value method is a required part of the interface that this bulk-fetch source does not really use, which surprises people the first time. PydanticBaseSettingsSource was designed to support two styles: sources that resolve one field at a time (which implement get_field_value) and sources that return everything at once (which do the work in __call__). A path-prefix source is firmly the second kind — it grabs the whole prefix in one call — so get_field_value is satisfied by a minimal stub that returns (None, field_name, False). Once you understand that the real work happens in __call__ and that get_field_value is just an interface requirement for the per-field style you are not using here, the small piece of boilerplate stops looking arbitrary.

It is worth noting how cleanly this composes with the rest of pydantic-settings. Because the source returns a dict that merges into the model’s inputs by precedence, an SSM parameter and an environment variable of the same name simply resolve by the source order — the environment wins, as configured. There is no special case for “SSM values” versus “env values” in the model; they are all just inputs to validate. That uniformity is what lets you add or remove the SSM source without touching a single field definition, and what lets a value move from an env var in development to an SSM parameter in production with no code change at all — the field stays the same, only the source that happens to supply it changes between environments.

Gotchas & version-specific behaviour

• settings_customise_sources returns sources highest-priority first — put env_settings before the SSM source.
• get_parameters_by_path is paginated; use the paginator or you will miss parameters past the first page.
• WithDecryption=True requires kms:Decrypt on the parameter’s KMS key.
• Strip the path prefix to map /myapp/database_url to the field database_url.

The pagination gotcha is the one that fails silently and at the worst time. get_parameters_by_path returns at most ten parameters per page, so a service that starts with eight parameters works fine when you call the API directly — and then breaks the day someone adds the eleventh, which lands on a second page you never fetched. Using the paginator from the start (get_paginator(...).paginate(...)) handles every page transparently, so the source scales with your parameter count without a lurking cliff waiting at the eleventh parameter. The source-order gotcha is the other easy mistake: settings_customise_sources returns sources in priority order, highest first, so listing env_settings before your SSM source is what keeps environment variables winning — reverse them and SSM would override an operator’s env var, breaking local-override parity and making a developer unable to point their local process at a different database without editing Parameter Store. The name-mapping gotcha rounds out the set: because the source keys the dict by the parameter’s last path segment, a parameter must be named to match its field. If /myapp/db_url is stored but the field is database_url, the mapping produces db_url, which the model does not recognise — under extra="forbid" that is a hard error, which is actually helpful because it surfaces the mismatch at startup rather than leaving the field mysteriously unset.

Production parity checklist

• A single custom source, not scattered get_parameter calls.
• WithDecryption=True with kms:Decrypt granted.
• IAM scoped to the exact path prefix.
• Secret fields typed SecretStr; extra="forbid" set.
• Environment variables outrank SSM for local overrides.

The single-source item is the one that ties the rest together. Fetching through one custom source rather than scattered get_parameter calls is what makes the other four checks possible: it gives you one place to grant the IAM scope, one place to enable decryption, one place to type secrets as SecretStr, and one place to order it below the environment. Scatter the fetch and each of those becomes a per-call decision that will eventually be made inconsistently. The custom source is the boundary that keeps every Parameter Store interaction fetched in bulk, decrypted correctly, validated, masked, and correctly ordered — the same single-boundary discipline that governs environment variables, applied to a managed store. Once it exists, adding a new SSM-sourced setting is exactly as simple as adding an env-sourced one: declare the field on the model, store the parameter under the prefix, and the existing source picks it up on the next fetch. No new boto3 code, no new call site, no new place for the decryption or masking to be forgotten — just one more field on the one model and one more parameter under the prefix it already reads.

Key takeaways

A PydanticBaseSettingsSource subclass makes Parameter Store just another validated source under the environment. The two fetch mechanics that make it work are bulk fetching — one paginated get_parameters_by_path call instead of a get_parameter per field — and explicit decryption, WithDecryption=True paired with kms:Decrypt, so SecureString parameters arrive as plaintext. Wrap both of them in a single custom source, map each parameter’s name to its field name by stripping the path prefix, and the parameters flow into the very same typed, validated model as your environment variables do.

Everything else is the same discipline as any settings source: scope the IAM to the exact path, type secrets as SecretStr, keep extra="forbid" so the store and schema stay in sync, and order the source below the environment so local overrides still win. Get the fetch right once, in one place, and you never have to scatter another raw get_parameter call through the codebase again. To keep it fast at scale, add caching — see Cache Parameter Store Values to Reduce API Calls.