Customise Settings Sources Order in Pydantic v2

pydantic-settings reads from several places in a fixed order, and most applications never need to change it. The two cases that do are worth knowing precisely: adding a source the library does not ship — a secret manager, a remote configuration service, a bespoke file format — and changing which of the built-in sources outranks the others. Both go through one hook, and both need the resulting order to be explicit and tested, because precedence is invisible until two sources disagree. This page covers them, extending the pydantic-settings fundamentals topic.

Problem 1: fetching from a store outside the model

# ANTI-PATTERN: a second configuration system with its own precedence
secrets = fetch_from_manager()                 # a dictionary from somewhere
settings = Settings(**secrets)                 # these now outrank everything

Passing fetched values as initialisation arguments puts them above every other source, including environment variables — which is rarely what anyone intends and is invisible in the model definition. An operator who sets a variable to override a stored value during an incident finds it silently ignored, and nothing in the code says why.

Registering the store as a source makes its position explicit and adjustable. The model definition then shows the complete precedence order in one place, and moving the store above or below the environment is a one-line change with an obvious meaning rather than a restructuring of how settings are constructed.

Problem 2: putting a remote store above the environment

# ANTI-PATTERN: an operator can no longer override anything during an incident
return (store_source, init_settings, env_settings, dotenv_settings)

It seems natural to give the authoritative store the highest priority — it is, after all, where the values are supposed to come from. The consequence appears during an incident: an engineer sets an environment variable to point a service at a healthy replica, restarts it, and nothing changes, because the store still supplies the old value and outranks the override.

Keep the environment above the store. The store is the normal source of truth and the environment is the escape hatch, which is the same relationship the twelve-factor ordering encodes for files. Overrides are then possible without a store write, which matters when the store is the thing that is broken, and the ordinary path is unchanged because nobody sets those variables day to day.

Problem 3: a source that silently returns nothing

# ANTI-PATTERN: an unreachable store looks exactly like an empty one
def __call__(self) -> dict[str, Any]:
    try:
        return fetch_from_manager()
    except Exception:
        return {}            # the model now falls through to defaults, quietly

Swallowing the error turns an outage into a service that boots with defaults — a local database URL, a disabled feature, an empty credential — and reports itself healthy. That is strictly worse than failing, because the failure at least stops the deploy while the wrong values may take hours to notice.

Let the exception propagate at startup. The process exits, the orchestrator reports a crash loop, and the previous version keeps serving. If a store genuinely is optional — a feature-flag service whose absence should mean “use the defaults” — say so explicitly with a comment and a narrow exception type, so the fallback is a decision somebody made rather than a catch-all somebody wrote while debugging.

Secure implementation

# sources.py — a custom source, positioned deliberately
from typing import Any
from pydantic.fields import FieldInfo
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict


class SecretStoreSource(PydanticBaseSettingsSource):
    """Fetches every value once, at construction, and serves them from a dictionary."""

    def __init__(self, settings_cls: type[BaseSettings], names: dict[str, str]) -> None:
        super().__init__(settings_cls)
        # 1. one fetch per model construction, not one per field
        self._values = fetch_from_manager(names) if names else {}

    def get_field_value(self, field: FieldInfo, field_name: str) -> tuple[Any, str, bool]:
        # 2. returns (value, key, is_complex); None means "this source has no opinion"
        return self._values.get(field_name), field_name, False

    def __call__(self) -> dict[str, Any]:
        return {k: v for k, v in self._values.items() if v is not None}


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="forbid")

    environment: str = "local"
    database_url: str
    api_token: str

    @classmethod
    def settings_customise_sources(
        cls, settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings
    ):
        # 3. the complete precedence, readable in one place, highest first
        return (
            init_settings,                    # tests and explicit construction
            env_settings,                     # 4. the operator's escape hatch stays on top
            SecretStoreSource(settings_cls, SECRET_NAMES),
            dotenv_settings,                  # local convenience
            file_secret_settings,             # mounted secrets directory
        )

The whole precedence contract is the returned tuple, which is why keeping it as a flat, commented list is worth more than any amount of documentation elsewhere. A reader answers “what beats what?” by reading five lines, and changing the order is a visible diff that a reviewer can reason about — as opposed to the alternative, where precedence is an emergent property of where various dictionaries get merged.

Fetching once in __init__ rather than per field matters for any remote source. The source object is constructed once per model construction, so a single fetch there costs one round trip; implementing get_field_value to fetch on demand would issue one request per field, which is both slow and a reliable way to meet a rate limit at startup.

Keeping the order honest

Precedence is invisible until two sources supply the same key, which means a reordering can ship and sit dormant for months before it changes anything. That makes it exactly the kind of behaviour worth pinning with tests: supply the same key from two sources, assert which one survives, and repeat for each adjacent pair in the order. Four or five short tests cover the whole contract, and they fail immediately if somebody reorders the tuple.

The second habit is to expose the order at runtime. A line in the startup log listing the source names in priority order — no values, just names — costs nothing and answers a question that is otherwise a code-reading exercise, including in a container where the code is not to hand. It pairs naturally with the per-field source reporting described on the source-debugging page.

The third is restraint. Every source added multiplies the combinations somebody has to reason about, and a model with six sources has fifteen adjacent pairs whose relative order somebody may one day depend on. Most applications need the defaults plus at most one custom source; if you find yourself adding a third, it is worth asking whether some of those values would be better resolved before construction — merged into a dictionary and passed as initialisation arguments — rather than as another layer in the precedence chain.

There is a simpler alternative that covers many of the cases people reach for this hook to solve: resolve the values before construction and pass them as initialisation arguments. Fetching from a store into a dictionary and handing it to the model gives you the store’s values at the highest priority with no custom class at all, which is fine when the store genuinely should outrank everything — a deployment pipeline that has already decided the values, for instance. The custom source earns its place specifically when you need the store below something else in the order, because that position cannot be expressed by passing arguments.

Gotchas and version-specific behaviour

• The default order is init arguments, environment, dotenv, secrets directory, defaults, highest first.
• settings_customise_sources returns a tuple whose order is the precedence; there is no other declaration of it.
• A source returning None for a field means “no opinion”, and the next source in the order is consulted for that field alone.
• Construct expensive sources once in __init__, not per field, or a remote source issues one request per field.
• An exception from a source propagates out of model construction, which is the right default at startup.
• Reordering sources is invisible until two of them supply the same key — pin the order with tests.
• Custom sources are constructed on every model construction, so a test that builds many models will hit a remote source many times unless it substitutes a fake.

Production parity checklist

• The source tuple is a flat, commented list, ordered highest priority first, that reads as the precedence contract for the whole application.
• Environment variables rank above any remote store, so an override is always possible.
• Remote sources fetch once per construction and let failures propagate at startup.
• Tests pin the relative order of each adjacent pair of sources, so a rearranged tuple fails immediately.
• The startup log names the active sources in priority order, without values, so the precedence is readable from a running container.
• Tests substitute a fake for any remote source so the suite makes no network calls and needs no credentials to run.

Key takeaways

settings_customise_sources is the one place precedence is declared, and the tuple it returns is the contract. Keeping it as a flat, commented list means anybody can answer “what beats what?” in five lines, and adding a store as a real source — rather than fetching values and passing them as initialisation arguments — keeps its position visible and adjustable.

Put the environment above any remote store so an override remains possible when the store is the problem, fetch once per construction rather than per field, and let a startup failure propagate rather than falling through to defaults. Then pin each adjacent pair with a short test, because a reordering changes nothing until two sources disagree and can otherwise ship unnoticed. Resist adding a third or fourth custom source; values that could be merged before construction usually should be. The default behaviour these customisations start from is on the pydantic-settings fundamentals topic page.