Core Configuration Patterns & File Formats

Every production incident that starts with “it worked on my machine” traces back to configuration that loaded differently in two places. This section is the map for loading configuration in Python deterministically: where values come from, which source wins when they collide, how to parse structured files without opening a code-execution hole, and how to validate the result before a single request is served.

Configuration bugs are disproportionately expensive for a specific reason: they hide. A logic error usually fails the same way every time and shows up in a test, but a configuration error depends on the environment, so it passes every test on a developer’s laptop and in CI, then fails only in the one place that is hardest to debug and most costly to break. The variable that is set locally but forgotten in production, the .env that overrides a rotated secret, the boolean flag that is a truthy string, the YAML value silently retyped from a number to a float — none of these are visible in the code alone, and all of them are invisible until the exact combination of environment and value lines up. The patterns in this section exist to drag those failures into the light: to make configuration load the same way everywhere, fail immediately when something is wrong, and prove itself correct at startup rather than surprising you under load. Get this layer boring and predictable and a whole category of 3 a.m. pages simply stops happening.

What this section covers

Configuration in Python is a layered problem: raw sources at the bottom, a precedence order in the middle, and a validated object at the top. The seven topics below map onto those layers — five on how values arrive and are resolved, two on what happens once they reach an application — and each has a dedicated page with runnable code.

The three-layer model is worth holding in mind as you read the rest of this section, because every topic slots into it. The bottom layer is sources: the raw, untyped places a value can come from — the process environment, a .env file, a YAML or TOML document, a command-line flag. None of these is trustworthy on its own; each is just bytes or strings that may or may not be present and may or may not be well-formed. The middle layer is resolution: a fixed precedence order decides which source wins when several define the same key, and a safe parser turns files into plain data without executing them. The top layer is the validated object: a single typed structure that every part of the application imports, built once, that either constructs cleanly or refuses to start. Bugs happen when code reaches across layers — reading os.environ directly in a request handler instead of the settings object, or trusting a parsed file without validating it — so the discipline throughout this section is to keep the layers separate and let each do its one job. The pages linked in the table above go deep on a single layer or a single source; this overview is the map that shows how they fit together.

Environment variables are the baseline

A 12-factor process reads its configuration from the environment. The problem is that os.environ returns strings only, and a missing key raises KeyError at the worst possible moment — deep inside a request handler rather than at startup. Read every value through a single typed accessor so the failure is explicit and early, and so there is exactly one place that knows how to turn a string into an int, a bool, or a list.

# config/env.py
import os

def require(key: str) -> str:
    try:
        return os.environ[key]
    except KeyError as exc:
        raise SystemExit(f"Missing required environment variable: {key}") from exc

DATABASE_URL = require("DATABASE_URL")  # fails fast at import, not mid-request

Reading at import time means a missing variable stops the process before it binds a port, so an orchestrator sees the container fail its health check immediately instead of serving 500s. That is the whole point of fail-fast configuration: turn a slow, mysterious runtime error into a fast, obvious startup error.

The reason environment variables are the baseline rather than merely one option is that they are the one configuration source every runtime already agrees on. A container image, a serverless function, a systemd unit, a CI job, and a laptop shell all expose the same KEY=value interface, so a process that reads its configuration from the environment is portable across all of them without carrying a file format or a client library. That universality is also why the 12-factor guidance puts config in the environment: it keeps the deploy artifact — the built image — identical across environments, with only the injected variables differing. The tax you pay for that portability is the one this section keeps returning to: everything arrives as a string, and a string is not yet configuration. Types, ranges, and formats have to be reimposed on the way in, which is precisely the job a typed accessor here, and a validated settings model in the validation section, exist to do.

Key rule: never call os.getenv with a silent default for a value the app cannot run without. A wrong default is more dangerous than a crash, because a crash is visible and a wrong default is not. The full typing rules — booleans, ints, lists — live in Environment Variables & os.environ.

Typing the strings the environment hands you

Every value in os.environ is a str, and the most dangerous consequence is that bool("false") is True — a non-empty string is truthy in Python, so a feature flag read naively is stuck on forever. Numbers have the same trap in reverse: os.environ["WORKERS"] is "4", not 4, and passing that string where an integer is expected either raises deep in a library or silently concatenates. Give each type its own small, total function, so the coercion rules live in exactly one place and every read is unambiguous.

# config/typing.py
import os

def env_bool(key: str, default: bool) -> bool:
    raw = os.environ.get(key)
    if raw is None:
        return default
    return raw.strip().lower() in {"1", "true", "yes", "on"}   # everything else is False

def env_int(key: str, default: int) -> int:
    raw = os.environ.get(key)
    return default if raw is None else int(raw)                # ValueError is intentional

def env_list(key: str, default: list[str] | None = None) -> list[str]:
    raw = os.environ.get(key)
    return [p.strip() for p in raw.split(",") if p.strip()] if raw else (default or [])

The boolean helper enumerates the truthy spellings explicitly rather than trusting bool(), the integer helper lets a malformed value raise ValueError instead of limping on with a wrong type, and the list helper splits and trims a comma-separated string. These three cover the overwhelming majority of real configuration; anything more structured belongs in a file, not an environment variable. In practice you graduate from hand-written helpers to pydantic-settings, which performs exactly this coercion — with range checks and clear error messages — from a declared type annotation.

The other half of “read it once” is when you read it. Module-level reads run at import, so a container that is missing DATABASE_URL dies during startup and an orchestrator’s readiness probe never goes green — far better than a pod that accepts traffic and then 500s on the first query. It also makes configuration trivially testable: a test sets the variable with monkeypatch.setenv before importing the module, or constructs the settings object directly with overrides, and asserts on the typed result. The rule that falls out of all of this is simple and worth repeating: read each variable in exactly one place, coerce it there, and never scatter os.environ.get calls with silent defaults through the codebase where they will inevitably drift. A codebase with a single, typed configuration boundary is one you can reason about; a codebase that reads the environment from twenty scattered call sites is one where no one can say with confidence what any given value will be at runtime.

.env files belong in development, never in git

A .env file makes local development ergonomic, but it must never overwrite a value the platform already injected — an IAM role, a Kubernetes secret, a CI variable. Load it into an isolated mapping and let the real environment win, so the same code behaves correctly on a laptop and in production.

# config/loader.py
from pathlib import Path
from dotenv import dotenv_values

# Read into an isolated dict; do NOT mutate os.environ blindly.
file_values = dotenv_values(Path(__file__).parent / ".env")

The reason to read into a separate dict rather than calling load_dotenv(override=True) is precedence: platform-injected values represent the real, audited environment, and a leftover .env on a shared box should never shadow them. Committing that file, meanwhile, is the single most common way secrets leak into version control.

Key rule: override=False is the safe default — platform-injected values must win over a developer’s .env. See .env File Management for the gitignore and pre-commit setup.

The .env file ecosystem

There is not one .env file but a small family, and knowing which member does what keeps secrets out of git while staying convenient. The real file, .env, holds live local values and is gitignored — it never enters version control. Beside it lives .env.example, which is committed: it lists every variable the app needs with dummy or placeholder values, so a new contributor can copy it to .env and fill in the blanks, and so code review can see when a new required variable is introduced. Some teams add .env.local for machine-specific overrides and .env.{environment} for shared non-secret defaults; whatever the layering, the invariant is that anything containing a real credential is ignored and anything committed is safe to publish.

# config/loader.py
from pathlib import Path
from dotenv import dotenv_values, find_dotenv

# find_dotenv walks up from the current file so the loader works from any CWD.
base = dotenv_values(find_dotenv(".env.example"))   # committed defaults / structure
local = dotenv_values(Path(find_dotenv(".env") or ".env"))
merged = {**base, **local}                           # local fills in over the example

Two functions from python-dotenv express two different intents. dotenv_values() returns a plain dict and touches nothing else, which is what you want when you are building a settings object and controlling precedence yourself. load_dotenv() mutates os.environ in place, and its override argument is the whole game: load_dotenv(override=False) — the default and the correct choice — fills only variables that are not already set, so a platform-injected value always wins, while override=True lets a stale file clobber production secrets. For a fully hands-off local experience, direnv loads and unloads a directory’s environment automatically as you cd in and out, but it is a developer convenience, never a production mechanism.

Precedence is a contract, not an accident

When the same key is set in three places, the winner must be defined in advance and identical in every environment. The conventional order, highest priority first: CLI flags, then the OS environment, then the .env file, then a config file, then hard-coded defaults. Encode that order in one function and every collision resolves the same way in local dev, CI, and production.

# config/resolve.py
def resolve(key, cli, env, dotenv, defaults):
    for source in (cli, env, dotenv, defaults):   # first hit wins
        if key in source and source[key] is not None:
            return source[key]
    raise KeyError(key)

Precedence drift — where local and production disagree about which source wins — is the root cause of most “works on my machine” incidents. Writing the order down as code, rather than leaving it implicit in the order your modules happen to import, removes the ambiguity.

The reason CLI flags sit at the top and defaults at the bottom is that priority should track specificity of intent. A flag typed on the command line is the most deliberate signal a human can send — “for this run, use exactly this” — so it must override everything. An environment variable is the deployment’s considered choice, more authoritative than a file someone committed months ago but less than a one-off override. A .env file captures a developer’s local convenience; a config file in the repository captures the team’s shared baseline; and a hard-coded default captures the last-resort value that keeps the process runnable when nothing else is set. Ordering the sources this way means the more someone went out of their way to set a value, the more that value is respected — which is exactly the behaviour an operator expects under pressure.

The one rule this ordering must never break is that lower-priority sources may supply a value but must never silently overwrite a higher one. This is why loading a .env file should read into an isolated dictionary rather than blindly calling something that mutates os.environ with override=True: an operator who exported DATABASE_URL in the shell has made a higher-priority choice, and a .env load that clobbers it has inverted the contract. Get the direction of precedence wrong once and every “I set the variable but the app ignored it” bug becomes possible.

Key rule: document the order once and enforce it everywhere. Details, including how pydantic-settings orders its own sources, are in Configuration Precedence Rules.

How pydantic-settings orders its own sources

You rarely have to write the resolver above by hand, because pydantic-settings already implements a documented, deterministic order: arguments passed directly to the model win first, then environment variables, then the .env file, then values read from a secrets directory, then the field defaults. That order is deliberate — it is exactly the “platform beats file beats default” precedence you want — and it is the same in every environment because it is a property of the library, not of your import graph. When the built-in order is not enough, settings_customise_sources lets you insert a source (a call to a secrets manager, a remote config service) at a specific priority, so a Vault lookup can sit above the .env file but below an explicit CLI override.

Walking through one collision makes the value concrete. Suppose PORT is set three ways during a debugging session: a config file ships port: 8080, the deployment environment exports PORT=8000, and an engineer runs the process with --port 9000. Under the fixed order the CLI flag wins and the service binds 9000; remove the flag and the environment’s 8000 takes over; remove that and the file’s 8080 remains. Nothing about that outcome depends on which module imported first or which .env happened to be present, which is precisely why the contract holds. The failures this prevents are the subtle ones — a value that resolves differently in staging than in production because the two environments load their sources in a different order — and the fix is always the same: make the order a property of one settings object and assert it in a test.

Structured files: parse safely

YAML, JSON, and TOML express nested configuration that flat environment variables cannot — feature maps, routing tables, per-tenant overrides. The danger is yaml.load, which can instantiate arbitrary Python objects from the document and is therefore a remote-code-execution vector on any input you do not fully control.

# config/files.py
import yaml

with open("config.yaml") as fh:
    data = yaml.safe_load(fh)   # never yaml.load() on untrusted input

safe_load restricts the document to plain dicts, lists, and scalars, which is exactly what configuration should be. For deeply nested files, validate the parsed structure with a pydantic model so a mistyped key or missing section is caught at load time rather than on first access.

The yaml.load-versus-safe_load distinction is not a hypothetical purity concern. The full loader supports YAML tags such as !!python/object/apply:os.system, which means a document you parse can invoke arbitrary code the instant it is loaded — no field ever has to be read for the payload to fire. That makes any config file an attacker can influence, directly or through a supply-chain path like a vendored template or a fetched remote fragment, a remote-code-execution vector. safe_load refuses those tags outright, and because legitimate configuration never needs them, switching costs you nothing and closes the hole permanently. Treat yaml.load on anything but a string you literally wrote in the same function as a defect to be flagged in review, the same way you would flag eval on external input.

Key rule: safe_load only, always. The nested-config and TOML-versus-YAML trade-offs are covered in YAML & JSON Parsing Strategies.

Validate the parsed structure, do not trust it

safe_load guarantees you get plain data, but plain data is not the same as correct data: a nested YAML file can still be missing a section, spell a key wrong, or put a string where a number belongs, and by default those mistakes surface only when some code path finally reads the value. Feed the parsed dictionary straight into a pydantic model and the whole document is validated at load time — every required section must be present, every field must have the right type, and an unexpected key is rejected if you set extra="forbid". The format on disk becomes a pure readability choice; the correctness guarantee comes from the model, and it is identical whether the bytes arrived as TOML, YAML, or JSON.

# config/files.py
import tomllib                       # stdlib on Python 3.11+ (use `tomli` on 3.10)
from pathlib import Path
from pydantic import BaseModel

class Cache(BaseModel):
    host: str
    ttl_seconds: int = 300

class FileConfig(BaseModel):
    model_config = {"extra": "forbid"}   # a mistyped section name fails loudly
    cache: Cache

data = tomllib.loads(Path("config.toml").read_text())
config = FileConfig.model_validate(data)   # nested, typed, checked once
config.cache.ttl_seconds                    # a real int, not "300"

The choice of format then follows from the job. TOML (tomllib, standard library since Python 3.11) is the right default for a human-edited application config: it is explicitly typed, forgiving of comments, and already the format of pyproject.toml. YAML earns its place only for genuinely deep, repeated structure — and only ever through safe_load, because yaml.load will happily construct arbitrary Python objects from a !!python/object tag, which is a remote-code-execution vector on any document you did not write yourself. JSON is best left to machine-generated config and wire formats, since it forbids comments and rejects the trailing commas that human editors leave behind.

Gate it in CI before it ships

Configuration errors should fail a pipeline, not a pod. A standalone validation step that instantiates the settings object catches missing keys and malformed values before the container is promoted, and it runs in seconds. Add a secret scanner alongside it so a committed credential is caught in the same stage. The difference in blast radius is the entire argument: a validation error caught in CI is a red check on a pull request that one engineer fixes in a minute, while the same error caught at deploy time is a crash-looping pod, a rolled-back release, and a handful of people staring at logs. The check is cheap and the failure it prevents is not, so there is no reason to run configuration for the first time in production when a pipeline can run it first for free.

# scripts/check_config.py — run in CI
from config.base import AppConfig

AppConfig()  # non-zero exit on any ValidationError fails the build
print("configuration OK")

Running this against a staging-shaped environment surfaces the “one variable is unset in prod” class of bug at review time. In a real pipeline the same idea becomes a short job that runs on every pull request, before anything is built or deployed:

# .github/workflows/config.yml
name: config
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    env:
      PYTHONWARNINGS: error        # deprecation warnings become failures
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -e . && python scripts/check_config.py
      - uses: gitleaks/gitleaks-action@v2   # fail if a secret was committed

Three things are happening in that job, and each catches a different class of mistake. Constructing the settings object turns a missing or malformed variable into a failed check with a precise field name. PYTHONWARNINGS=error promotes the deprecation warnings that a library upgrade introduces into hard failures, so a class Config left over from pydantic v1 cannot quietly rot until it breaks in production. A secret scanner — gitleaks or detect-secrets — reads the diff and blocks the merge if a credential slipped into a committed file. For the strongest signal, snapshot config.model_dump() with secrets excluded and diff it between environments in the pipeline; a value that is present in staging but absent in production shows up as a diff long before it shows up as an incident. The full GitHub Actions and GitLab CI recipes are in CI/CD Config Validation.

Hand the validated object to the framework

The three layers end at a validated object, and an application then has to give that object to whatever framework it runs on. Each framework has one integration point and one moment at which it reads configuration — Django at the import of its settings module, Flask inside the application factory before extensions initialise, FastAPI through a cached dependency, Celery when the worker boots — and matching your model to that moment is what turns a missing variable into a failed boot rather than a failed request.

# settings.py — the Django adapter: assignments, no logic of its own
from config import settings

SECRET_KEY = settings.secret_key.get_secret_value()
DEBUG = settings.debug
ALLOWED_HOSTS = settings.allowed_hosts

The rule is that the framework never owns the schema. One model defines every field; each framework gets a thin adapter that renames already-resolved values into the shape it expects and does nothing else. That keeps a management command, a test and a web request all reading the same validated object, and it means adding a field is one edit rather than four. See framework configuration integration for the adapter per framework and the per-process scoping that keeps a worker from holding credentials it never uses.

Prove the rules with tests, not with a deploy

Everything above is a set of rules — required fields, types, precedence, safe parsing — and rules are cheap to test because the model needs no database, no application object and no network. That makes the rejections worth asserting: that a malformed URL raises naming the field, that an unknown variable is refused by extra="forbid", that DEBUG=true is rejected when the environment is production.

# tests/test_settings.py — the assertion that protects the fail-fast guarantee
def test_missing_required_field_is_rejected(env):
    env(DATABASE_URL="postgresql://u:p@db/app")     # SECRET_KEY deliberately absent
    with pytest.raises(ValidationError) as exc:
        Settings(_env_file=None)
    assert exc.value.errors()[0]["loc"] == ("secret_key",)

The one discipline these tests need is isolation: environment variables come from monkeypatch, file sources are disabled with _env_file=None, remote sources are mocked, and any cached settings instance is cleared. Without that, a test passes because of a developer’s .env file and proves nothing about the schema. Testing configuration code covers the fixtures, and the division of labour that keeps unit tests on the rules while the pipeline checks a specific environment against them.

Anti-patterns & common mistakes

• Silent defaults for required values — os.getenv("DATABASE_URL", "sqlite:///dev.db") ships a dev database to production when the real var is misspelled.
• override=True on dotenv loading — overwrites platform-injected secrets with stale local values.
• yaml.load on untrusted input — a remote-code-execution vector; always safe_load.
• Branching business logic on ENV strings — if os.environ["ENV"] == "prod" scatters environment knowledge across the codebase instead of into config.
• Reading os.environ in twenty modules — fragments configuration and makes it untestable. Centralize on one settings object.
• Committing .env — the single most common secret leak. Gitignore it and scan for it in pre-commit.

Two of these deserve a full incident narrative, because they cause the most expensive outages. The silent default — os.getenv("DATABASE_URL", "sqlite:///dev.db") — is dangerous precisely because it never fails: misspell the real variable, or forget to set it in a new environment, and the process starts cheerfully against a local SQLite file. Every write appears to succeed, every read returns plausible data, and the truth only surfaces hours later when someone notices production orders are missing, because they were being written to an ephemeral file inside a container that has since been replaced. A crash at startup would have been trivially diagnosable; the silent default turned it into a data-loss investigation. The override=True mistake is its mirror image on the secrets side: a rotated database password is injected correctly by the platform, but a stale .env left in the image overrides it, so the service authenticates with the old credential, gets rejected, and takes down every request until someone traces the failure back to a file nobody thought was even being read. Both failures share a root cause — configuration that resolves differently than the author assumed — and both are eliminated by the same discipline: fail fast on missing required values, and let the platform always win over a file.

Decision flow: which source for which value?

Before adding a value anywhere, decide which layer owns it. Secrets come from a manager, environment-specific values from the environment, structured data from a checked-in file, and stable constants from a default inside the model. The decision matters because putting a value in the wrong layer creates work later: a secret hard-coded as a default has to be chased down and rotated the moment the repository is cloned to a laptop; an environment-specific URL baked into a file forces a rebuild for every environment instead of a config change; a deeply nested structure crammed into flat environment variables becomes an unreadable wall of SECTION__SUBSECTION__KEY names. The tree below resolves those trade-offs in one pass — ask what kind of value you are holding, and the branch tells you which mechanism owns it and how it is read at startup.

The one branch worth lingering on is the secret. A credential is the only value that lives in two places at once: it is authored and rotated in a secret manager, and it is mirrored into a local .env so a developer can run the app offline. That mirroring is safe only because the .env is gitignored and holds a scoped development credential, never the production one — which is exactly why the “secret” branch points at a manager first and a local file second, and never at a committed default.

CI/CD integration checklist

1. Add a .env.example with dummy values and keep the real .env gitignored.
2. Run a secret scanner (gitleaks or detect-secrets) as a pre-commit hook and a CI job.
3. Add a pipeline stage that imports and instantiates the settings object; fail the build on any error.
4. Run that stage with PYTHONWARNINGS=error to surface deprecation and syntax warnings.
5. Diff the settings schema between staging and production to catch drift before promotion.
6. Block deployment if any required key is unset in the target environment.

The ordering is not arbitrary: it moves from the cheapest, fastest check to the most expensive, so a pull request fails on the obvious problems in seconds and only reaches the slower gates once the basics pass. Shipping a .env.example and scanning for secrets costs nothing and catches the two most common mistakes — a missing variable and a committed credential — before a reviewer even looks at the code. Constructing the settings object is a few hundred milliseconds and catches every type and range error. Treating warnings as errors catches the slow decay of a dependency upgrade. Diffing the schema across environments is the one check that catches drift, the class of bug that no single environment can reveal on its own. Adopt them in that order and the marginal cost of each new gate is small, while the class of production incident it removes is large — which is the whole economic argument for gating configuration in the first place.

Bringing it together

Configuration in Python is reliable when it is boring: every value enters through one typed settings object, the precedence order is fixed and identical across environments, .env files stay local and uncommitted, structured files are parsed safely, os.environ is read once and typed, and CI gates reject mistakes before they ship.

None of these practices is individually clever, and that is the point. Configuration is not where a system should be interesting; it is the flat, dependable floor that everything else stands on, and the goal is to make its behaviour so predictable that no one has to think about it during an incident. The recurring theme across every topic here is fail loudly and early: a missing variable stops the process at startup rather than mid-request, a mistyped value raises with a field name rather than corrupting data, a stray .env is ignored rather than allowed to shadow a real secret, and a misconfiguration fails a pull request rather than a production deploy. Each of those is the same move applied to a different layer — turn a slow, ambiguous, expensive failure into a fast, precise, cheap one.

From here, the natural next step is to stop hand-writing accessors and let a schema do the work: layer type-safe validation with pydantic-settings, which turns the patterns on this page into a single declared model, and pull secrets from a managed store under enterprise secrets management so credentials are fetched at runtime rather than sitting in any file at all. Together those three sections form the complete path from a loose environment variable to a validated, secret-safe configuration object that proves itself correct before the application accepts a single request. Master this floor first, and everything you build on top of it inherits the same predictability.