Testing Configuration Code

The configuration test that passes on every machine and proves nothing is a familiar object: it constructs the settings model, asserts the database URL is not empty, and goes green because the developer’s .env file happened to be in the working directory. Then CI runs it in a clean container and it fails — or worse, it does not fail, because CI also has variables set, and the first genuine discovery of a missing key happens in staging. Configuration is the cheapest thing in a codebase to test properly and the easiest to test uselessly, and the difference comes down to isolation. This page covers how to isolate the environment, how to defeat caching, and what is actually worth asserting, extending the core configuration patterns into the test suite.

The reason this class of test goes wrong so consistently is that a settings model reads from ambient state. An ordinary unit test passes its inputs as arguments, so the test is a closed statement: given these inputs, expect this output. A settings test’s inputs arrive from the process environment, the working directory, a file that may or may not exist, and sometimes a network service — none of which appear in the test body. Unless the test explicitly takes control of all of them, it is not testing your model, it is testing your model plus whatever the machine happened to be holding. Every technique below is a way of closing that openness so the test says what it means.

Where configuration tests sit in the suite

Configuration tests belong at the very bottom of the suite: no database, no HTTP client, no application object, no network. Their subject is the model — the same model the framework adapters consume and the same one CI constructs against a real environment. Because the model can be imported and built in isolation, these tests run in milliseconds and can be exhaustive in a way that integration tests never can: you can enumerate every invalid value a field should reject without paying a container start per case.

That speed changes what is worth asserting. In an expensive test tier you assert the happy path and move on; in a tier this cheap, the rejections are the valuable assertions. A test that proves DATABASE_URL=not-a-url raises a ValidationError naming database_url is a test of the guarantee your production boot depends on. A test that proves an unknown variable is rejected is a test of extra="forbid". A test that proves DEBUG=true is refused when ENVIRONMENT=production is a test of a cross-field rule that would otherwise only be exercised by a bad deploy. None of these need infrastructure, and together they turn the model from a piece of code you hope is right into a specification with proof attached.

There is a boundary worth drawing clearly, though. These tests should not verify that a particular environment has the right values — that is CI’s job, using the real environment, described on the validation gate page. Tests verify the rules; the CI gate verifies a specific environment against those rules. Conflating them produces a test suite that fails whenever staging changes a value, which teaches everyone to ignore failing configuration tests.

Secure implementation

The complete pattern: isolate the environment, disable file sources, clear the cache, then assert on both acceptance and rejection.

# tests/conftest.py — isolation applied to every test in the suite
import pytest
from config import get_settings


@pytest.fixture(autouse=True)
def isolate_config(monkeypatch):
    """Remove ambient configuration so no test inherits the developer's machine."""
    for key in ("DATABASE_URL", "SECRET_KEY", "REDIS_URL", "ENVIRONMENT", "DEBUG"):
        monkeypatch.delenv(key, raising=False)   # raising=False: absent is fine
    get_settings.cache_clear()                   # discard any instance built earlier
    yield
    get_settings.cache_clear()                   # and leave nothing behind


@pytest.fixture
def env(monkeypatch):
    """Set exactly the variables a test declares — nothing implicit."""
    def _set(**values: str):
        for key, value in values.items():
            monkeypatch.setenv(key.upper(), value)
    return _set

# tests/test_settings.py
import pytest
from pydantic import ValidationError
from config import Settings


def test_valid_environment_builds(env):
    env(DATABASE_URL="postgresql://u:p@db:5432/app", SECRET_KEY="s3cret")
    settings = Settings(_env_file=None)          # ignore any .env on disk
    assert settings.database_url.host == "db"
    assert settings.secret_key.get_secret_value() == "s3cret"


def test_missing_required_field_is_rejected(env):
    env(DATABASE_URL="postgresql://u:p@db:5432/app")   # SECRET_KEY deliberately absent
    with pytest.raises(ValidationError) as exc:
        Settings(_env_file=None)
    assert exc.value.errors()[0]["loc"] == ("secret_key",)   # assert the field, not the message


def test_unknown_variable_is_rejected(env):
    env(DATABASE_URL="postgresql://u:p@db:5432/app", SECRET_KEY="s", DATBASE_URL="typo")
    with pytest.raises(ValidationError):
        Settings(_env_file=None)                 # extra="forbid" turns a typo into a failure

Three details carry most of the weight. monkeypatch.delenv(..., raising=False) clears inherited values without failing on machines where they were never set, which is what makes the fixture safe to mark autouse. _env_file=None disables the file source for the duration of the test, so a .env sitting in the repository root cannot supply a value the test did not ask for — the single most common cause of a configuration test that passes locally and fails in CI. And cache_clear() on both sides of the yield means neither a previously-built instance nor the one this test builds can bleed into a neighbour.

The assertion style matters too. exc.value.errors()[0]["loc"] names the field that failed; asserting on the human-readable message ties the test to a library version’s wording, and pydantic has changed that wording between minor releases. Field locations are part of the API in a way messages are not, so tests written against loc survive upgrades — including the v1-to-v2 migration, which rewrote nearly every message string.

Technique reference

The Settings(**overrides) row is the shortcut worth knowing: initialisation arguments sit above environment variables in the default source order, so a test that cares about one field can pass it directly and let the rest come from a valid baseline. It keeps tests short and readable, and it is the mechanism behind most fixture factories — a settings_factory(**overrides) helper that builds a valid baseline once and lets each test perturb a single value is usually the most-used fixture in the file.

The mocked-client row hides a design question that is worth answering early: what exactly is the unit under test when a secret comes from a remote store? It is not the store’s API, and it is not the network. It is your model’s contract with the store — that it asks for the right key names, that it maps the response onto the right fields, and that it fails cleanly when the store returns nothing. A mock that records calls lets you assert all three in a millisecond. Anything beyond that belongs to a separate, deliberately small integration test that runs against a disposable resource on a schedule, not in the pull-request path, because its purpose is to detect provider-side change rather than to guard your code.

One caution on secrets_dir: point it at pytest’s tmp_path rather than a fixed directory. Tests that write into a shared path are order-dependent and fail confusingly under pytest -n auto, and a fixed path under the repository can end up committed. With tmp_path each test gets a fresh directory that the framework cleans up, and the same fixture works unchanged when the suite starts running in parallel.

Deployment parity, step by step

1. Write the model’s rules as tests first. For each field, one accepting case and one rejecting case. These are the fastest tests you will ever write and they encode the guarantee everything else assumes.
2. Make isolation automatic. The autouse fixture that clears the environment and the cache should exist before the second configuration test is written, not after the first mysterious failure.
3. Run the suite in a clean container locally. If docker run with no environment produces a different result from your machine, the suite is inheriting something — find it before CI does.
4. Add the CI construction gate separately. Tests prove the rules; a CI job proves the target environment satisfies them. Keep the two jobs distinct so a staging value change never turns the unit suite red.
5. Test the adapters once each. One smoke test per framework entry point — that the Django settings module imports, that the Flask factory produces a configured app — is enough. The schema is already covered.
6. Re-run the rejection tests after every dependency upgrade. They are the tests most likely to catch a behavioural change in a validation library, and the cheapest to run.

Step 1 is worth taking literally. Writing the rejection test before the field exists forces you to decide what the field’s failure looks like — is a missing value fatal, or is there a sane default? does an empty string count as absent? should a value that parses but is nonsensical be caught here or at the point of use? Those decisions are cheap to make while writing a test and expensive to revisit once three environments have been configured around an accidental answer. The test then doubles as the field’s documentation: a reader who wants to know what LOG_LEVEL accepts reads a parametrised list of accepted and rejected values rather than guessing from a type annotation.

Step 3 is the one that repays the effort most visibly. Running the suite in a container with an empty environment is the closest cheap approximation of CI, and it surfaces the two classic leaks — an inherited shell export and a .env file the model quietly found — in seconds rather than after a push. It also tends to reveal an unstated assumption or two: a test that relies on the repository being the working directory, or one that depends on a HOME that exists.

Testing the source order, not just the schema

Most configuration suites test the schema and stop there, which leaves the other half of the model untested: the order in which sources are consulted. That order is a real behaviour with real consequences — it decides whether a value baked into a container image can be overridden at runtime, whether a .env file left on a production host wins over the injected environment, and whether an operator’s emergency export actually takes effect. It is also invisible in the model definition, since it lives in defaults or in a settings_customise_sources hook, so nothing about reading the class tells you what wins.

Testing it is straightforward because you can supply two sources at once and assert which value survives. Write a file into tmp_path, set the same key in the environment, construct the model pointing env_file at that file, and assert the environment value is what appears. Then repeat for the pairs that matter to you: initialisation argument versus environment, environment versus file, file versus default. Four small tests pin the entire precedence contract, and they fail loudly if a future refactor reorders the sources — which is exactly the kind of change that otherwise ships unnoticed and only manifests as a value that stops being overridable in production.

# tests/test_source_order.py
def test_environment_beats_env_file(env, tmp_path):
    env_file = tmp_path / ".env"
    env_file.write_text("ENVIRONMENT=from-file\n")
    env(ENVIRONMENT="from-process", DATABASE_URL="postgresql://u:p@db/app", SECRET_KEY="s")
    settings = Settings(_env_file=env_file)
    assert settings.environment == "from-process"   # the process environment wins

The same technique covers the negative case that bites hardest in production: proving a value cannot be set from a source you did not intend. If secrets must never come from a file, write the key into a tmp_path env file, leave it out of the environment, and assert the construction raises. That test documents an intentional restriction, and it is the only kind of evidence that survives a well-meaning contributor adding a convenient file source later. Pair these with the precedence rules the site describes elsewhere, and the model’s behaviour is fully specified by executable statements rather than by a paragraph in a README that nobody re-reads.

Security boundaries and operational guardrails

• Never commit a real secret to a fixture. Test values should be obviously fake ("s3cret", "postgresql://u:p@db/app"), so a scanner that flags them costs a glance rather than a rotation.
• Do not print settings objects in test output. SecretStr masks in repr, but a failing assertion that calls get_secret_value() will render the raw value into the CI log.
• Keep the autouse fixture explicit about which keys it clears. A wildcard clear of os.environ breaks tools that need PATH and HOME; enumerate the application’s own variables.
• Never point a unit test at a real secret store. Beyond flakiness, a test suite with production credentials is a credential distribution mechanism.
• Assert on field locations, not messages. Message text is not a stable interface.
• Fail the suite on an unknown variable, not just a missing one. The extra="forbid" test is the one that catches the typo class of failure, which is more common than the omission class.

The fixture-value rule deserves elaboration because it is where test hygiene and secret hygiene meet. Test fixtures are the most-copied code in any repository: a value that appears in conftest.py will be pasted into a dozen tests, a snippet in a pull request, and eventually a wiki page. If that value resembles a real credential, every one of those copies is a false positive for whoever runs a secret scanner, and the team learns to dismiss scanner output — which is a far more expensive outcome than the original mistake. Deliberately shaped fake values ("u:p@db/app", "s3cret", obviously-invalid key prefixes) keep the scanner’s signal clean, and they make review easier because a reviewer can tell at a glance that nothing real is present.

The masking guardrail has a subtle edge worth knowing. SecretStr protects repr, which covers most accidental disclosure — a printed settings object, a captured traceback, a logged model dump. It does not protect an assertion you wrote yourself: assert settings.secret_key.get_secret_value() == expected renders both sides into the failure output when it fails, and CI logs are usually far more widely readable than production logs. Where a test genuinely needs to compare a secret, compare a hash or compare the masked form, and reserve get_secret_value() for the single test that proves unwrapping works at all.

Troubleshooting

A test passes alone and fails in the full suite. Something earlier in the run left state behind — usually a direct os.environ["X"] = ... assignment that was never reversed, or a settings instance still sitting in the cache. Run the failing test with pytest -p no:randomly and then bisect with --last-failed; the fix is almost always to convert the offending assignment to monkeypatch.setenv.

The suite passes but production still boots with wrong values. The rules are proven and the environment was never checked against them. This is the expected outcome of testing rules only, and it is why the construction gate exists as a separate job: unit tests cannot know what staging contains. If the gate is already in place and the boot still went wrong, look for a value that is valid by the schema but wrong in fact — a well-formed URL pointing at the previous database, for instance. Schema validation cannot catch that class, and the remedy is a narrow assertion in the deploy step rather than a broader model.

Everything fails after adding a required field. Every test that builds the model now needs the new variable. Rather than editing dozens of tests, give the baseline fixture the field and let tests override it — the fixture factory pattern exists precisely so that adding a required field is a one-line change.

ValidationError is not raised even though the value is invalid. The model found a valid value from a source you forgot to disable, typically the .env file. Add _env_file=None, and check whether a secrets_dir or a remote source is also in play.

The mocked secret client is never called. The settings instance was built before the mock was installed — the cached one. Clear the cache after installing the mock, or install the mock in a fixture that runs before the model is constructed.

A parallel run fails but a serial run passes. Two tests are sharing a path or a directory — usually a fixed secrets_dir, a written .env in the repository root, or a temporary file with a hard-coded name. Under pytest -n auto those tests execute in different processes at the same time and race. Switch every filesystem interaction to tmp_path, which is unique per test, and the race disappears without any locking or ordering hacks.

Tests are slow for no visible reason. A model with a remote source constructs a client on every build, and a suite that builds settings hundreds of times pays for it every time. Cache a valid baseline instance at session scope and derive per-test variants from it with initialisation arguments.

Key takeaways

The invariant this topic enforces is that a configuration test controls every input the model can read. Environment variables come from monkeypatch, file sources are disabled with _env_file=None or pointed at tmp_path, remote sources are mocked, and the cached instance is cleared on both sides of every test. A test that skips any one of those is not testing the model; it is testing the machine, and it will eventually disagree with CI in a way that costs an afternoon.

What makes the effort worthwhile is how much the rejection tests buy for how little. Because the model needs no infrastructure, you can afford to assert every rule it encodes — required fields, type parsing, unknown-key rejection, cross-field constraints — in a suite that runs in milliseconds. Those assertions are the same guarantees your production boot depends on, so proving them here means the CI gate has only one job left: confirming that a particular environment satisfies rules already known to be sound. Keep that division, keep the fixtures boring, and configuration stops being a source of surprises in either place.