Debug Which Config Source Set a Value

The value is wrong, the file says otherwise, and nobody can explain where the running process got what it has. Precedence questions are among the most time-consuming configuration problems precisely because the answer is invisible: a settings model reports what a field is, not where it came from. Building a small amount of reporting into the application converts these from an archaeology exercise into a command. This page shows what to build and how to use it, extending the configuration precedence rules topic.

Problem 1: guessing from the files

# ANTI-PATTERN: reading the repository and hoping it matches the container
print(open("config/production.yaml").read())      # what the repo says
# ... says nothing about the process that is actually running

Files in a repository describe an intention. The running process may have been built from a different commit, may have an environment variable overriding the file, may be reading a dotenv file that was accidentally included in the image, and may have a secrets directory mounted that the repository knows nothing about. Every one of those is common, and none of them is visible from the source tree.

Ask the process instead. Anything else is inference, and inference is how a precedence question turns into an afternoon: two people read the same files, reach the same wrong conclusion, and only discover the real source when somebody finally runs env inside the container.

Problem 2: a debug print that leaks

# ANTI-PATTERN: solves the mystery and creates an incident
print(settings.model_dump())          # a DSN type serialises with its password

Under time pressure somebody dumps the whole configuration to find the offending value, and the dump includes a URL whose password is not masked by anything. The output goes into a terminal, a log, a chat message and a ticket, and now a credential needs rotating on top of the original problem.

Build the diagnostic before you need it, with masking baked in. A report that prints field names, source names and values — with wrapped fields rendered as placeholders and URL types masked — is safe to run anywhere and safe to paste anywhere. That safety is what makes it usable in the moment it matters, which is inside a production container during an incident.

Problem 3: reasoning about precedence without testing it

# ANTI-PATTERN: "the environment variable should win"
# ... should it? has anyone checked, on this model, in this version?

Source order is a property of the model’s configuration, and it can be customised, inverted, or changed by a refactor nobody flagged. Assuming the default order is still in force is how a team spends an hour investigating a value that was, in fact, being correctly overridden by exactly the source they had ruled out.

Pin the order with tests, as described on the testing topic, and have the diagnostic report the actual order the model is using. Two lines of output — the source order, and which source won each field — turn precedence from a belief into an observation.

Secure implementation

# config_report.py — a safe, always-available answer to "where did this come from?"
import os
from pathlib import Path
from pydantic import SecretStr
from config import Settings, SECRETS_DIR, ENV_FILE


def _mask(value: object) -> str:
    # 1. wrapped secrets and anything URL-shaped never render in full
    if isinstance(value, SecretStr):
        return "**********"
    text = str(value)
    return text if "://" not in text or "@" not in text else text.split("@", 1)[0].split("//")[0] + "//***@***"


def source_of(field: str) -> str:
    """Report the highest-priority source that actually supplies this field."""
    if field.upper() in os.environ:                    # 2. checked in precedence order
        return "environment variable"
    if SECRETS_DIR.is_dir() and (SECRETS_DIR / field).exists():
        return "secrets file"
    if ENV_FILE.exists() and field.upper() in ENV_FILE.read_text().upper():
        return "env file"
    return "default"


def report(settings: Settings) -> list[tuple[str, str, str]]:
    # 3. one row per field: name, effective value (masked), and where it came from
    return [(name, _mask(getattr(settings, name)), source_of(name)) for name in settings.model_fields]


if __name__ == "__main__":
    s = Settings()
    width = max(len(n) for n, _, _ in report(s))
    for name, value, source in report(s):
        print(f"{name:<{width}}  {value:<32}  {source}")

# app.py — the same information, once, at startup
import logging
from config_report import report

logger = logging.getLogger(__name__)

for name, _value, source in report(settings):
    logger.info("config %s <- %s", name, source)       # 4. names and sources only, never values

The startup log deliberately omits values entirely and prints only the source per field. That is safe under every circumstance, it is small enough to leave enabled permanently, and it answers the most common form of the question — “is this process reading the file or the environment?” — from the first few lines of a container’s log, without anybody needing to exec into anything.

The source_of function checks in precedence order and returns the first hit, which is a deliberate simplification: it reports the source that wins, not every source that contains the key. That is almost always the question being asked. If you need the full picture — every source and its value — extend it to return a list, and keep the masking exactly as it is.

A diagnostic order that resolves most cases in minutes

When the report is not available yet — an inherited service, a container you cannot rebuild — there is a sequence that resolves most precedence puzzles quickly. Run it inside the actual running process’s container, not on a similar one.

First, print the environment variable directly: env | grep -i DATABASE. Roughly half of all cases end here, because the variable is set to something nobody expected, or is absent when everyone assumed it was present. Second, list the file sources that exist: ls -la /app/.env /etc/secrets/ 2>/dev/null. A .env file that was copied into an image is a frequent culprit and is invisible from the repository if it was created by a build step. Third, check the working directory with pwd, since relative file paths resolve against it and a process manager may start the application somewhere unexpected.

Only after those three should you start reading application code. The order matters because each step is cheap and each eliminates a whole class of explanation, whereas reading code first tends to confirm what you already believed. Write the sequence into the runbook next to the report command, so the next person does not have to rediscover it under pressure.

There is a category the report cannot help with, and it is worth recognising early so you stop looking for a precedence bug that does not exist: a value that is correct by every rule and wrong in fact. A well-formed URL pointing at last quarter’s database, a valid key belonging to the wrong account, a real region name that is not the one this service runs in — each passes validation, each is reported faithfully by the source report, and each produces behaviour nobody expects. If the report shows the value the environment intended to supply, the problem has moved from configuration mechanics to configuration content, and the next question is who set that variable and when rather than which source won.

Gotchas and version-specific behaviour

• A settings model exposes values, not their origins; provenance has to be built deliberately.
• .env files resolve relative to the working directory, so a process manager’s start directory changes which file is found.
• A file copied into an image by a broad COPY . . is a source nobody sees in the repository — exclude it explicitly.
• Case handling differs between sources: environment variables are conventionally upper case while fields are lower case, so a naive lookup can miss.
• Values that pass validation but are wrong — a valid URL pointing at the previous database — never surface as errors, only as behaviour.
• Two processes of the same service can hold different configurations if one restarted after a change and the other did not.
• Mask before printing, always, including in the throwaway code you write “just to check something” — that is the code most likely to end up pasted into a ticket.

Production parity checklist

• A source report exists in the codebase, masks secrets, and can be run inside any environment.
• Startup logs one line per field naming its source and no values.
• The runbook lists the three-step diagnostic order before any code reading.
• Tests pin the model’s source order so a refactor that reorders sources fails loudly.
• Container images exclude stray dotenv files by explicit rule rather than by habit.
• A health or status endpoint reports the environment name and configuration load time for cross-pod comparison.

Key takeaways

Precedence questions are slow because provenance is invisible by default, and the fix is to make it visible on purpose. A short report that walks the sources in order, names the winner for each field and masks anything sensitive turns the question into a command, and a startup log line naming sources without values answers the common case before anybody asks.

Ask the process, not the repository. Files describe intent; the running container may have an injected variable, a mounted directory or a stray dotenv file that the source tree knows nothing about. Keep the three-step diagnostic — print the variable, list the file sources, check the working directory — in the runbook ahead of any code reading, and pin the model’s source order with tests so a refactor cannot silently change the answer. The ordering rules themselves are on the configuration precedence topic page.