Merge Config Layers with Deep Merge in Python

A base configuration file plus a per-environment overlay is one of the oldest patterns in configuration, and it goes wrong in the same way every time: base.update(overlay) replaces whole branches instead of individual leaves, and the values that vanish are the ones nobody thought to check. Getting the merge right is a short piece of code with a few decisions worth making deliberately — especially about lists, deletion and provenance. This page covers them, extending the configuration precedence rules topic.

Problem 1: update replaces branches, not leaves

# ANTI-PATTERN: the overlay wipes every sibling key under "database"
base = {"database": {"host": "db.internal", "port": 5432, "pool_size": 20}}
overlay = {"database": {"host": "db.staging.internal"}}
base.update(overlay)
# {"database": {"host": "db.staging.internal"}} — port and pool_size are gone

The overlay intended to change one value and silently removed two others. Whether that matters depends on what happens next: if the model has defaults for port and pool_size, the application starts with values nobody chose; if they are required, it fails at startup with an error that points at the schema rather than at the merge.

The fix is a recursive merge that descends into dictionaries and only replaces at the leaves. It is a dozen lines and it is worth writing yourself rather than pulling in a dependency, because the interesting decisions — list behaviour, type conflicts, deletion — are choices you want to make explicitly rather than inherit from a library’s defaults.

Problem 2: merging lists because it seems more helpful

# ANTI-PATTERN: an overlay can add to this list but never remove from it
merged["allowed_origins"] = base["allowed_origins"] + overlay["allowed_origins"]

Concatenating lists feels generous and creates a configuration that only grows. An environment that must not include a development origin has no way to express that, so somebody adds a second key — allowed_origins_exclude — and now the effective value depends on two lists and an unwritten subtraction rule.

Replacement is the better default. An overlay that specifies a list means “this is the list here”, which is unambiguous, expressible in both directions, and easy to read in a diff. Where a genuine append is needed — a base set of middleware plus environment-specific additions — model it as two separate keys with different names, so the composition is visible in the schema rather than hidden in the merge.

Problem 3: no way to express deletion

# ANTI-PATTERN: how does staging turn this off?
# base.yaml
telemetry:
  endpoint: https://collector.internal
  sample_rate: 0.1

An overlay that omits telemetry inherits it, because omission is how every unchanged key looks. There is no way to say “this environment has none” without either a sentinel the merge understands or a field that can hold an explicit null.

Prefer the second: make the field optional in the schema and let an overlay set it to null. That keeps the merge dumb — it just replaces a leaf — and moves the meaning into the model, where a validator can enforce that a null endpoint disables telemetry consistently. A sentinel like __delete__ works too and costs you a special case in the merge plus a rule everyone has to learn.

Secure implementation

# merge.py — recursive for dicts, replacement for everything else
from typing import Any


def deep_merge(base: dict[str, Any], overlay: dict[str, Any]) -> dict[str, Any]:
    """Merge overlay onto base. Dicts recurse; every other type is replaced."""
    result = dict(base)                                  # 1. never mutate the inputs
    for key, value in overlay.items():
        current = result.get(key)
        if isinstance(current, dict) and isinstance(value, dict):
            result[key] = deep_merge(current, value)     # 2. descend to the leaves
        else:
            # 3. lists, scalars and type changes all replace — one rule, no surprises
            result[key] = value
    return result


def merge_all(*layers: dict[str, Any]) -> dict[str, Any]:
    merged: dict[str, Any] = {}
    for layer in layers:                                 # 4. later layers win
        merged = deep_merge(merged, layer)
    return merged

# config.py — merge files, then let environment variables win over all of them
import yaml
from pathlib import Path
from pydantic_settings import BaseSettings, SettingsConfigDict


def load_files(environment: str) -> dict:
    base = yaml.safe_load(Path("config/base.yaml").read_text()) or {}
    overlay_path = Path(f"config/{environment}.yaml")
    overlay = yaml.safe_load(overlay_path.read_text()) if overlay_path.exists() else {}
    return merge_all(base, overlay)                      # 5. one dictionary, fully resolved


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_nested_delimiter="__", extra="forbid")
    database: DatabaseSettings
    telemetry: TelemetrySettings | None = None           # 6. optional: an overlay can null it


def build(environment: str) -> Settings:
    # 7. env vars outrank the merged files because init values are the highest source
    return Settings(**load_files(environment))

Notice what the merge does not handle: environment variables. Files are merged into one dictionary, and then the settings model applies its own precedence, where initialisation values sit above environment variables — or below them, if you invert the call. Keeping file merging and source precedence separate means each is simple: the merge only knows about dictionaries, and the model only knows about sources. Trying to express “environment variable beats staging overlay beats base file” inside one recursive function is how merge code becomes unmaintainable.

The dict(base) copy on the first line matters more than it looks. A merge that mutates its input makes the order of calls significant in ways nobody expects, and it breaks the natural pattern of merging the same base against several overlays — which is exactly what a test comparing environments wants to do.

Recording where each value came from

Once more than two layers exist, “why is this value 30?” becomes a real question, and reconstructing the answer by reading files in order is slow and error-prone. Tracking provenance during the merge is cheap: alongside the merged dictionary, build a parallel structure mapping each leaf path to the name of the layer that last set it. The merge already knows — it is doing the assignment.

def deep_merge_tracked(base, overlay, source, provenance, path=()):
    result = dict(base)
    for key, value in overlay.items():
        here = path + (key,)
        current = result.get(key)
        if isinstance(current, dict) and isinstance(value, dict):
            result[key] = deep_merge_tracked(current, value, source, provenance, here)
        else:
            result[key] = value
            provenance[".".join(here)] = source        # the layer that won this leaf
    return result

Expose that map through a small command — python -m app.config explain — printing each key, its effective value and the layer it came from, with secret values masked. It costs almost nothing to maintain and it converts the most common configuration question into a command anybody can run, including in a production container during an incident. The same output is useful in code review, where a diff to an overlay can be checked against what it actually changes.

One caution about layer count: each additional overlay multiplies the number of combinations somebody has to reason about, and three is usually where the pattern stops paying. A base plus a per-environment file is easy to hold in your head; a base plus a region file plus an environment file plus a tenant file means the effective value for any key depends on four documents whose interaction nobody has tested. If you find yourself adding a fourth layer, consider whether the thing you are modelling is really configuration or has become a small deployment language — and if it is the latter, generating one flat file per target from a template is easier to review than resolving four layers at runtime.

Gotchas and version-specific behaviour

• dict.update and the | operator both replace at the top level; neither descends into nested dictionaries.
• Copy the base before merging, so the function has no side effects and can be reused across overlays.
• Replacing lists is the predictable default; append semantics should be a separate key with its own name.
• A type change between layers — a string where the base had a dictionary — should replace, and is worth logging as a warning since it usually indicates a mistake.
• Deletion needs an explicit representation: an optional field set to null, or a sentinel the merge understands.
• Keep environment-variable precedence in the settings model, not in the merge, or you end up with two systems that disagree.
• YAML anchors can express shared structure within one file and do not help across files, which is what the merge is for.

Production parity checklist

• One merge function, used everywhere, with no mutation of its inputs.
• List semantics documented next to the function and consistent across the codebase.
• Deletion expressed as an optional field set to null rather than by omission.
• Provenance tracked during the merge and exposed through a command that masks secrets.
• Environment variables resolved by the settings model, above the merged file values.
• A test merges the base against each environment overlay in turn and asserts that the settings model validates the result.

Key takeaways

A correct merge is short and its behaviour is entirely decided by three rules: dictionaries recurse, everything else replaces, and deletion is an explicit null rather than an omission. Writing those rules yourself rather than importing them means each is a choice you can defend in review, and the whole function stays small enough that its behaviour is obvious from reading it.

Keep the merge separate from source precedence. Files compose into one dictionary; the settings model then resolves that dictionary against environment variables and defaults using its own ordering. Add provenance tracking while you are there — one extra parameter — and expose it through a command that prints where each effective value came from, with secrets masked. That single tool answers the question people actually ask about layered configuration. The ordering rules it plugs into are on the configuration precedence topic page.