Handling breaking changes in production config schemas

A rolling deployment runs the old and new schema against the same environment at the same time. Rename a variable, make an optional field required, or remove a key, and one side breaks the instant the change lands — unless you stage it. This page catalogs and sequences breaking schema changes safely, extending Schema Evolution & Versioning.

This page is a taxonomy: the specific kinds of change that break a production config schema, and the safe recipe for each. Not every schema change is breaking — adding an optional field with a default is free — so the first skill is recognising which changes are dangerous. A change is breaking if it makes the old and new schemas disagree about what counts as a valid configuration, because during a rolling deploy both schemas are live and any disagreement strands whichever side is on the wrong end of it. The dangerous changes fall into a short list: renaming a variable, promoting an optional field to required, removing a field, changing a default, and tightening a constraint. Each is subtractive in effect — it either demands something new from the environment that was not required before, or rejects something that used to be perfectly acceptable — and each has a staged recipe that keeps the two schemas compatible throughout the transition.

The unifying tool across the recipes is additive-then-subtractive: ship the part that widens compatibility first (an alias, an optional field, a warning), let every environment and every instance catch up, and apply the narrowing part (removing the alias, promoting to required, deleting the field) only afterward. Get that order right and a schema can change as freely as any other part of your system.

The reason this matters more for configuration than for most code is the rolling deploy. Ordinary code runs in one version per process, so a function you rename is renamed everywhere that process is concerned. A settings schema is different: during every rolling deploy, two versions of your code — old and new — read the same injected environment through different schemas, for a window that lasts as long as the rollout takes. That shared environment is the crux. Any change that makes the old and new schemas want different things from it will break whichever version is caught on the wrong side, and there is always a window where both are running. The additive-then-subtractive discipline exists to guarantee that during that window, the environment satisfies both schemas at once.

Problem 1: rename in a single release

# ANTI-PATTERN: old pods set DB_DSN, new model only accepts DATABASE_URL
class Settings(BaseSettings):
    database_url: str    # old pods inject DB_DSN -> ValidationError during rollout

The moment you switch the injected variable, every pod still running the old image fails to boot. A rename done in a single release is the canonical breaking change because it changes the schema and the environment together, leaving no window where both old and new code can validate. The old pods want DB_DSN; the new manifest provides DATABASE_URL; and during the minutes a rolling deploy takes to cycle every pod, the old ones raise ValidationError and crash. The safe recipe is the alias overlap — accept both names for a migration window — but the point of listing it here is that a rename is breaking by default and only becomes safe when you deliberately stage it across an overlap window.

A single-release rename breaks old pods during the rollout When the variable is renamed in one release, the injected environment provides DATABASE_URL but old pods still reading DB_DSN raise ValidationError until the rollout finishes. env now provides DATABASE_URL new pods validate old pods (DB_DSN) ValidationError partial outage until rollout finishes
Renaming in one release gives no overlap window, so old-code pods crash until every one has cycled.

Problem 2: optional made required immediately

# ANTI-PATTERN: a field that was optional is now mandatory
class Settings(BaseSettings):
    region: str          # environments that never set REGION now crash

Promoting a field to required without a grace period breaks every environment that did not already set it. This is the subtler breaking change, because it does not look like one — you are not removing or renaming anything, just deleting a default. But removing a default turns an optional field into a required one, and every environment that relied on the default (that is, every environment that never set REGION) now fails to construct the model. The blast radius is exactly the set of environments that were happy with the default, which is often most of them, so the change that looked like a small tightening becomes a fleet-wide startup failure at the very next deploy, affecting precisely the environments that were quietly working before.

The safe recipe inverts the naive order: introduce the field as optional with a default, deploy it, then get every environment to set the variable explicitly, and only once telemetry confirms that every last one of them does can you safely remove the default to make the field required. The field’s requiredness is the very last thing to change, not the first — a promotion is subtractive (it shrinks the set of valid configurations to those that explicitly set the field) and so follows the same additive-then-subtractive discipline as a removal. A useful test to apply before deleting any default is to ask: which environments never set this variable? Those are exactly the ones the change will break, and if the answer is “most of them”, the change is far more disruptive than the one-line diff suggests. Answer that question first, get those environments to set the variable, and only then remove the default.

Promoting an optional field to required breaks environments on the default Removing a default makes a field required, so every environment that never set REGION and relied on the default now fails to construct the model. env never set REGION relied on the default default removed field now required missing → crash fleet-wide deleting a default is a breaking change in disguise
Removing a default makes the field required, so every environment that relied on the default fails at once.

Secure implementation

# config/migration.py
from pydantic import AliasChoices, Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(populate_by_name=True, extra="forbid")

    # Step 1: accept both names during the overlap window.
    database_url: str = Field(validation_alias=AliasChoices("DATABASE_URL", "DB_DSN"))

    # Step 2: introduce the new field as optional with a safe default first.
    region: str = "us-east-1"

    @model_validator(mode="after")
    def warn_legacy(self) -> "Settings":
        # Emit a metric/log when the legacy name is still in use so you know
        # when it is safe to drop the alias.
        return self

The change ships in stages: accept both names, roll out, switch every environment to the new name, confirm the legacy name is unused, then remove the alias in a later release. New required fields land as optional-with-default first and are promoted only once every environment sets them.

Seeing the change types side by side clarifies the pattern. A rename is breaking; its recipe is an alias overlap, removed last. A promotion to required is breaking; its recipe is optional-with-default first, requiredness last. A removal is breaking under extra="forbid"; its recipe is to stop setting the variable everywhere, then delete the field. A default change is breaking for environments on the old default; its recipe is announcement and a staged rollout. A constraint tightening is breaking for environments carrying a now-invalid value; its recipe is canary, observe, widen if needed, then roll out. And the one non-breaking change — adding an optional field with a default — needs no ceremony at all. The recipes differ in detail but share one shape: widen compatibility first, narrow it only after the fleet has caught up.

Notice why the two “invisible” breaking changes — a promotion to required and a default change — are the ones that catch teams out. Neither adds or removes a variable name, so neither looks like a schema change in review; both are a one-line edit to a field’s default. Yet a promotion to required breaks every environment that relied on the default, and a default change silently alters behaviour for exactly those same environments. The lesson is that the default is part of the contract, and editing it is a contract change with a blast radius equal to the set of environments that never overrode it. Reviewing a config-model diff, the edits to watch are not just the added and removed field names but the changed and deleted defaults, because those unassuming one-line edits are where the quiet breakage actually lives.

The verification step between “widen” and “narrow” is what makes the whole discipline safe rather than merely well-intentioned. It is not enough to believe every environment has set the new variable or adopted the new name; you have to know, because under extra="forbid" a single missed environment turns the narrowing step into a startup failure. The two reliable signals are telemetry — a metric that counts which name or field each process actually used — and CI, which can fail the build if a deprecated field is still set (with PYTHONWARNINGS=error) or prove the overlap works by constructing the model under both shapes. Wait for the signal to go clean before you narrow, and the removal or promotion is safe.

A taxonomy of config-schema changes and their safe recipes Rename, promote-to-required, remove, change-default, and tighten-constraint are breaking and each has a staged recipe; adding an optional field with a default is not breaking. change safe recipe rename a variable AliasChoices overlap, remove old name last optional → required default first, require once all environments set it remove a field stop setting it everywhere, then delete (forbid) change a default announce, stage the rollout, note in changelog add optional field + default not breaking — ship anytime
Each breaking change has a staged recipe; only adding an optional defaulted field is free.

Gotchas & version-specific behaviour

  • AliasChoices returns the first matching source — document which name wins if both are set.
  • extra="forbid" means a removed field becomes an error; stop injecting it before deleting it.
  • Use populate_by_name=True so both the field name and its alias resolve.
  • Run CI with PYTHONWARNINGS=error::DeprecationWarning so deprecations cannot be ignored.

The extra="forbid" interaction is the gotcha that turns removals dangerous. forbid is what makes the schema strict — it rejects unknown variables — but it also means that the instant you delete a field, any environment still injecting the old variable hits extra not permitted and fails to boot. So removal is always the last step, gated on confirming no environment sets the variable. The PYTHONWARNINGS=error::DeprecationWarning setting is what makes the “confirm it is unused” step reliable: Python suppresses deprecation warnings by default, so without this a still-set deprecated field passes CI silently and you never learn the deadline is being missed until the removal itself breaks something in production.

Four breaking-change gotchas AliasChoices returns the first match so document which wins, forbid makes a removed field's variable an error, use populate_by_name so both names resolve, and run CI with deprecation warnings as errors. First match winsforbid + removal populate_by_namewarnings as errors document which name wins if both are set a removed field makes a leftover variable an error so both the field name and its alias resolve PYTHONWARNINGS=error so deprecations surface
Document alias precedence, gate removals on forbid, enable populate_by_name, and surface deprecations in CI.

Production parity checklist

  • Every rename ships with a two-name alias overlap window.
  • New required fields are optional-with-default first, promoted later.
  • Removal happens only after telemetry shows the old name/field is unused.
  • CI validates the model with both old and new variable names during the window.
  • Schema changes are recorded in a changelog operators read before deploying.

Across every change type, the checklist reduces to one habit: separate the additive part of a change from the subtractive part, and put a verification step between them. Ship the alias, the default, the deprecation warning; wait; confirm via telemetry or CI that the fleet has caught up; then remove the alias, promote to required, delete the field. The changelog entry is what makes the whole transition legible to the operators who actually set the variables, and the CI test that validates both the old and new shapes is what keeps the overlap honest. None of these steps is expensive, and together they turn “changing production config” from a feared operation, put off until it is unavoidable, into a routine one you can do whenever the schema needs it. The teams that struggle with configuration are usually the ones that never built this muscle — every rename is a risky event, so names calcify and the whole schema slowly grows awkward and hard to reason about — while the teams that internalised the additive-then-subtractive discipline change their schema freely, because they know each change is staged and reversible.

Breaking-change production-parity checklist Every rename ships with a two-name overlap, new required fields are optional-with-default first, removal happens only after telemetry shows disuse, CI validates both names, and schema changes are recorded in a changelog. Every rename ships with a two-name alias overlap window New required fields are optional-with-default first, promoted later Removal happens only after telemetry shows the old name is unused CI validates the model with both old and new names during the window Schema changes are recorded in a changelog operators read first
Five checks that keep every breaking change staged, verified, and legible to operators.

Key takeaways

Stage every breaking change — alias the rename, default the new field, remove last — and old and new pods coexist without a failed boot. The value of thinking in a taxonomy is that it lets you recognise a breaking change before you ship it: a rename, a promotion to required, a removal, a default change, and a constraint tightening are all breaking, and each looks deceptively small in a diff. Learn to spot each of them, and the safe recipe follows almost automatically — widen compatibility first, verify the fleet has caught up, narrow it last.

The one non-breaking change — adding an optional field with a default — is the exception that proves the rule: it is safe precisely because it demands nothing new from any environment at all and rejects nothing that was valid the day before. Every breaking change is breaking because it fails one of those two tests, and every single recipe on this page is a way to restore that compatibility property temporarily, through a deliberate overlap window, until the whole fleet has demonstrably moved to the new shape and the old one can be retired safely. For the alias mechanism itself, see Backward-Compatible Config with validation_alias.

Back to Schema Evolution & Versioning