YAML & JSON Parsing Strategies

Flat environment variables cannot express a routing table or a per-tenant feature map; structured files can express both naturally. But yaml.load is a remote-code-execution vector — a single !!python/object/apply tag in a config file can run arbitrary code during parsing. This page parses structured configuration safely and then validates its shape, treating both as non-negotiable steps rather than optional hardening.

Structured files are the source the configuration pipeline reaches for when data is nested. The parsed result should always flow into a pydantic-settings model so its structure is validated, never trusted raw.

Two rules govern safe structured-config parsing, and they are independent. The first is parse safely: use the safe loader for each format — yaml.safe_load, json.loads, tomllib.loads — never the full yaml.load, which can instantiate arbitrary Python objects from the document and is therefore a code-execution vector. The second is validate the shape: the safe loaders give you plain dicts, lists, and scalars, but plain data is not the same as correct data, so you feed the parsed dict into a pydantic model that checks types, required keys, and constraints. Parse safely and validate the shape, and a config file becomes a typed, checked object; skip either and you have opened a security hole or admitted malformed configuration that fails at first use, far from its cause. The two rules are genuinely independent — you can parse safely and still forget to validate the shape, or validate a dict you parsed unsafely — so both must be applied together, every time, on every file the process reads.

The reason yaml.load is worth this much caution is that the danger is invisible. A YAML document parsed with the full loader can carry a tag like !!python/object/apply:os.system that executes the moment it is loaded — no field ever has to be read for the payload to fire. So any YAML file an attacker can influence, directly or through a vendored template or a fetched fragment, becomes a remote-code-execution vector the instant your process loads it with the full loader. safe_load refuses those tags outright, and because legitimate configuration never needs them, switching costs nothing. Treat yaml.load on anything but a string you wrote in the same function the way you would treat eval on external input.

Secure implementation

# config/structured.py
import json
import tomllib                      # stdlib in Python 3.11+
from pathlib import Path

import yaml
from pydantic import BaseModel


class CacheConfig(BaseModel):
    host: str
    port: int = 6379
    ssl: bool = True


class FileConfig(BaseModel):
    cache: CacheConfig
    feature_flags: dict[str, bool] = {}


def load_config(path: Path) -> FileConfig:
    text = path.read_text()
    if path.suffix in {".yaml", ".yml"}:
        raw = yaml.safe_load(text)      # NEVER yaml.load on untrusted input
    elif path.suffix == ".json":
        raw = json.loads(text)
    elif path.suffix == ".toml":
        raw = tomllib.loads(text)
    else:
        raise ValueError(f"Unsupported config format: {path.suffix}")
    return FileConfig.model_validate(raw)   # validate shape before use

safe_load removes the code-execution risk; model_validate turns an untyped dict into a checked object, rejecting missing keys and wrong types at load time.

The load_config function embodies both rules. It picks the safe loader by file extension — yaml.safe_load for YAML, json.loads for JSON, tomllib.loads for TOML — and rejects unknown formats, so there is no path where an unsafe loader runs. Then it calls FileConfig.model_validate(raw) on the parsed dict, which is where the shape is enforced: cache must be present and must itself be a valid CacheConfig, feature_flags must be a dict[str, bool], and any missing required field or wrong type raises a ValidationError naming the problem. The result is that the format on disk becomes a pure readability choice — YAML, JSON, or TOML — while the correctness guarantee comes entirely from the model and is identical whichever format the bytes arrived in.

The yaml.load attack, concretely

It is worth seeing exactly why yaml.load is dangerous, because “it can execute code” sounds abstract until you see the mechanism. YAML supports tags — annotations prefixed with !! that tell the loader to construct a specific type. The full loader honours tags like !!python/object/apply:os.system, which instructs it to call os.system with the tag’s argument during parsing. So a YAML document containing key: !!python/object/apply:os.system ["rm -rf /tmp/x"] runs that shell command the instant yaml.load parses it — before any of your code reads a single field. The payload fires at load time, from data, with no code path required to trigger it.

That makes the threat model broad. Any YAML your process loads that an attacker can influence is a code-execution vector: a config file in a repository an attacker can commit to, a vendored template pulled from a dependency, a fragment fetched over the network, a file uploaded by a user, or a merge of a base config with an override an attacker influenced. safe_load (and the SafeLoader) simply refuse to honour those tags — they construct only standard scalars, lists, and dicts, and raise a ConstructorError on an unknown tag. Because real configuration never legitimately needs !!python/... tags, using safe_load costs you nothing and closes the hole completely. The discipline is absolute: yaml.load and yaml.full_load are forbidden on any document you did not literally write in the same function, exactly as you would never eval a string from outside your program.

Why validating the parsed shape matters

safe_load guarantees you get plain data, but plain data is not the same as correct data — and the gap between them is where a whole class of configuration bugs lives. A nested YAML file can be missing an entire section, spell a key wrong, put a string where a number belongs, or nest a value one level too deep, and none of those is a parse error; the safe loader happily returns a dict with the wrong shape. Without validation, those mistakes surface only when some code path finally reaches for the missing or mistyped value, producing a KeyError or AttributeError deep in the application, far from the file that caused it and long after the file was loaded.

Feeding the parsed dict into a pydantic model closes that gap. The model checks that every required section is present, that every field has the right type, that nested sub-models are themselves valid, and — with extra="forbid" — that no unexpected key slipped in. A malformed file now fails at load time with a ValidationError that names the exact field, rather than at first use with an opaque error. This is the same single-boundary principle that governs environment variables, applied to files: the model is the one place a config file’s structure is checked, and everything downstream receives a typed, validated object it can trust without defensive checks. The format on disk is a readability choice; the correctness comes from the model.

Configuration reference

Each format has a niche, and matching the format to its use keeps configuration readable. JSON is best for machine-generated config — it is ubiquitous and unambiguous, but its lack of comments and trailing-comma intolerance make it awkward to hand-edit. TOML is best for hand-edited application settings: it is the format pyproject.toml uses, supports comments, and has a clear, flat-leaning structure that resists the indentation mistakes YAML invites. YAML earns its place when you genuinely need its features — anchors for de-duplication, or deep nesting that TOML expresses clumsily — but it comes with the safe_load caveat and the whitespace-sensitivity that makes a mis-indented key silently become a string. The last row is the constant across all three: whatever the format, the parsed dict goes through model_validate, so the choice of format never affects the correctness guarantee. This decoupling is genuinely useful in practice: a team can start with a hand-edited TOML file, later switch a machine-generated portion to JSON, or adopt YAML for a section that needs anchors, and the validation model does not change at all — it validates the parsed dict regardless of which loader produced it. The loader is chosen per file by extension; the model is chosen by the shape you want; and the two are independent.

Nested structure is the reason to use a file at all

The whole reason to reach for a structured file instead of environment variables is that files express nesting that flat KEY=value pairs cannot: a routing table, a per-tenant feature map, a list of upstream services each with its own settings. That nesting is exactly what pydantic sub-models are built to validate. A FileConfig with a cache: CacheConfig field means the cache section of the file must itself be a valid cache configuration — host present, port an integer, TLS a boolean — and the model checks the whole tree in one model_validate call. Nesting the models to mirror the file’s structure gives you validation that goes as deep as the data does, so a mistake three levels down in the file is caught with a path like cache.pool.size in the error rather than surfacing as an obscure failure when that specific value is finally read. That precise path is what makes debugging a broken config file quick: instead of “something in the config is wrong”, pydantic tells you “cache.pool.size must be an integer, got ‘ten’”, and the fix is obvious. The deeper and more complex the file, the more valuable this precision becomes, because a large nested file has that many more places for a small mistake to hide.

This is where files and the settings model complement each other cleanly. Flat environment variables map naturally onto a flat model or onto nested sub-models via a delimiter; a structured file maps naturally onto nested sub-models directly. In both cases the model is the validation boundary, and in both cases the goal is the same: turn untyped input — strings from the environment, a dict from a file — into a typed, validated object before any application code touches it. The file’s advantage is expressiveness; the model’s job is to make sure that expressiveness does not come at the cost of correctness. A useful way to think about it: environment variables are good for a flat set of simple values, and structured files are good for genuinely hierarchical configuration, but both should terminate in the same validated object, because the application benefits from a single typed configuration surface regardless of how many sources fed it. Many real services use both — flat variables for the handful of top-level toggles and secrets, a structured file for the rich nested configuration — and merge them into one validated model that the application reads from a single place.

Deployment parity: local to production

1. Local dev — load the committed config.yaml; validate with the model so a malformed edit fails immediately.
2. CI — parse and validate every config file as a test; a broken file fails the build, not the deploy.
3. Staging/Production — mount the same file (or a per-environment overlay) read-only; the identical model validates it on boot.

The CI step is the one that turns file config from a runtime risk into a build-time guarantee. Because the same model validates the file everywhere, a test that simply loads and validates every config file catches a malformed edit — a mis-indented YAML key, a missing required section, a wrong type — on the pull request, not at deploy. This matters more for files than for environment variables, because a config file is edited by hand and its structure is easy to break in ways that only surface when a specific code path reads a specific nested value. Validating the whole file against the model at parse time collapses all of those latent failures into one clear error at load, and the CI test moves that error earlier still, to the commit that introduced it. The test itself is trivial — call load_config on each config file and let a ValidationError fail the test — but its value is outsized, because a config file is the kind of artifact people edit without running the code that consumes it, so a broken edit can otherwise sit unnoticed until deploy. Add the parse-and-validate test to CI and every config change is checked the moment it is committed, exactly as your Python is checked by its own tests.

Security boundaries & guardrails

• yaml.safe_load only — yaml.load and yaml.full_load are forbidden on any file you do not fully control.
• Set a size limit before parsing to defend against billion-laughs / entity-expansion attacks.
• Validate the parsed dict with extra="forbid" so unexpected keys are rejected, not silently kept.
• Keep secrets out of structured config files; reference them from a secret store instead.
• Mount config files read-only in containers.

Beyond safe_load, two guardrails address other file-parsing attacks and hygiene. A size limit before parsing defends against entity-expansion attacks like “billion laughs”, where a small YAML file with nested anchors expands to gigabytes in memory and exhausts the process — checking the file size (and, ideally, using a parser configured to bound expansion) before you parse it caps that risk. The billion-laughs attack is the classic example: ten nested anchors, each referencing the previous ten times, produce a document that is tiny on disk but expands to billions of nodes in memory, hanging or crashing the process. A simple if path.stat().st_size > MAX_CONFIG_BYTES: raise before parsing rejects the pathological file cheaply, and for genuinely untrusted YAML you can go further with a parser configured to limit alias expansion. For a config file you fully control this is belt-and-braces, but for any file an attacker might influence it is a real defence. Keeping secrets out of config files matters because a structured config file is typically committed to version control, so a password in config.yaml is a password in git history; reference secrets from a manager and keep the file free of credentials. A config file’s job is to describe structure — which cache, which routes, which flags — not to hold secrets; the secret values it references should come from a SecretStr field sourced from a manager, so the committed file names the secret without containing it. This keeps the file safe to review, diff, and share, which is much of the point of using a readable structured format in the first place. Mounting config files read-only in containers is the small operational touch that prevents a compromised process from rewriting its own configuration. Together with safe_load and extra="forbid" validation, these turn file config from a broad attack surface into a narrow, well-defended one. None of these guardrails is expensive — a size check is one line, keeping secrets out is a policy, a read-only mount is a container flag — and each closes a specific, known failure mode, so applying all of them is simply the cost of doing file-based configuration responsibly.

Troubleshooting

• ConstructorError on load — the file uses a tag safe_load refuses; that tag is exactly what makes yaml.load dangerous. Remove it.
• Nested value is a string, not a dict — indentation error in YAML; validate with the model to localize the failure. See Handling Nested Configuration in YAML Safely.
• ValidationError: extra fields not permitted — a typo’d key; with extra="forbid" this is caught instead of ignored.
• TOML KeyError in older Python — tomllib is 3.11+; on 3.10 install tomli.

These symptoms cluster around the two rules. A ConstructorError raised by safe_load means the file used a tag the safe loader deliberately refuses — which is exactly the tag that would have made yaml.load dangerous, so the correct fix is to remove the offending tag from the file, never to switch to the unsafe loader to make the error go away. A nested value arriving as a string instead of a dict is almost always a YAML indentation mistake — YAML’s whitespace sensitivity makes it easy to under-indent a block so it collapses into a scalar — and validating with the model localises it: instead of an AttributeError deep in the code when something tries to treat the string as a dict, you get a ValidationError naming the field that has the wrong type, right at load. The extra fields not permitted error is extra="forbid" catching a typo’d key at load time rather than silently ignoring it — which is exactly the behaviour you want, because a config key that is set but ignored is a value the author believed they configured and did not, one of the most confusing bugs to diagnose. And the TOML KeyError is a version reminder — tomllib is standard from Python 3.11, so on Python 3.10 and earlier you install the tomli backport, which exposes the same API. Each of these symptoms points back to the same two-part discipline: use the safe loader for the format, and validate the parsed shape with a model.

Key takeaways

The invariant: structured config is parsed with the safe loader for its format and immediately validated by a model with extra="forbid". Never let an unvalidated dict from a file reach application code. Two independent rules produce that invariant. Parse safely — yaml.safe_load, json.loads, tomllib.loads — so a config file can never execute code, because the full yaml.load will instantiate arbitrary Python objects from a tag in the document. And validate the shape — feed the parsed dict into a pydantic model — so missing keys, wrong types, and typo’d fields are caught at load time rather than at first use.

Getting both right makes the format on disk a free choice. JSON for machine-generated config, TOML for hand-edited settings, YAML for genuinely nested data with anchors — whichever you pick, the same model validates the result, so the correctness guarantee is identical and swapping one format for another changes nothing downstream at all. The format decision is therefore a low-stakes, reversible one about editing ergonomics — comments, whitespace sensitivity, tooling — rather than a high-stakes architectural one, because the safety and validation layers sit above it and do not care which format the bytes arrived in. Add the surrounding guardrails — a size limit against entity-expansion attacks, no secrets in the file, a read-only mount — and structured configuration becomes what flat environment variables cannot be (nested, expressive) without becoming what unvalidated file parsing usually is (a security hole and a source of late failures).

The mental checklist for any config file is short: what loader am I using (must be the safe one for the format), and where is the parsed dict validated (must be a model, before use). If both answers are solid, the file is safe to parse and its structure is guaranteed; if either is missing, you have a latent vulnerability or a latent bug. Wiring the file into the same validated settings object the rest of your configuration flows through means a config file is not a special, dangerous input but simply another source that produces a typed object — no different, from the application’s point of view, from an environment variable that was coerced to its type. Never let an unvalidated dict from a file reach application code.