Implementing custom validators for AWS ARNs and URLs

A typed str field accepts "arn:aws:lol" and "http://internal" without complaint — both are strings. Only a domain validator catches a malformed ARN or a non-TLS URL at startup instead of at the first AWS call. This page writes those validators, extending Custom Validators & Constraints.

An ARN is a structured identifier with a fixed shape — arn:partition:service:region:account-id:resource — so “is this a valid ARN?” has a precise, checkable answer that a plain str type ignores entirely. The same is true of the webhook and callback URLs a service is configured with: the rule “must be an HTTPS URL to a real host” is knowable at startup. Validating both up front converts what would otherwise be a runtime failure — an opaque AWS InvalidParameter, a webhook that silently posts over plaintext — into a clear boot-time error naming the field.

This page pairs that goal with a security concern specific to hand-written format checks: regular-expression denial of service. An ARN validator is a natural place to reach for a regex, and a carelessly written pattern can be made to hang on crafted input. So the page shows not just a validator but a ReDoS-safe one — anchored, with bounded character classes and no nested quantifiers — and explains why that structure matters.

It is worth being honest about the threat level, because ReDoS is often overstated for configuration. Config values are typically set by operators you trust, not attackers, so a pathological ARN pattern is unlikely to be handed a malicious input in normal operation. But two things keep it from being a non-issue: an ARN can arrive from a config file or a fetched fragment that an attacker might influence in some deployments, and even a non-malicious but unusually long or nearly-matching value can trigger the exponential behaviour by accident. Since writing the safe pattern costs nothing over writing the unsafe one — it is the same amount of code and effort, just structured differently — there is no reason to leave the risk in. Treat a ReDoS-safe pattern as the default way you write validators, not a special hardening step.

Problem 1: a plain str accepts anything

# ANTI-PATTERN: malformed ARN passes validation, fails at runtime
class Settings(BaseSettings):
    topic_arn: str        # "arn:aws:lol" is a valid str — and a broken ARN

The error surfaces later as an opaque AWS InvalidParameter, far from the config that caused it. "arn:aws:lol" is a perfectly good Python string, so the model constructs and the process starts; the malformed ARN only bites when the code finally calls SNS, SQS, or IAM with it, at which point AWS returns an InvalidParameter or ValidationException whose message is about the API call, not your configuration. The value that was wrong at startup is diagnosed at runtime, in a different subsystem, by whoever is on call — a long way from the actual cause, which was that someone truncated TOPIC_ARN when copying it between environments. Checking the ARN’s structure at load time keeps the failure where the cause is, at startup, with a message that names the field.

The reason a str type cannot help here is that an ARN’s validity is about its internal structure, not its Python type. Every ARN has six colon-separated fields, and each carries meaning: the literal arn, the partition (aws, aws-us-gov, aws-cn), the service, the region, the twelve-digit account id, and the resource. A value that is missing fields, has an empty account id, or uses a bogus service is still a str and still passes the type check — only a validator that understands the six-field shape can reject it. The webhook URL has the same character: http://internal is a valid str and a valid URL, but it is plaintext, so a value that really should have been https:// sails right through the type system and quietly sends callbacks over an unencrypted channel. In both cases the type is satisfied and the meaning is wrong, which is precisely the gap a domain validator exists to close.

Problem 2: a greedy regex that backtracks

# ANTI-PATTERN: nested quantifiers — catastrophic backtracking (ReDoS)
ARN_RE = re.compile(r"^(arn:(.*)+:.*)+$")   # can hang on long malicious input

Unanchored, nested-quantifier patterns can take exponential time on crafted input. The pattern ^(arn:(.*)+:.*)+$ contains quantifiers nested inside quantifiers — a + applied to a group that itself contains (.*)+ — and that structure is the signature of catastrophic backtracking. When the regex engine meets an input that almost matches but ultimately fails, it tries an explosively growing number of ways to partition the string among the nested quantifiers before giving up. A modest input of a few dozen characters can take seconds; a slightly longer one can take minutes, pinning a CPU core the whole time. Because a validator runs synchronously during model construction, a hang there stalls startup or, if a config value can be influenced by an attacker, hands them a denial-of-service primitive.

The fix is structural, not a matter of care: anchor the pattern with ^ and $ so the engine is not retrying at every offset, and never nest quantifiers. An ARN has a fixed number of colon-separated segments, so you match each segment with a bounded character class ([a-z0-9-]+, \d{12}) rather than a greedy .* that can overlap with its neighbours. A pattern built that way runs in linear time on any input, malicious or not, because there is exactly one way to attempt the match.

Secure implementation

# config/identifiers.py
import re
from pydantic import field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict

# Anchored, linear-time: each segment is a bounded character class.
ARN_RE = re.compile(r"^arn:aws:[a-z0-9-]+:[a-z0-9-]*:\d{12}:[\w\-/:.*]+$")

class Settings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")
    topic_arn: str
    webhook_url: str

    @field_validator("topic_arn")
    @classmethod
    def check_arn(cls, v: str) -> str:
        if not ARN_RE.match(v):
            raise ValueError(f"topic_arn is not a well-formed ARN: {v!r}")
        return v

    @field_validator("webhook_url")
    @classmethod
    def check_https(cls, v: str) -> str:
        if not v.startswith("https://"):
            raise ValueError("webhook_url must use https")
        return v

The ARN pattern is anchored with bounded character classes, so it runs in linear time. The URL check enforces TLS. Both raise a precise error naming the field and the bad value.

Reading the ARN pattern segment by segment shows how it maps onto the six-field ARN structure. ^arn:aws: fixes the literal prefix and partition; [a-z0-9-]+: matches the service (a required, bounded token); [a-z0-9-]*: matches the region (which can be empty for global services like IAM, hence * not +); \d{12}: requires exactly a twelve-digit account id; and [\w\-/:.*]+$ matches the resource, which is the most free-form segment and varies widely by service. Each piece corresponds to one field of the ARN and uses a bounded class, so the pattern both documents the ARN shape and stays linear-time. If you target GovCloud or China regions you widen the partition (aws-us-gov, aws-cn), but the segmented structure is the same and the pattern stays linear-time.

The webhook URL check is deliberately not a regex — v.startswith("https://") plus a host check is simpler, faster, and impossible to make backtrack. This is the general lesson: use a regex only where you genuinely need pattern matching (the segmented ARN), and prefer plain string operations everywhere a startswith, split, or length check will do. A validator built mostly from string methods, with a single anchored regex where structure demands it, is both the safest and the most readable option.

How strict to make the ARN pattern is a real judgement call worth naming. You can validate loosely — “six colon-separated parts, non-empty where required” — which catches truncations and typos while accepting any service and region. Or you can validate strictly — pinning the exact service (sns, sqs) and even the account id — which catches an ARN pointed at the wrong resource but risks rejecting a legitimate value when you add a new service or move accounts. The right level depends on the field: a topic_arn that should always be an SNS topic can afford to pin :sns: in the pattern and reject anything else, turning “someone pasted a queue ARN into the topic variable” into a startup error instead of a runtime publish failure. A general “any ARN” field should validate only the structure. Choosing the tightest pattern the field’s meaning allows is what turns a validator from a shape check into a genuine correctness check.

Gotchas & version-specific behaviour

• In pydantic v2 use @field_validator with @classmethod; the v1 @validator is deprecated.
• For URLs you can also use pydantic’s HttpUrl/AnyUrl types, but a custom check gives a clearer message and lets you enforce https only.
• Anchor every pattern (^...$) and avoid nested quantifiers to stay ReDoS-safe.
• Partition (arn:aws-us-gov:) and Suffix segments vary — widen the service/region classes if you target GovCloud or China regions.

The partition gotcha is the one that bites teams operating outside the commercial AWS partition. A pattern anchored on ^arn:aws: will reject a perfectly valid arn:aws-us-gov:... or arn:aws-cn:... ARN, so if you deploy to GovCloud or China you must widen the partition segment to accept those values — arn:(aws|aws-us-gov|aws-cn):. This is exactly the kind of “too strict” validator failure the CI fixture is meant to catch: it rejects a legitimate value that a real environment carries, and a test that builds the model with a representative GovCloud ARN surfaces it before deploy rather than blocking a production rollout. The HttpUrl/AnyUrl note is the URL analogue of the DSN advice — the built-in types do parse and validate URL structure well, but a custom startswith("https://") check gives a clearer message and lets you require TLS specifically, which the general-purpose type does not know to require in the first place.

Production parity checklist

• Every ARN and URL field has an anchored, linear-time validator.
• URLs enforce https://; no plaintext endpoints accepted.
• extra="forbid" rejects stray keys.
• A CI fixture builds the model with valid and invalid values to guard the regex.
• Error messages name the field and the offending value.

One caution on that last item: naming the offending value is helpful for an ARN or a webhook URL, which are not secret, but the same instinct is dangerous for a field that embeds a credential. Keep the “name the value” habit for identifiers that are safe to log, and redact anything sensitive — the rule is to make the error actionable without turning the log into a disclosure. For ARNs and public webhook URLs, echoing the bad value is exactly right, because it lets an operator spot the truncation or typo at a glance instead of comparing the printed value against what they intended to set. The distinction to internalise is between identifiers and credentials: an ARN, a region, a public webhook URL are identifiers and safe to echo; a password, a token, or a URL that embeds one is a credential and must be redacted. Applying that split per field keeps error messages maximally useful for the fields where it is safe and closed for the fields where it is not.

Key takeaways

Anchored, ReDoS-safe validators turn malformed ARNs and URLs into a clear startup failure instead of a runtime mystery. An ARN has a fixed six-field structure, so a validator that matches each field with a bounded, anchored segment both documents the shape and rejects the malformed values a str type ignores — a truncated ARN, a missing account id, a bogus service. And because ARN validation invites a regex, it is the natural place to get regex safety right: anchor the pattern, use bounded character classes, and never nest quantifiers, so a crafted or merely unlucky input cannot turn your validator into a denial-of-service that stalls the whole process.

The broader habit is to reach for a regex only where structure genuinely demands it and to prefer plain string operations everywhere else — startswith("https://") for a URL, a split and length check for an ARN’s coarse shape. Back the validators with a CI fixture that exercises a valid ARN, a GovCloud ARN if you target that partition, a plaintext URL, and a malformed value, so a too-strict or too-loose pattern fails the build before it can block or slip past a deploy. For URL-specific database and cache identifiers, see Validate Database and Redis URLs with Pydantic.