GitLab CI config validation stage

GitLab pipelines run stages in order, which makes a validate stage before deploy the natural gate for configuration. This page wires it up, extending CI/CD Config Validation.

GitLab’s distinctive contribution to this problem is its variable model. Where other platforms give you one bucket of secrets and expect the pipeline to pick correctly, GitLab attaches flags and scopes to each variable: Masked controls whether it can appear in logs, Protected controls whether unprotected branches can see it at all, and an environment scope controls which jobs receive it. Used together, those three settings mean a merge request from a feature branch physically cannot read the production database password — not by convention, but because the runner never receives it.

GitLab variable flags and what each one prevents Masked stops a value printing in job logs, Protected stops unprotected branches receiving it, and environment scope limits which jobs see it. three independent controls on one variable Masked value is replaced with [MASKED] in job output Protected only jobs on protected branches and tags receive it environment scope only jobs declaring that environment receive it A feature-branch merge request never receives a protected production variable at all.
Masking is about logs; Protected and environment scope are about which runner ever sees the value.

Problem 1: deploy stage runs regardless

# ANTI-PATTERN: no validate stage; deploy runs even with broken config
stages: [build, deploy]
deploy:
  stage: deploy
  script: ./deploy.sh

Nothing checks the configuration before the deploy script boots the app. The pipeline’s only opinion about correctness is “did the build succeed”, which says nothing about whether the environment the artifact is heading into has the keys it needs.

A partial version of this mistake is more common than the complete absence: a validate job exists but sits in the same stage as deploy, so the two run in parallel. GitLab’s stage ordering is the mechanism that makes sequencing work, and a job in the wrong stage is a job that reports rather than gates. Reading the stages: list and asking “is validation strictly before every deploy job?” catches it immediately.

GitLab’s pipeline graph makes this unusually easy to audit visually. The pipeline view draws jobs in columns by stage with dependency edges between them, so a validation job that gates nothing is visible as a column with no outgoing edge to the deploy job. Spending thirty seconds on that graph after any pipeline restructuring is a cheap habit, and it catches the class of mistake that no test can — a change to the pipeline definition itself that removes an ordering guarantee while leaving every job green.

Problem 2: secrets in plain CI variables

# ANTI-PATTERN: unmasked, unprotected variable visible in job logs
variables:
  API_KEY: "sk_live_xxxx"     # in the repo, in logs, in every fork

A secret defined inline in .gitlab-ci.yml lives in the repository and the logs. It is also, unavoidably, in the history of that repository and in every fork made since it was added — so the fix is not deleting the line but rotating the credential and then deleting the line.

The variables: block is fine for what it is meant to hold: non-secret configuration that belongs with the pipeline definition, such as an image tag, a Python version, or a feature flag for the pipeline itself. The distinction to hold on to is that anything in .gitlab-ci.yml is code and gets code’s exposure — readable by everyone with repository access, recorded in history, and copied by forks. Anything that must not have that exposure belongs in project CI/CD variables with the flags set.

What belongs in the pipeline file versus in CI/CD variables Non-secret pipeline configuration such as image tags belongs in the variables block, while credentials belong in masked and protected project variables. .gitlab-ci.yml variables: PYTHON_VERSION: "3.12" PIP_CACHE_DIR: .cache/pip non-secret, reviewed with the code exposure: anyone who can read the repo project CI/CD variables DATABASE_URL, API_KEY Masked + Protected scoped to an environment exposure: only matching jobs
The pipeline file is code and carries code's exposure — credentials need the variable store's flags instead.

Secure implementation

# .gitlab-ci.yml
stages: [validate, deploy]

validate-config:
  stage: validate
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - gitleaks detect --no-banner
    - python -m ci.validate_config        # constructs Settings(); fails on error
  # DATABASE_URL and API_KEY come from masked, protected CI/CD variables.

deploy:
  stage: deploy
  script: ./deploy.sh
  needs: ["validate-config"]              # blocked until validation passes
  environment: production

Masked, protected CI/CD variables supply the secrets; the validate stage constructs the model and runs gitleaks; deploy declares needs so it cannot start until validation succeeds. The needs: key is doing something slightly different from GitHub’s here — stage ordering alone would already sequence the two jobs — but declaring it makes the dependency explicit and survives a later refactor that moves jobs between stages.

Getting per-environment validation right means the validation job must also declare an environment, so it receives that environment’s scoped variables:

# validate each target with that target's own variables
.validate: &validate
  stage: validate
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - python -m ci.validate_config

validate:staging:
  <<: *validate
  environment: staging          # receives staging-scoped variables

validate:production:
  <<: *validate
  environment: production       # receives production-scoped variables
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy:production:
  stage: deploy
  script: ./deploy.sh
  environment: production
  needs: ["validate:production"]

The YAML anchor keeps one definition of what validation is while letting each target supply its own variables. Without the environment declaration on the validation job, GitLab hands it the unscoped defaults, and you are back to the failure this page opened with: a green check that examined something other than the deployment target.

Per-environment validation jobs feeding their own deploys A shared validation definition is instantiated once per environment, each receiving that environment's scoped variables, and each deploy job needs its matching validation job. .validate one definition validate:staging staging-scoped vars validate:production production-scoped vars deploy:staging needs its own validation deploy:production needs its own validation
One validation definition, instantiated per environment — each deploy gated by the check that saw its own variables.

Running validation on merge requests

Catching a configuration error at merge-request time is better than catching it at deploy time, and GitLab’s rules: keyword is how you express when a job runs. The complication is that merge-request pipelines usually run on unprotected branches, which is exactly the situation where Protected variables are withheld — so a naive “run validation on every merge request” produces a job that always fails.

The arrangement that works splits the question in two. Structural validation — does the model construct, are the field types coherent, does extra="forbid" still hold — needs no real secrets and can run on every merge request against placeholder values. Environment validation needs real variables and runs only where they are available.

# structural checks everywhere, environment checks only where secrets exist
validate:model:
  stage: validate
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - pytest tests/test_settings.py      # model's own tests; no secrets needed
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

validate:production:
  stage: validate
  image: python:3.12
  environment: production
  script:
    - python -m ci.validate_config       # needs protected variables
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The first job gives contributors fast, meaningful feedback on every change: adding a required field without updating the tests fails immediately, and so does relaxing forbid or turning a SecretStr into a plain string. The second job asserts that the production environment actually satisfies the model, and it only runs where that question can be answered honestly.

Splitting this way also fixes a social problem. A check that fails for reasons a contributor cannot influence teaches them to ignore red checks in general, which erodes every other gate in the pipeline. Keeping the always-runnable checks separate from the environment-dependent ones means a red check on a merge request is always something the author can act on.

Splitting structural checks from environment checks Model tests run on every merge request without secrets, while environment validation runs only on the default branch where protected variables are available. two questions, two jobs, two rule sets validate:model runs on every merge request no secrets required "is the model still strict?" a contributor can always fix it validate:production default branch only protected variables available "is production configured?" gates the deployment
A contributor should never face a red check they have no way to make green.

Gotchas & version-specific behaviour

  • Mark secret CI/CD variables Masked and Protected so they only appear on protected branches and never print.
  • needs: enforces the ordering even when stages would otherwise parallelize.
  • Use rules: to run validation on merge requests so config errors are caught pre-merge.
  • Scope environment: production so deploy-time variables match the target.
  • Masking has format requirements — a value with newlines, or shorter than the minimum length, silently cannot be masked, and GitLab will tell you at save time.
  • A variable that is Protected but referenced by a merge-request pipeline simply arrives empty, producing a “field required” error that looks like a missing key rather than a permissions issue.

That last point is the most confusing failure in GitLab pipelines, and it is worth recognising by its shape. Validation passes on the default branch and fails on every merge request with the same missing-key error, which looks like a bug in the model but is the Protected flag working exactly as designed. The correct response is not to unprotect the variable — that would hand production credentials to any branch — but to run the merge-request pipeline against non-production values, or to split the job so that merge requests validate the model’s structure while only protected branches validate against real environments, which is the split described in the previous section.

Making the job fast enough to keep

GitLab jobs start from a clean container every time, so the dependency install dominates a validation job that otherwise takes under a second. Left unaddressed, a check that should be instant costs a minute or two on every pipeline, and the pressure to move it “later in the pipeline” — where it stops gating anything — grows accordingly.

validate-config:
  stage: validate
  image: python:3.12
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  cache:
    key:
      files: [requirements.txt]      # cache invalidates when deps change
    paths: [.cache/pip]
  timeout: 5m                        # a hung gate is a broken gate
  script:
    - pip install -r requirements.txt
    - python -m ci.validate_config

Keying the cache on requirements.txt is what makes it correct rather than merely fast: the cache is reused while dependencies are unchanged and rebuilt the moment they are not, so there is no stale-dependency failure mode. Setting PIP_CACHE_DIR inside the project directory matters because GitLab caches paths relative to the build directory, and pip’s default cache location lives outside it, so leaving the variable unset produces a cache configuration that looks correct in the YAML and silently caches nothing at all on every run.

The timeout: entry addresses a different risk. A job that hangs — waiting on a network call in a validator, or on an interactive prompt from a mis-invoked tool — otherwise occupies a runner until the project’s default timeout expires, which is typically an hour. For a job that should finish in seconds, a short timeout converts a hang into a fast, obvious failure and frees the runner for everyone else.

The deeper version of “keep it fast” is to keep it hermetic. A validation job that reaches the network for anything other than the dependency install is a job that will fail for reasons unrelated to configuration, and every such failure teaches people to hit retry. Connectivity checks — can we actually reach the database at this URL — are worth running, but as a separate job whose flakiness cannot contaminate the deterministic check that gates the deploy.

Keeping the validation job fast and deterministic Caching the dependency install, setting a short timeout, and avoiding network calls keep the job quick and its failures meaningful. three settings keep the gate credible cache: key on deps install is the slow part rebuilds when deps change seconds, not minutes timeout: 5m a hang becomes a failure runner freed immediately no silent hour-long stalls hermetic script no network in validators connectivity is a separate job red always means red
A gate people retry reflexively has already stopped being a gate — speed and determinism are what prevent that.

Production parity checklist

  • A validate stage precedes deploy, joined by needs.
  • Secrets are Masked + Protected CI/CD variables, never inline.
  • gitleaks runs in the validate stage.
  • Validation runs on the production Python version.
  • extra="forbid" catches stray injected variables.
  • Every validation job declares the environment whose variables it is checking.

The Python version item is easy to overlook and occasionally decisive. Validation running on 3.11 while production runs 3.12 means the check exercises a different interpreter, and while configuration parsing rarely differs between minor versions, dependency resolution does — a pydantic version pinned for 3.12 may resolve differently on 3.11 and validate with subtly different behaviour. Pinning the job’s image: to the same version the deployment uses removes the question, and using the same base image as the production Dockerfile removes it entirely.

gitleaks in the validate stage is worth one clarification about scope. GitLab clones with a shallow depth by default, so a scan in an ordinary job examines only recent history — fine for “does this change introduce a secret?”, useless for “is there anything already in here?”. Setting GIT_DEPTH: 0 on a scheduled job gives the scanner the full history for a periodic sweep, while leaving normal pipelines fast. Running both jobs — a shallow scan on every pipeline and a deep scan on a schedule — is the arrangement that actually covers the question.

Key takeaways

A validate stage with needs makes correct configuration a precondition for deploy in GitLab. What distinguishes a working setup from a decorative one is that the validation job declares the same environment as the deploy job, so it receives that environment’s scoped variables, and that secrets live in Masked and Protected project variables rather than in the pipeline file. Get those right and the confusing failures — green validation followed by a broken deploy, or a merge request that fails on a key the default branch resolves fine — stop being mysteries and become predictable consequences of the variable model. If you are adopting this incrementally, the order that pays off fastest is: move any inline secret into a Masked and Protected variable and rotate it, add the validate stage with a needs: edge from every deploy job, then split the merge-request checks from the environment checks so contributors always see a check they can act on. Each step is independently useful, and none of them requires touching the application. For the GitHub Actions version, see Validate Config in GitHub Actions Before Deploy.

Back to CI/CD Config Validation