Dual-Key Rotation for API Keys

The simplest rotation — generate a new key, update the configuration, deploy — has a gap in it. Between the moment the old key stops being accepted and the moment every caller is using the new one, some requests present a credential the other side rejects, and the length of that window is decided by rolling restarts, cached credentials and retry queues rather than by anything you control. Keeping two keys valid at once removes the window entirely. This page covers the sequence and its checkpoints, extending the automated secret rotation patterns topic.

Problem 1: a swap that assumes simultaneity

# ANTI-PATTERN: exactly one key is valid at any moment
API_KEY = settings.api_key.get_secret_value()      # the new one, after the deploy
# ... meanwhile the old pods are still sending the previous key

A rolling deploy replaces instances over minutes, so both keys are in use during that period whether or not both are accepted. Queued work makes it worse: a task enqueued before the rotation may run an hour later with a credential captured at enqueue time, and a retry with backoff can present an old key well after every process has restarted.

The fix is not faster deploys, it is accepting both keys for a period long enough that no in-flight work can still be carrying the old one. The rotation then has no instant where anything can fail, and each individual step is safe on its own.

A single-key swap leaves a failure window With one valid key at a time, the period between the verifier changing and every client restarting is a window where requests fail, while accepting both keys removes the window entirely. one key old key accepted failures new key accepted everywhere two keys old key accepted both accepted new key accepted everywhere the overlap is the whole technique — everything else is sequencing
The middle band is either a failure window or an overlap, depending on one decision.

Problem 2: changing the client before the verifier

# ANTI-PATTERN ordering
1. deploy clients with the new key
2. update the verifier to accept it       # every request between 1 and 2 fails

Ordering decides whether each step is independently safe. If clients start sending a key the verifier does not yet accept, every request fails until step two lands — and if step two has a problem, there is no way back except redeploying every client.

The verifier must always move first. Once it accepts both, clients can migrate in any order, at any pace, with a rollback that is simply “keep using the old key”. That property is what makes the sequence usable across teams: a consumer owned by somebody else can move next week without coordination, because both keys work the whole time.

Problem 3: revoking on a schedule instead of on evidence

# ANTI-PATTERN: "we said 24 hours, so it has been 24 hours"

A calendar-based revocation assumes every consumer restarted within the window. The consumer that breaks this assumption is always the one nobody remembers: a scheduled job that runs weekly, a partner integration, a mobile client with a cached credential, a service in a region that deploys on a different cadence.

Count usage per key instead. When the old key’s counter has been zero for longer than the longest plausible gap between deploys, revocation is a fact rather than a hope — and if the counter is not zero, the number tells you how much traffic you would have broken. Most verifiers can report this; where yours cannot, logging the key identifier (never the key) on each request gives you the same signal.

Secure implementation

# verifier.py — accept either key during the overlap, and count which was used
import hmac
from config import settings
from metrics import counter


def verify(presented: str) -> bool:
    # 1. both keys are optional fields; the old one is absent outside a rotation
    candidates = [("current", settings.api_key)]
    if settings.api_key_previous is not None:
        candidates.append(("previous", settings.api_key_previous))

    for label, key in candidates:
        # 2. constant-time comparison: a timing side channel is a real leak here
        if hmac.compare_digest(presented, key.get_secret_value()):
            counter("api_key_used", labels={"key": label}).inc()   # 3. the revocation signal
            return True
    counter("api_key_rejected").inc()
    return False
# config.py — one required key, one optional predecessor
from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")

    api_key: SecretStr                       # 4. the key clients should be using
    api_key_previous: SecretStr | None = None  # 5. present only during an overlap
# client.py — always sends the current key; nothing here changes during a rotation
def call(path: str) -> Response:
    return httpx.get(
        f"{settings.api_base}{path}",
        # 6. one place reads the key, so migrating is a configuration change only
        headers={"Authorization": f"Bearer {settings.api_key.get_secret_value()}"},
    )

The per-key counter in step 3 is what turns the final step from a judgement into a check, and it costs one line. Label by key identity rather than by value — current and previous, or a short key identifier — so the metric is safe to expose on a dashboard that many people can see.

hmac.compare_digest rather than == is not optional. A naive string comparison returns as soon as it finds a differing byte, so the time it takes leaks how many leading bytes were correct, and an attacker who can measure that can recover a key one byte at a time. It is a one-word change and there is no situation where the ordinary comparison is preferable for a credential.

The four phases of a dual-key rotation The verifier accepts both keys, clients migrate at their own pace, usage of the old key is confirmed to be zero, and only then is the old key revoked. 1. accept both verifier changes first 2. migrate clients any order, any pace 3. confirm zero the counter is the evidence 4. revoke a separate deploy every step is individually reversible, which is what makes the sequence safe
Four steps, each safe alone — the discipline is not skipping the third.

Where the same shape applies

Dual-key rotation is not only for API keys. Signing keys use the same structure with the direction reversed: verifiers must accept signatures from both keys before any signer starts using the new one, so the rollout order is verifiers first, signers second, retirement third. Django’s SECRET_KEY_FALLBACKS is exactly this pattern for session signing, and webhook signature verification usually supports a list of accepted secrets for the same reason.

Database credentials can work this way too, where the engine allows two users or a password change with a grace period, and the same four phases apply — grant the new credential, migrate consumers, confirm the old one is unused, revoke. What changes is the mechanism for “accept both”, which for a database is usually a second user rather than a second password.

The pattern does not fit credentials the far side controls exclusively, such as a third-party API that issues one key at a time and invalidates the previous on rotation. There, the overlap is unavailable and the answer is a coordinated window: rotate during low traffic, restart every consumer promptly, and accept a brief failure period — which is a worse outcome, and worth raising with the provider, because supporting two active keys is a small feature that removes an outage from every customer’s rotation.

One point about automation is worth making explicitly, because the four phases invite it. Phases one, two and four are mechanical and safe to script; phase three is a gate, and scripting it as a timer defeats the entire design. If you automate the sequence, automate it as a state machine that advances on evidence — the counter reading zero — rather than on elapsed time, and have it stop and notify when the evidence does not arrive. An automation that pauses indefinitely because one consumer is still using the old key is doing exactly the right thing, and is far more useful than one that revokes on schedule and pages somebody afterwards.

The related discipline is to run the sequence often enough that it stays working. A rotation path exercised once a year decays: a consumer is added that nobody wires into the metric, a runbook step references a dashboard that moved, the person who understood it changes team. Rotating on a routine cadence — even when nothing has leaked — keeps every part of the mechanism exercised and turns an emergency rotation into a familiar procedure rather than a first attempt under pressure.

Gotchas and version-specific behaviour

  • Always change the verifier first; clients sending a key nothing accepts is an immediate outage.
  • Use a constant-time comparison for every candidate key, not just the current one.
  • Label metrics by key identity, never by key value, so a dashboard is safe to share.
  • Queued and retried work can present an old key long after every process has restarted — size the overlap for the longest retry, not the deploy.
  • Revocation is a separate deploy, so a mistake in it is independently reversible.
  • Leaving the previous key configured after revocation is harmless but misleading; remove the variable in the same change.
  • If the old key’s counter never reaches zero, you have discovered a consumer nobody knew about, which is itself worth the exercise.

Production parity checklist

  • The verifier accepts a current key and an optional previous key, both compared in constant time.
  • A per-key usage counter exists and is visible on the same dashboard as the rotation runbook.
  • The runbook states the four phases and requires evidence, not elapsed time, before revocation.
  • The overlap window is sized by the longest retry or queue delay, not by the deploy duration.
  • Revocation is its own change, after the counter has read zero for a full extra cycle.
  • A rehearsal has been run in staging, end to end, including the revocation step.
Sizing the overlap window The overlap must outlast the rolling deploy, any cached credentials, the longest retry backoff and the slowest-deploying consumer. rolling deploy cached credentials expire longest retry backoff drains the overlap must outlast all of them, plus the consumer that deploys weekly
Size the window by the slowest thing that can still be holding the old key.

Frequently asked questions

Why does a straight key swap cause an outage?

Because the new key and the deploy that uses it never land at exactly the same instant. Any gap — a rolling restart replacing instances over minutes, a cached credential, a task enqueued before the change and executed after it, a retry with backoff — is a window where a request presents a key the other side no longer accepts. The window’s length is determined by infrastructure behaviour rather than by anything in your control, which is why the answer is to remove the window rather than to shrink it.

How long should both keys stay valid?

Long enough for every consumer to have restarted and for the longest retry or queue delay to have drained. A day is a common floor; a week costs almost nothing and covers the consumer nobody remembered — the weekly batch job, the partner integration, the service in a region with a different deploy cadence. Since the only cost of a longer overlap is that an old key remains valid, and that key has not leaked, generosity here is close to free.

Which side has to change first?

The verifier. It must accept both keys before any client starts sending the new one, which is what makes each subsequent step independently safe and reversible. With that ordering, a client can migrate whenever it suits its owners, a rollback is simply continuing to use the old key, and no cross-team coordination is required beyond announcing that the new key is available.

How do I know it is safe to revoke the old key?

Count usage per key. Revocation is safe when the old key’s counter has read zero for longer than the longest plausible gap between deploys — which is evidence, where a calendar date is an assumption. If the counter is not zero, it also tells you how much traffic a scheduled revocation would have broken, and identifies a consumer you did not know about. That discovery alone often justifies the metric.

Does this work for signing keys as well?

Yes, and the shape is the same with the direction reversed: verifiers must accept signatures from both keys before any signer starts using the new one, so the verifier rollout comes first and the signer migration second. Django’s SECRET_KEY_FALLBACKS implements exactly this for session signing, and most webhook verification supports a list of accepted secrets for the same reason. The four phases and the evidence-based revocation are unchanged.

Key takeaways

Rotation without downtime comes from accepting two credentials at once, not from tighter coordination. The verifier moves first and accepts both, clients migrate at whatever pace suits them, and no instant exists where an in-flight request can present a credential the other side refuses — which is what makes each step reversible on its own.

The step that gets skipped is the third one. Counting usage per key turns revocation from a date on a calendar into a fact you can point at, and the counter that refuses to reach zero is how you discover the weekly job or the partner integration nobody listed. Size the overlap by the longest retry rather than the deploy duration, make revocation its own change, compare keys in constant time, and rehearse the whole sequence in staging once so the runbook describes something that has actually been done. The wider rotation practices are on the automated secret rotation topic page.

Back to Automated Secret Rotation Patterns