Renew Vault Leases in Long-Running Workers

A short-lived request handler authenticates to Vault, reads a secret, and exits — nothing to renew. A long-running worker is different: its token and its dynamic database lease both expire on a TTL, and if it never renews them it starts throwing permission errors an hour into the job. This page keeps a HashiCorp Vault session alive with hvac, complementing the dynamic database credentials and AppRole workflows.

Problem 1: authenticate once, never renew

# ANTI-PATTERN
client.auth.approle.login(role_id=RID, secret_id=SID)   # token expires in 1h
while True:
    do_work(client)   # PermissionError once the token TTL lapses

The token silently expires mid-job and every subsequent call fails.

Problem 2: ignoring the database lease

Even with a live token, the dynamic database credential has its own lease. When it expires the database rejects the connection, even though Vault itself is still reachable.

Secure implementation

Renew the token and the lease before expiry, and re-authenticate on failure.

# vault_session.py
import time
import hvac


class VaultSession:
    def __init__(self, url, role_id, secret_id):
        self.url, self.role_id, self.secret_id = url, role_id, secret_id
        self.client = hvac.Client(url=url)
        self._login()

    def _login(self):
        self.client.auth.approle.login(role_id=self.role_id, secret_id=self.secret_id)

    def maintain(self, lease_id: str):
        """Call periodically from the worker loop (e.g. every 60s)."""
        try:
            self.client.auth.token.renew_self()          # extend the token TTL
            self.client.sys.renew_lease(lease_id=lease_id)  # extend the secret lease
        except hvac.exceptions.InvalidRequest:
            self._login()                                # token/lease gone -> re-auth
            return "reauthenticated"                     # caller should refetch + reconnect
        return "renewed"

Renew at half the TTL from the worker loop; on InvalidRequest, re-authenticate, fetch a fresh credential, and reconnect the database client.

Gotchas & version-specific behaviour

• A lease has a max TTL; past it, renewal fails and you must fetch a new dynamic credential.
• renew_self extends the token but not any secret leases — renew both.
• Recreate the database engine/connection pool after refetching credentials; old connections keep the expired password.
• Add jitter to the renewal interval so a fleet of workers does not renew in lockstep.
• Handle Vault being briefly unreachable with retries; do not crash the worker on one failed renew.

Production parity checklist

• Renew token and lease at roughly half their TTL from the worker loop.
• Re-authenticate and reconnect on InvalidRequest rather than crashing.
• Rebuild the connection pool when the dynamic credential is replaced.
• Add jitter to renewal timing across the worker fleet.
• Alarm on repeated re-authentications, which signal a TTL that is too short.

Key takeaways

A long-running worker must renew both its Vault token and its secret lease before expiry, and re-authenticate then rebuild its connections when a lease reaches its maximum TTL.