Read Secrets from Google Secret Manager in Python

Secret Manager is explicit where other stores are implicit: you address a version, not a secret, and the API hands back bytes rather than a string. Both details are improvements, and both trip up code ported from elsewhere — a missing decode, a latest alias where a pin was intended, or a permission that lets you see a secret exists without letting you read it. This page covers reading secrets into a validated settings model correctly, extending the Azure and Google secret stores topic.

Problem 1: assuming latest means the same thing everywhere

# ANTI-PATTERN for a signing key: two pods can hold different values
path = f"projects/{project}/secrets/{name}/versions/latest"
key = client.access_secret_version(name=path).payload.data

latest resolves at call time to the highest-numbered enabled version. For a database password rotated on a schedule that is exactly right — restarting pods pick up the new value with no code change. For a signing key it is a bug waiting for a rotation: a pod started before the rotation signs with version 3, a pod started after signs with version 4, and whichever component verifies signatures rejects half of them.

Decide per secret and make the decision visible. Values consumed independently by each process — passwords, API tokens — suit latest. Values that must agree across processes, or that a rollback has to reproduce exactly, need a pinned version number, which then becomes a deployment parameter that changes deliberately rather than a value that drifts.

Problem 2: the permission that is not the one you granted

# ANTI-PATTERN: a role that lets you see the secret but not read it
roles/secretmanager.viewer      # metadata only — access_secret_version is refused

Listing secrets and viewing their metadata is a different permission from accessing a version’s payload. A principal granted the viewer role can see the secret in the console, confirm it exists, check when it was updated — and receive PermissionDenied from access_secret_version. Because everything visible suggests access is working, the diagnosis usually starts in the wrong place.

Grant the accessor role, and grant it on the individual secret rather than the project. Project-level access means a workload that needs one value can read every secret in the project, which is both broader than intended and impossible to audit meaningfully. Per-secret bindings take one line each and give you a true answer to “which workloads can read this?”.

Problem 3: treating the payload as a string

# ANTI-PATTERN: bytes where a string is expected, plus a trailing newline
token = client.access_secret_version(name=path).payload.data     # this is bytes
headers = {"Authorization": f"Bearer {token}"}                   # b'...' in the header

The API returns payload.data as bytes, deliberately, because a secret may hold binary material. An f-string will happily interpolate the bytes repr, producing a header containing b'...' and an authentication failure that looks like a wrong credential. Decoding without stripping produces the other common failure: values created by piping a file carry a trailing newline that most APIs reject.

Decode and strip in one place, in the adapter, so every field arrives clean regardless of how it was written. For genuinely binary material — a certificate, a key file — keep the bytes and store them base64-encoded in the secret, decoding once in the adapter so both providers behave identically.

Secure implementation

# gcp_secrets.py — decode, strip, and name the secret in every error
import logging
from google.api_core import exceptions as gexc
from google.cloud import secretmanager

logger = logging.getLogger(__name__)


class SecretUnavailable(RuntimeError):
    pass


class SecretManagerReader:
    def __init__(self, project_id: str) -> None:
        # 1. application default credentials: workload identity in GCP, gcloud locally
        self._client = secretmanager.SecretManagerServiceClient()
        self._project = project_id

    def fetch(self, names: dict[str, str], versions: dict[str, str] | None = None) -> dict[str, str]:
        """{secret name: field name} -> {field name: value}, versions optional per secret."""
        versions = versions or {}
        out: dict[str, str] = {}
        for secret, field in names.items():
            # 2. an explicit version where pinning matters, latest otherwise
            version = versions.get(secret, "latest")
            path = f"projects/{self._project}/secrets/{secret}/versions/{version}"
            try:
                payload = self._client.access_secret_version(name=path).payload.data
            except gexc.NotFound:
                raise SecretUnavailable(f"secret '{secret}' version '{version}' not found") from None
            except gexc.PermissionDenied:
                # 3. name the permission, because viewer-versus-accessor is the usual cause
                raise SecretUnavailable(
                    f"no access to '{secret}': the accessor permission is required, not viewer"
                ) from None
            out[field] = payload.decode("utf-8").strip()   # 4. bytes in, clean string out
        return out

# config.py — versions recorded beside the names they apply to
from pydantic import SecretStr, PostgresDsn
from pydantic_settings import BaseSettings, SettingsConfigDict

SECRET_NAMES = {
    "app-database-url": "database_url",
    "app-signing-key": "signing_key",
}
SECRET_VERSIONS = {
    "app-signing-key": "3",        # 5. pinned: every process must agree on this one
}                                   #    database-url is unlisted, so it uses latest


class Settings(BaseSettings):
    model_config = SettingsConfigDict(extra="forbid")
    environment: str = "local"
    database_url: PostgresDsn
    signing_key: SecretStr


def load(project_id: str | None) -> Settings:
    if project_id is None:
        return Settings()                                   # 6. local: env vars and .env
    reader = SecretManagerReader(project_id)
    return Settings(**reader.fetch(SECRET_NAMES, SECRET_VERSIONS))

Keeping SECRET_VERSIONS as a separate dictionary rather than folding versions into the names mapping means the pinning decisions are visible as a short list. Anyone can read it and see which values are deliberately frozen, and bumping a pin is a one-line change that appears in a diff — which is precisely the property you want for a decision that would otherwise live only in somebody’s memory of a rotation two quarters ago.

The PermissionDenied message earns its length. Naming the distinction between viewer and accessor in the error itself saves the next person the twenty minutes it takes to discover that the console showing the secret does not mean the service can read it. Error messages that encode the most likely cause are cheap to write once and pay back every time somebody hits them.

Quotas, cost and how many secrets to create

Access operations are quota-limited per project and billed per call, so the caching rule from the parent topic applies here without modification: fetch at startup, refresh on a timer, never read on the request path. The number worth internalising is that a service handling a hundred requests per second and reading one secret per request generates millions of access operations per day, which is both an unnecessary bill and a reliable way to meet the quota.

A related decision is granularity — one JSON blob holding every value, or one secret per credential. A blob is fewer access operations and one version history, which sounds efficient until you rotate. Rotating one credential inside a blob creates a new version of everything, so every consumer of every value in that blob is affected, and the audit trail cannot distinguish which value somebody actually needed. Separate secrets cost one access operation each at startup — a handful, once per process — and keep rotation, access grants and audit records aligned with the individual credential.

The reasonable middle ground is to group values that genuinely change together. A database’s host, user and password rotate as a set when the instance is replaced, so a blob for that group is coherent; the payment API token that rotates on its own schedule is a separate secret. Making that judgement per group, rather than choosing one style for everything, gives you the small operation count where it is safe and the independent rotation where it matters.

Gotchas and version-specific behaviour

• payload.data is bytes; decode explicitly and strip whitespace before the value reaches your model.
• latest resolves at call time — two processes started either side of a rotation can hold different values.
• Destroying a version is irreversible, unlike disabling it, and a pinned reference to a destroyed version fails permanently.
• Viewer and accessor are different permissions; the console showing a secret says nothing about payload access.
• Grant per secret rather than per project, or a workload that needs one value can read them all.
• Application default credentials resolve workload identity in GCP and a user session locally, so one code path covers both.
• Fetching many secrets sequentially at startup costs a round trip each; group or parallelise if the count grows past a handful.
• Data access audit logs are configured separately from admin activity logs, so enable them explicitly before you need them.

Production parity checklist

• Deployed workloads authenticate with a workload identity; no key file is distributed.
• The accessor permission is granted per secret, and the grant list is reviewed as an inventory.
• Every value is decoded from bytes and stripped in the adapter, not at call sites.
• Version policy is recorded in a visible mapping, with pins for anything processes must agree on.
• Secrets are fetched at startup and refreshed on a timer, never per request.
• Errors name the secret and, for permission failures, the role distinction that usually causes them.

Key takeaways

Secret Manager’s explicitness is its strength and the source of every porting mistake. You address a version, so decide per secret whether latest or a pin is correct and record that decision where the names live. You receive bytes, so decode and strip in the adapter rather than at call sites. And access to a payload is a distinct permission from seeing that a secret exists, which is worth encoding directly into your error messages.

Everything else follows the pattern shared with the other stores: authenticate with a platform identity so nothing in configuration needs rotating, fetch once at startup so quotas and latency stay off the request path, grant read access per secret rather than per project, and let the validated model handle typing and masking once the values arrive. See the Azure and Google secret stores topic for the comparison with Key Vault’s version semantics.