Secrets Management

The 30-second version

Here's a story that happens every week somewhere. A developer is testing an upload feature, pastes an AWS access keyAPI keyA long random string a service gives you so your program can call its API. Whoever sends the key is treated as you, so it is a password for a program. into config.js "just for now", and pushes the repo to a public GitHub project. Ten minutes later they notice, delete the line, and push again.

It's already too late. Automated bots watch GitHub's public feed of new commits and grep every one for key patterns. Researchers who planted keys on purpose saw them found and tried within minutes (exact times vary from seconds to an hour). By the time the developer deletes the line, someone may already be starting expensive servers on their account.

Two lessons are already hiding in that story:

1. Deleting the line doesn't un-leak the key. Git keeps every old version of every file (§02 shows exactly why). The only real fix is to make the leaked key stop working: you rotateKey rotationRegularly replacing a key or credential with a new one (and retiring the old), so a leaked key stops working and the damage window stays short. it.
2. The mistake was having a long-lived key in a file at all. Most of this topic is about designs where there's nothing worth stealing in the repo in the first place.

What counts as a secret?

If it would let a stranger read data, spend money, or pretend to be your service, it's a SecretSecretAny value that grants access if someone else gets it: passwords, API keys, database credentials, private keys, signing keys, tokens.:

Not secret: your database host name, a feature flag, a public key, or an ID that's useless without a password. Mixing secrets with ordinary config is how they end up printed in logs, so it pays to keep the two apart.

Three jobs, plus one that runs through them all

The fourth job runs through everything: know who read what, and when (an audit log), so that when something leaks you can answer "what did they touch?". §11 puts it all together at company scale.

Git never forgets

Each commitCommit (git)A saved snapshot of every file in the repo, plus a pointer to the previous snapshot. Old commits are never changed; a new commit only adds a new snapshot on top. is a full snapshot of your files plus a pointer to the commit before it. Git never edits an old commit. "Removing" a key just adds a new snapshot without it, on top of the old one.

Let's prove it. Here's the exact sequence from §01, run in a fresh repo on this machine (the key is AWS's official fake example, AKIAIOSFODNN7EXAMPLE):

$ git commit -m "add config"             # config.js contains the key
$ # …edit config.js to read process.env instead…
$ git commit -am "remove hardcoded keys"
$ grep -rn AKIA config.js                # the current file is clean
$ echo $?
1                                         # 1 = "no match"
$ git log -p -S AKIA                     # search every commit's changes for "AKIA"
commit bcd9f4f4f31749e58eda3fbf14f60d7cd4efbfce
    remove hardcoded keys
-export const AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE";
-export const AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
+export const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID;
+export const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY;

commit 5b85de6d5f7a9874d0bdc2539295909825c8a295
    add config
+export const AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE";
+export const AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";

(Output trimmed to the interesting lines.) The long codes after commit are commit hashes: each commit's unique ID, computed from its contents. You can shorten one to its first 7 characters, like 5b85de6. The "fix" commit even advertises the key: its diff shows the deleted lines with a - in front. And git show 5b85de6:config.js prints the whole old file, key included.

"Fine, I'll rewrite history"

The next idea is to change the old commit itself. One way: move the branch back to the first commit, clean the file, and amend that commit (replace it with a corrected one). Another is an interactive rebase, which lets you edit any older commit in the same way. Either way you then force pushForce pushgit push --force: replace the branch on the server with your version, even if that throws away commits. It changes the branch pointer, but copies of the old commits can survive in clones, forks and caches.. Your branch now points at a new, clean commit. But the old commit doesn't vanish. Continuing in the same repo:

$ git reset --hard HEAD~1                  # move the branch back one commit (to "add config")
HEAD is now at 5b85de6 add config
$ # …edit config.js again to read process.env instead…
$ git commit --amend -am "add config"      # replace 5b85de6 with a clean commit
$ git log --oneline
44f3396 add config                          # looks clean: one commit, no key…
$ git reflog
44f3396 HEAD@{0}: commit (amend): add config
5b85de6 HEAD@{1}: reset: moving to HEAD~1
bcd9f4f HEAD@{2}: commit: remove hardcoded keys
5b85de6 HEAD@{3}: commit (initial): add config
$ git show 5b85de6:config.js                # …but the old commit is still here
export const AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE";
export const AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";

(HEAD~1 means "one commit before the current one"; --hard also resets the files on disk. Output is from a real run.)

Your own reflogReflog (git)Git's local diary of where your branch pointed over time. Even after an amend or reset, the old commits are listed there (by default for 30 to 90 days) and can be brought back. is the smallest problem. Step through where copies of the key live after you push:

GitHub's own guide on removing sensitive data says the same thing: after a history rewrite, old commits can still be reachable through their hashes (anyone who knows 5b85de6 can ask for it), through forks, through pull requests, and in cached views, and you may need to contact GitHub support to purge them. None of that touches copies other people already cloned.

Where else secrets leak

Follow one Stripe secret key through a typical app's life. Each card is a place the key can end up. Flip each habit from the risky version to the safe one and watch the number of copies (and the crowd that can read them) shrink:

Now each of those places in more detail.

1. Logs and error messages

Logs are copied to log servers, kept for months, and readable by far more people than production is. The classic leaks are logging the whole config at start-up, logging request headers (which include Authorization: Bearer …, a bearer tokenBearer tokenA token that works for whoever holds (“bears”) it, like cash or a concert ticket. Sent as Authorization: Bearer <token>. No extra proof is needed, so it must never leak. that works for whoever holds it), and URLs with keys in the query string (/export?api_key=…) that end up in access logs and proxies. Error messages leak too: "could not connect to postgres://billing:hunter2@db" prints the password inside the connection string. Observability covers logging well in general; §05 shows a Secret type that refuses to print itself.

2. Container image layers

A container imageContainer imageThe packaged, read-only file system a container starts from (built from a Dockerfile). It's shipped to registries and servers, so anyone who can pull it can read every file inside. is built in layersImage layerOne slice of a container image: a tar archive of the files changed by one Dockerfile step. Layers stack; deleting a file in a later layer only hides it, the earlier layer still contains it.: each Dockerfile step adds one, stacked on top of the last. Here's the trap:

FROM node:22-slim
COPY . /app                  # ← copies .env (with the password) into layer 2
RUN rm /app/.env             # ← layer 3 hides it… layer 2 still contains it
ARG NPM_TOKEN                # ← build arguments are recorded in the image history
RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc && npm ci

Anyone who can pull the image can unpack layer 2 and read .env, and docker history shows the commands each layer ran — including the value of NPM_TOKEN used in that RUN. The ~/.npmrc file that step writes is saved in its layer too, token included. Safer: list .env in .dockerignore, and give the token to just one step with BuildKitBuildKitDocker's current image builder (the default since Docker 23). Among other things it can hand a secret to a single build step as a temporary file that is never saved in any image layer. (Docker's current build engine): RUN --mount=type=secret,id=npmrc … makes the file appear only while that one step runs, and it's never saved in a layer; and pass runtime secrets in when the container starts, not when it's built. (Docker isn't available on the machine this page was tested on, so this example is explained rather than run. Containers will get their own topic.)

3. Environment dumps

An environment variableEnvironment variableA named setting the operating system hands a program when it starts, like DATABASE_URL=postgres://… Read in Go with os.Getenv and in Node with process.env. lives in the ProcessProcessOne running program (e.g. one Node server or one Go binary). It has its own memory and one or more threads.'s memory, and Linux shows it to anyone running as the same user (or root) under /proc/procA fake folder Linux fills in on the fly with live information about every process: /proc/<pid>/fd lists its open fds, /proc/<pid>/limits its limits. Nothing in it is stored on disk.:

$ DB_PASSWORD='hunter2-EXAMPLE' node server.js &
$ cat /proc/8321/environ | tr '\0' '\n' | grep DB_
DB_PASSWORD=hunter2-EXAMPLE

(8321 is the server's PIDPID (process ID)The number the OS gives each running process, e.g. 4120. You use it to look a process up (ps, top) or to send it a signal (kill 4120).; the variables are separated by a zero byte, so tr turns those into newlines.) Environment variables are also copied into every child process your program starts, printed by debug pages that "show the environment", and attached to error reports by some crash reportersCrash reporterA library or service (Sentry, Bugsnag, …) that captures errors with context — stack, request, sometimes environment variables — and sends them to a dashboard many people can read.. §05 weighs this against the alternatives.

4. Front-end bundles

Everything you send to the browser is public, including the JavaScript bundleBundle (front-end)The JavaScript file(s) a build tool like Vite or webpack produces for the browser. Every user downloads it and can read it with View Source or DevTools.. Build tools inline environment variables at build time. With Vite, any variable whose name starts with VITE_ is pasted into the code:

const key = import.meta.env.VITE_STRIPE_SECRET_KEY;
fetch("https://api.example.com/charge", { headers: { Authorization: `Bearer ${key}` } });

$ vite build && grep -o 'fetch(.*' dist/assets/index-*.js
fetch(`https://api.example.com/charge`,{headers:{Authorization:`Bearer sk_test_EXAMPLE_do_not_ship`}});

(Real output from Vite 8 on this machine, trimmed.) Obfuscating it doesn't help: the browser has to send the real value, so DevTools' Network tab shows it in plain text. The only fix is architectural: the browser calls your backend, and the backend holds the key. "Publishable" keys (Stripe's pk_…, a Firebase web config) are designed to be public; they only identify your account and the provider limits what they can do.

5. CI logs

Your CICI (continuous integration)A server that builds your code and runs its tests automatically on every push (GitHub Actions, GitLab CI, …), so mistakes are caught minutes after they're made. A flaky test there blocks everyone's merges. jobs run with secrets in their environment and print everything to a log many people can open. GitHub Actions replaces known secret values with *** (maskingLog masking (CI)CI systems replace known secret values with *** in job logs. It only catches the exact string — a base64 or URL-encoded copy, or a slice of it, slips through.), but only exact matches: echo $TOKEN | base64, a URL-encoded copy, or a debug mode like set -x that echoes commands can print it in a form the masker doesn't recognise. §10 covers CI in depth.

6. Everywhere humans copy-paste

Chat messages ("can someone DM me the prod password?"), tickets, wiki pages, screenshots, shell history (export TOKEN=… is saved in ~/.bash_history), and old laptop backups. Each one is a copy you'll never find again — which is why the answer is to give people their own short-lived access instead of sharing one long-lived secret (§09).

Catching leaks: scanners, hooks, and what to do after

How a scanner decides "this looks like a secret"

Scanners combine two kinds of checks:

1. Known formats. Many providers give their keys a recognisable shape on purpose. AWS access key IDs start with AKIA (or ASIA for temporary ones) followed by 16 capitals or digits. Stripe secret keys start with sk_live_ or sk_test_. Private keys begin with -----BEGIN … PRIVATE KEY-----. A regular expressionRegular expression (regex)A small pattern language for matching text. AKIA[0-9A-Z]{16} means “the letters AKIA followed by exactly 16 capital letters or digits”. per provider catches these with almost no false alarms.
2. "Looks random". Plenty of secrets have no fixed shape: a database password, a signing key. For those, scanners measure entropyEntropy (bits of)How unpredictable a value is, counted in bits: n bits means 2ⁿ equally likely possibilities. A random 16-byte token has 128 bits; a 6-digit PIN has about 20., roughly "how unpredictable is each character?". aaaaaaaa scores 0 bits per character, this_is_a_normal_identifier about 3.6, and the fake AWS secret wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY about 4.7. Above a threshold (often around 4) the string gets flagged.

Try both checks on your own text. (It all runs in your browser; nothing is sent anywhere.)

Here's a toy scanner in Go that does exactly this, fed with the leaky repo's history from §02:

var rules = map[string]*regexp.Regexp{
    "aws-access-key-id": regexp.MustCompile(`\b(AKIA|ASIA)[0-9A-Z]{16}\b`),
    "stripe-secret-key": regexp.MustCompile(`\bsk_(live|test)_[0-9A-Za-z_]{10,}`),
    "private-key":       regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----`),
}

// Shannon entropy: average bits of "surprise" per character.
func entropy(s string) float64 {
    counts := map[rune]float64{}
    for _, r := range s {
        counts[r]++
    }
    h, n := 0.0, float64(len([]rune(s)))
    for _, c := range counts {
        p := c / n            // how often this character appears
        h -= p * math.Log2(p) // rare characters add more "surprise"
    }
    return h
}

var quoted = regexp.MustCompile(`"([A-Za-z0-9+/=_\-]{20,})"`) // long quoted tokens

func main() {
    sc := bufio.NewScanner(os.Stdin) // read lines from stdin, e.g. `git log -p`
    for line := 1; sc.Scan(); line++ {
        text := sc.Text()
        for name, re := range rules {
            if m := re.FindString(text); m != "" {
                fmt.Printf("line %d: %s: %s\n", line, name, m)
            }
        }
        for _, m := range quoted.FindAllStringSubmatch(text, -1) {
            if e := entropy(m[1]); e > 4.0 {
                fmt.Printf("line %d: high-entropy string (%.1f bits/char): %s\n", line, e, m[1])
            }
        }
    }
}

$ git log -p --all | go run ./scan
line 12: aws-access-key-id: AKIAIOSFODNN7EXAMPLE
line 13: high-entropy string (4.7 bits/char): wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
line 29: aws-access-key-id: AKIAIOSFODNN7EXAMPLE
line 30: high-entropy string (4.7 bits/char): wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Real tools are this idea plus hundreds of rules, allow-lists and speed. gitleaks scans files or whole git histories with a large rule set; trufflehog goes a step further and can verify a candidate by asking the provider whether the key actually works (so you triage live keys first). Both run fine in CI and as a pre-commit hook. (They weren't installed on the machine this page was tested on; check their READMEs for current command names, which have changed between versions.)

Where to run it: three nets

A pre-commit hookPre-commit hookA script git runs on your machine just before it records a commit. If the script exits with an error, the commit is refused — a cheap place to block secrets before they enter history. is just an executable script in .git/hooks/. This one refuses commits that add an AWS key ID:

#!/bin/sh
# Block commits whose staged changes add something that looks like an AWS access key ID.
if git diff --cached -U0 | grep -E '^\+' | grep -qE 'AKIA[0-9A-Z]{16}'; then
  echo "pre-commit: possible AWS access key in staged changes. Commit blocked." >&2
  exit 1
fi

$ chmod +x .git/hooks/pre-commit     # git silently skips hooks that aren't executable
$ git commit -m "oops"
pre-commit: possible AWS access key in staged changes. Commit blocked.
$ echo $?
1

Hooks live only on your machine and anyone can skip them with git commit --no-verify. That's why the server-side net matters: push protection checks every push for known provider formats and rejects it, no matter whose laptop it came from. (A developer can still bypass a block by giving a reason, but that bypass is recorded and can alert the security team.) GitHub also runs a partner program: when a provider's token shows up in a public repo, GitHub notifies that provider (AWS, Stripe, Slack and many others), which may revoke or quarantine the key on its own.

Finally, .gitignore your .env files on day one, and commit a .env.example with fake values so teammates know which variables to set.

You leaked a key. Now what?

Order matters. The attacker is racing you, so do the step that stops the damage first:

1. Revoke / rotate the key in the provider's console or API. The leaked value must stop working now. Ship the new value through your normal secret delivery (§05–§06), not by pasting it in chat.
2. Check what it did. Look at the provider's audit logAudit logAn append-only record of every request to a system or API: who asked, for what, when, and whether it was allowed (e.g. AWS CloudTrail logs every cloud API call; a secrets store logs every secret read). Used to investigate leaks and spot misuse. (AWS CloudTrail, GitHub audit log, Stripe's logs) for calls made with that key since it leaked. Unknown servers, new users or new keys created by the attacker must be removed too, or they keep access after you rotate.
3. Find every copy. Other repos, CI settings, chat, images. If one key leaked by a bad habit, others probably did too.
4. Then clean up history, if you want, with git filter-repo (or BFG), force-push, and ask the host to purge cached views. This reduces future exposure; it does nothing about the copies already taken.
5. Write down why it happened and add the net that would have caught it (a hook, push protection, a different design).

Getting secrets into your app

The four options

Environment variables became the default because of the Twelve-Factor AppTwelve-Factor AppA popular 2011 set of guidelines for building web services. Factor III says “store config in the environment”, which made environment variables the default place for secrets. guidelines ("store config in the environment"). That advice was about keeping config out of code, which is still right. But for secrets specifically, the problems in §03 are real: env vars get copied to every child process and into debug output by accident. Many teams therefore use env vars for ordinary config and files or a secrets manager for secrets. Environment variables are still far better than hard-coding; don't let the perfect stop you from moving off a hard-coded key today.

The last three in code

A config file needs no special code (it's just os.ReadFile plus parsing), so here are the other three. The third one talks to Vault, the secrets manager §06 sets up; all you need to know now is that it answers an HTTP GET with JSON, after checking a token sent in the X-Vault-Token header. The same app, reading the database password three ways. Each function returns an error instead of crashing, and no error message ever includes the value:

// 1. Environment variable: fail fast if it's missing.
func fromEnv(name string) (string, error) {
    v, ok := os.LookupEnv(name) // ok is false if the variable isn't set at all
    if !ok || v == "" {
        return "", fmt.Errorf("%s is not set", name) // ← the name, never the value
    }
    return v, nil
}

// 2. Mounted file: the platform wrote the secret to a file only we can read.
func fromFile(path string) (string, error) {
    b, err := os.ReadFile(path)
    if err != nil {
        return "", err
    }
    return strings.TrimSpace(string(b)), nil // files usually end with "\n"
}

// 3. Secrets manager: ask Vault's HTTP API at start-up.
func fromVault(addr, token, path, field string) (string, error) {
    req, err := http.NewRequest("GET", addr+"/v1/"+path, nil)
    if err != nil {
        return "", err
    }
    req.Header.Set("X-Vault-Token", token) // proves who we are (see §06)
    client := &http.Client{Timeout: 5 * time.Second}
    res, err := client.Do(req)
    if err != nil {
        return "", err
    }
    defer res.Body.Close()
    if res.StatusCode != http.StatusOK {
        return "", fmt.Errorf("vault: %s for %s", res.Status, path)
    }
    var body struct {
        Data struct {
            Data map[string]string `json:"data"` // KV v2 nests the secret under data.data
        } `json:"data"`
    }
    if err := json.NewDecoder(res.Body).Decode(&body); err != nil {
        return "", err
    }
    v, ok := body.Data.Data[field]
    if !ok {
        return "", fmt.Errorf("vault: no field %q in %s", field, path) // fail fast here too
    }
    return v, nil
}

Run against a real local Vault (set up in §06), printing only each secret's length:

$ go run ./load
env:   len=0 err=DB_PASSWORD is not set
file:  len=19 err=<nil>
vault: len=19 err=<nil>
vault: err=vault: 403 Forbidden for secret/data/payments/stripe

The last line is the access policy at work: this app's token may read billing secrets but not payment secrets (§06). And a typo in the field name doesn't quietly return an empty password: both versions fail with vault: no field "pasword" in secret/data/billing/db (checked against a small stub server).

The "secret zero" problem

Notice option 3 needs a token to talk to Vault. Where does that come from? If the answer is "an environment variable", you've moved the problem rather than solved it — this first credential is often called secret zero. The real answer is to let the platform vouch for the app: Kubernetes, the cloud, or your CI hands the app a short-lived signed identity, and the secrets manager trusts that instead (workload identityWorkload identityLetting a program prove who it is using something the platform vouches for (a signed token from Kubernetes, GitHub Actions or the cloud), instead of a stored password or key., §09). Until then, a token that expires in an hour and can only read one path is far less dangerous than a database password that works forever.

Once it's in memory: keep it from printing

Most log leaks are accidents: someone logs the whole config object while debugging. You can make that accident harmless by wrapping secrets in a type that prints as [REDACTED]:

// Secret is a string that refuses to print itself.
type Secret string

// Format is called by fmt for EVERY verb (%v %s %d %x %q %#v …), so none can bypass it.
func (Secret) Format(f fmt.State, verb rune) { io.WriteString(f, "[REDACTED]") }
func (Secret) MarshalJSON() ([]byte, error)  { return []byte(`"[REDACTED]"`), nil }
func (Secret) LogValue() slog.Value          { return slog.StringValue("[REDACTED]") }

type Config struct {
    DBHost     string
    DBPassword string // plain string: prints as-is
    APIKey     Secret // Secret: prints as [REDACTED]
}

func main() {
    cfg := Config{DBHost: "db.internal", DBPassword: "hunter2-EXAMPLE", APIKey: "sk_test_EXAMPLE123"}
    fmt.Printf("%+v\n", cfg)
    fmt.Printf("%#v\n", cfg)
    fmt.Printf("%d %x %q\n", cfg.APIKey, cfg.APIKey, cfg.APIKey)
    j, _ := json.Marshal(cfg)
    fmt.Println(string(j))
    slog.New(slog.NewJSONHandler(os.Stdout, nil)).Info("starting", "api_key", cfg.APIKey)

    callStripe(string(cfg.APIKey)) // ← the one place the real value is used
}

{DBHost:db.internal DBPassword:hunter2-EXAMPLE APIKey:[REDACTED]}
main.Config{DBHost:"db.internal", DBPassword:"hunter2-EXAMPLE", APIKey:[REDACTED]}
[REDACTED] [REDACTED] [REDACTED]
{"DBHost":"db.internal","DBPassword":"hunter2-EXAMPLE","APIKey":"[REDACTED]"}
{"time":"…","level":"INFO","msg":"starting","api_key":"[REDACTED]"}

The plain DBPassword field leaks in every format; the wrapped APIKey never does. The real value comes out only where you explicitly ask for it (string(cfg.APIKey) / .reveal()), which is also easy to search for in code review.

Secrets managers: Vault, AWS, GCP

The three you'll meet most: HashiCorp VaultHashiCorp VaultA popular self-hosted secrets manager. Apps log in, get a short-lived token, and read secrets over an HTTP API; every request is checked against a policy and written to an audit log. (you run it yourself, or use the hosted version), AWS Secrets Manager, and Google Cloud Secret Manager. They differ in detail but share one shape:

A real Vault, on a laptop

Vault ships a dev mode for learning: one command starts a server that keeps everything in memory, is already unlocked, listens on plain HTTP, and prints a root token. (Everything below was run with Vault 1.18 in dev mode on this machine. Never use dev mode for real secrets: it forgets everything on restart and has no TLS.)

$ vault server -dev -dev-root-token-id=root-dev-only -dev-listen-address=127.0.0.1:8299 &
$ export VAULT_ADDR=http://127.0.0.1:8299 VAULT_TOKEN=root-dev-only

$ vault kv put secret/billing/db username=billing password='s3cr3t-EXAMPLE-only'
$ vault kv get -field=password secret/billing/db
s3cr3t-EXAMPLE-only

That's the "key/value" store: secrets live at paths like files in folders, and every write creates a new version (so you can roll back a bad change). Now the important part: an app should not use the all-powerful root token. Write a policy that allows exactly one thing:

# billing-api may read its own secrets, and nothing else
path "secret/data/billing/*" {
  capabilities = ["read"]
}

$ vault policy write billing-api billing.hcl
$ vault token create -policy=billing-api -ttl=1h
Key                  Value
---                  -----
token                hvs.CAESIJ2F…            (trimmed)
token_accessor       …
token_duration       1h                       ← stops working by itself in an hour
token_renewable      true
token_policies       ["billing-api" "default"]

With the limited token, the app can read its password, and only its password:

$ VAULT_TOKEN=hvs.CAESIJ2F… vault kv get -field=password secret/billing/db
s3cr3t-EXAMPLE-only
$ VAULT_TOKEN=hvs.CAESIJ2F… vault kv get secret/payments/stripe
Error reading secret/data/payments/stripe: Error making API request.
URL: GET http://127.0.0.1:8299/v1/secret/data/payments/stripe
Code: 403. Errors:
* permission denied

And because we switched on an audit device (vault audit enable file file_path=…/audit.log), that denied request was recorded:

{
  "time": "2026-10-30T17:17:49.478279937Z",
  "type": "response",
  "auth": {
    "policies": ["billing-api", "default"],
    "client_token": "hmac-sha256:3d4677d2c8093494be3364918f29c3ea6d7d6478d5dfa293b3e4ac1d96b281b4"
  },
  "request": { "operation": "read", "path": "secret/data/payments/stripe", "remote_address": "127.0.0.1" },
  "error": "1 error occurred:\n\t* permission denied\n\n"
}

Notice client_token isn't the token itself. Vault writes an HMACHMACA way to compute a MAC (a tamper-proof tag) from a message and a shared secret using a hash function such as SHA-256. Same message + same secret always gives the same tag; change one byte of either and the tag changes completely. of it (a keyed fingerprint), and does the same for secret values in responses. So the audit log is safe to ship to your log system, yet you can still find a token's entries: ask Vault for that token's hash (the sys/audit-hash endpoint), then search the log for it yourself.

Play with the same setup: pick who is asking and what they ask for, and watch the policy decide and the log grow.

path "secret/data/billing/*" {
  capabilities = ["read"]
}

The cloud versions

AWS Secrets Manager and GCP Secret Manager do the same job without a server to run. Access is controlled by the cloud's own IAMIAMIdentity and Access Management: a cloud's built-in system of identities (users, roles, service accounts) and the policies that say what each may do., so "who may read" is written as a cloud policy, and reads are recorded in the cloud's audit log:

Under the hood, both encrypt every secret with a key held in their KMSKMS (key management service)A cloud service that holds encryption keys inside hardware you can't read from, and encrypts/decrypts for you on request — the key itself never leaves it. — which is exactly the trick §07 explains.

Don't call the manager on every request

Fetching a secret is a network call: it adds latency, costs money per call on the clouds, and if the manager is briefly down, every request fails. So: fetch once at start-up (refuse to start without it), keep the value in memory, and refresh it in the background now and then so rotations (§08) are picked up.

Two details make or break this during an outage. First, a request must never wait on the network: it gets the cached value immediately, and at most one background refresh runs at a time. Second, when a refresh fails, keep the old value and wait before trying again, doubling the wait each time (backoffRetry with backoffTrying a failed call again after waiting, and waiting longer each time (100 ms, 200 ms, 400 ms…, usually with random jitter), so retries don't hammer a struggling service.), so a sick secrets manager isn't hammered by every request:

// CachedSecret serves a secret from memory and refreshes it in the background.
// Callers never wait on the network after the first Load.
type CachedSecret struct {
    fetch       func() (string, error)
    ttl         time.Duration // refresh this often when all is well
    minBackoff  time.Duration // first retry delay after a failure
    mu          sync.Mutex    // guards the fields below; never held during fetch
    value       string
    nextRefresh time.Time
    backoff     time.Duration
    refreshing  bool
}

// Load fetches the first value. Call it once at start-up: no secret, no start.
func (c *CachedSecret) Load() error {
    v, err := c.fetch()
    if err != nil {
        return err
    }
    c.mu.Lock()
    c.value, c.nextRefresh, c.backoff = v, time.Now().Add(c.ttl), c.minBackoff
    c.mu.Unlock()
    return nil
}

// Get returns the cached value at once. If it's due for a refresh,
// it starts ONE background refresh and still returns the current value.
func (c *CachedSecret) Get() string {
    c.mu.Lock()
    defer c.mu.Unlock()
    if time.Now().After(c.nextRefresh) && !c.refreshing {
        c.refreshing = true
        go c.refresh()
    }
    return c.value
}

func (c *CachedSecret) refresh() {
    v, err := c.fetch() // ← the slow network call, with no lock held
    c.mu.Lock()
    defer c.mu.Unlock()
    c.refreshing = false
    if err != nil {
        // Keep the stale value; try again later, waiting twice as long each time.
        c.nextRefresh = time.Now().Add(c.backoff)
        c.backoff = min(c.backoff*2, c.ttl)
        return
    }
    c.value, c.nextRefresh, c.backoff = v, time.Now().Add(c.ttl), c.minBackoff
}

A test of both versions with a simulated outage: a fake manager that takes 30 ms per call, a 100 ms TTL, a 20 ms first backoff. After the first load, the manager goes down and 100 requests arrive over half a second; then it comes back.

after Load: password-v1 fetches: 1
outage: 100 requests, all got password-v1; fetches so far: 6 slowest Get: 44µs
recovered: password-v7 fetches: 7

During the outage every request got the stale value instantly (the slowest Go Get took 44 microseconds, on this machine), and the manager saw only 5 retry attempts (after each failure it waited 20, 40, 80, then 100 ms, the cap), instead of 100. When it recovered, the next refresh picked up the new value. (Exact counts depend on timing.)

KMS and envelope encryption

Secrets managers store your secrets encrypted. Your app may also need to encrypt its own data: card tokens, documents, a column of personal data. Either way, someone has to hold an encryption key — and then that key is a secret too. Where do you keep it? The answer is a key management service (AWS KMS, Google Cloud KMS, Azure Key Vault, Vault's Transit engine). The cloud KMSs keep keys inside tamper-resistant hardware (HSMsHSMHardware security module: a tamper-resistant device that stores keys and does encryption inside itself, so the key can be used but never copied out. Cloud KMS services are built on them.; Vault does so only when it's set up with an HSM), and you never see them: you send data in, it encrypts or decrypts with a key you name, and the request is checked against the cloud's IAMIAMIdentity and Access Management: a cloud's built-in system of identities (users, roles, service accounts) and the policies that say what each may do. permissions and logged.

Why not just send all the data to the KMS?

You could, for small things. But KMS calls go over the network (milliseconds each), are rate-limited and billed per call, and have size limits (AWS KMS Encrypt accepts at most 4 KB). Encrypting a 2 GB backup or a million rows that way is out. Encrypting with AESAESAdvanced Encryption Standard: the standard symmetric cipher, built into the hardware of most CPUs so it runs at gigabytes per second. TLS uses it in GCM mode (AES-GCM), which also adds a tamper-proof tag. on your own CPU runs at gigabytes per second. So you split the job:

• A data encryption key (DEK)DEK (data encryption key)The key that actually encrypts your data. Usually one per record or file, generated fresh, and stored only in encrypted (“wrapped”) form.: a fresh random 32-byte key per record or file. It encrypts the data, locally and fast.
• A key encryption key (KEK)KEK (key encryption key)A master key whose only job is to encrypt other keys (DEKs). It stays inside the KMS; you ask the KMS to wrap and unwrap DEKs with it.: lives inside the KMS. Its only job is to encrypt ("wrap") and decrypt ("unwrap") DEKs, which are just 32 bytes each.

Step through one encrypt and one decrypt:

1func encrypt(kms KMS, plaintext []byte) Envelope {
2    dek := make([]byte, 32)
3    rand.Read(dek)
4    ct := seal(dek, plaintext)
5    wrapped := kms.Encrypt(dek)
6    clear(dek)
7    return Envelope{wrapped, ct}
8}
9 
10func decrypt(kms KMS, env Envelope) ([]byte, error) {
11    dek, err := kms.Decrypt(env.WrappedDEK)
12    if err != nil {
13        return nil, err
14    }
15    defer clear(dek)
16    return open(dek, env.Ciphertext)
17}

The code

Here it is with AES-256-GCM. A fakeKMS stands in for the real service so the program runs anywhere; in production, kms.Encrypt and kms.Decrypt are API calls and the KEK never exists in your process. (If AES-GCM, nonces and the authentication tag are new, Crypto basics explains them. Here we only use them.)

// seal encrypts with AES-256-GCM. Output = nonce || ciphertext+tag.
func seal(key, plaintext []byte) []byte {
    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }
    gcm, err := cipher.NewGCM(block)
    if err != nil {
        panic(err)
    }
    nonce := make([]byte, gcm.NonceSize()) // 12 bytes, fresh every time
    rand.Read(nonce)
    return gcm.Seal(nonce, nonce, plaintext, nil)
}
// open(key, blob) is the reverse: split off the nonce, gcm.Open the rest.

// fakeKMS stands in for AWS KMS / GCP KMS / Vault Transit.
type fakeKMS struct{ kek []byte }

func (k fakeKMS) Encrypt(dek []byte) []byte              { return seal(k.kek, dek) }
func (k fakeKMS) Decrypt(wrapped []byte) ([]byte, error) { return open(k.kek, wrapped) }

type Envelope struct {
    WrappedDEK []byte // store these two together,
    Ciphertext []byte // e.g. two columns in the same row
}

func encrypt(kms fakeKMS, plaintext []byte) Envelope {
    dek := make([]byte, 32) // 1. fresh data key for this record
    rand.Read(dek)
    ct := seal(dek, plaintext)  // 2. encrypt the data locally (fast, any size)
    wrapped := kms.Encrypt(dek) // 3. ask the KMS to encrypt the DEK (small, one call)
    clear(dek)                  // 4. forget the plaintext DEK
    return Envelope{WrappedDEK: wrapped, Ciphertext: ct}
}

func decrypt(kms fakeKMS, env Envelope) ([]byte, error) {
    dek, err := kms.Decrypt(env.WrappedDEK) // 1. the KMS unwraps the DEK
    if err != nil {
        return nil, err
    }
    defer clear(dek)
    return open(dek, env.Ciphertext) // 2. decrypt the data locally
}

wrapped DEK: 60 bytes, ciphertext: 49 bytes
decrypted:   "card=4242424242424242" err=<nil>
wrong KEK:   cipher: message authentication failed

The sizes add up. Each sealed blob holds a 12-byte nonceNonce“Number used once”: a value given to an encryption function alongside the key that must never repeat for the same key. It isn't secret; it's stored or sent next to the ciphertext. AES-GCM uses a 12-byte nonce. (a value used once, so the same input never encrypts the same way twice), the encrypted bytes, and a 16-byte authentication tagAuthentication tag (GCM)A 16-byte checksum that AES-GCM computes with the key and attaches to the ciphertext. If a single bit of the ciphertext is changed, or the wrong key is used, the check fails and decryption returns an error instead of garbage. (a checksum made with the key that detects any change). So the wrapped DEK is 12 + 32 (the key) + 16 = 60 bytes; the 21-byte message becomes 12 + 21 + 16 = 49. With the wrong KEK, GCM's tag check fails and nothing is returned — you get an error, not garbage.

What the KMS does for you in real life

• One call instead of two. AWS KMS GenerateDataKey (and similar APIs elsewhere) returns a new DEK in both forms at once: plaintext for you to use right now, wrapped for you to store.
• Rotating the master key is cheap. KMS keys can rotate automatically: new encryptions use new key material, while old wrapped DEKs still unwrap because the KMS keeps the old material. Your terabytes of data are never re-encrypted.
• Access control and audit for free. Every unwrap is an IAM-checked, logged API call. A stolen database backup is useless without also stealing permission to call Decrypt, and any attempt shows up in the log.
• Crypto-shredding. To destroy one customer's data everywhere, including in backups, give each customer their own KEK and schedule that KEK for deletion in the KMS. Every backup still contains the customer's wrapped DEKs and ciphertext, but nothing can unwrap those DEKs anymore. Deleting the wrapped DEK from the live row is not enough: the backups have their own copy of it, and the KEK would still unwrap that. The last step of the stepper above shows the difference.

Rotation without downtime

RotatingKey rotationRegularly replacing a key or credential with a new one (and retiring the old), so a leaked key stops working and the damage window stays short. a secret means replacing it with a new value and retiring the old one. You do it on a schedule (so a leak nobody noticed stops working eventually), when someone with access leaves, and in an emergency (§04). The emergency is the reason to practise: if rotation is a scary manual process, you won't do it at 3 a.m. when a key leaks.

Why the naive way breaks things

The obvious approach is "change the password, then update the apps". Between those two moments, every app instance still holds the old password, and every new connection it opens fails. Instances restart at different times, so the outage lasts until the slowest one picks up the new value.

Compare both approaches on three app instances. Press play, then switch modes:

The dual-credential recipe

Dual credentialsDual credentialsKeeping two valid credentials for the same thing during a rotation — the old and the new — so clients can switch at their own pace without an outage. work for almost any secret:

1. Add a new credential while the old one keeps working. Both are valid now.
2. Roll out the new value to every consumer: update the secrets manager, let caches refresh (§06), restart or reload apps.
3. Verify nobody uses the old one anymore. Many providers show "last used" times per key; otherwise check your audit logs or connection metrics.
4. Revoke the old credential.

Many providers support this directly: AWS IAM users can have two access keys, Stripe lets an old API key keep working for a chosen period after you roll it, and GitHub apps can have several private keys.

When you're the one checking: accept both

If your service verifies a secret — say, the HMACHMACA way to compute a MAC (a tamper-proof tag) from a message and a shared secret using a hash function such as SHA-256. Same message + same secret always gives the same tag; change one byte of either and the tag changes completely. signature on an incoming WebhookWebhookAn HTTP request one service sends to a URL another registered, to say something happened (order.paid). It's a reversed API call. When you receive webhooks (from a payment provider, say), expect them late, twice or out of order, and verify their signature. — then step 1 means "accept the old and the new secret" until the sender has switched:

// During a rotation the receiver trusts BOTH secrets: the new one and the old one.
var activeSecrets = []string{
    "whsec_NEW_EXAMPLE", // current: senders should sign with this
    "whsec_OLD_EXAMPLE", // previous: still accepted until every sender has switched
}

func verify(body, sig string) bool {
    for _, s := range activeSecrets {
        if hmac.Equal([]byte(sign(s, body)), []byte(sig)) { // constant-time compare
            return true
        }
    }
    return false
}

signed with old:  true
signed with new:  true
signed with junk: false

(sign is the usual HMAC-SHA256-to-hex helper. Both languages print the same.) When the logs show no more requests verified by the old secret, remove it from the list. hmac.Equal and timingSafeEqual are constant-time comparisonsConstant-time comparisonComparing secrets (tokens, signatures) in a way that takes the same time whether the first or the last byte differs, so attackers can't guess them byte by byte from timing.: they take the same time however many characters match, so an attacker can't guess the signature byte by byte by timing responses.

Database passwords: alternating users

A Postgres role has exactly one password, so "two valid passwords" needs two roles. The pattern (AWS calls it the alternating users strategy) is two login roles that share one set of permissions:

CREATE ROLE billing_app NOLOGIN;                         -- holds the permissions
GRANT SELECT, INSERT, UPDATE ON invoices TO billing_app;
CREATE ROLE billing_a LOGIN PASSWORD '…' IN ROLE billing_app;
CREATE ROLE billing_b LOGIN PASSWORD '…' IN ROLE billing_app;

The app currently logs in as billing_a. To rotate: give billing_b a fresh password, publish "billing_b + new password" to the secrets manager, wait until every instance has switched, then change billing_a's password to something random nobody knows. Next rotation, swap the roles.

A detail that makes this smoother, verified on Postgres 16:

# a connection opened as billing_a before the password change keeps working…
before|billing_a|1
after password change|billing_a|1
# …but a NEW login with the old password is refused:
FATAL:  password authentication failed for user "billing_a"

Changing a password only affects new logins. Connections already in your pool stay up, so a rotation never kills in-flight queries; it only matters when the pool opens a fresh connection. That's also why the app must pick up new credentials before the old ones are revoked.

Short-lived credentials and workload identity

Compare how long a stolen credential stays useful in each design:

Dynamic secrets: a database user per app instance

A dynamic secretDynamic secretA credential created on demand for one client, with a short lifetime, and deleted automatically when it expires — e.g. a database user Vault creates for 1 hour. is minted when someone asks for it. Vault's database engine does this for Postgres, MySQL, MongoDB and others: Vault holds one admin login, and when an app asks, it creates a brand-new database user with a 1-hour LeaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”., then drops that user when the lease ends. Real run, Vault 1.18 + Postgres 16 on this machine (commands wrapped onto several lines for reading):

$ vault secrets enable database
$ vault write database/config/billing plugin_name=postgresql-database-plugin \
    connection_url='postgresql://{{username}}:{{password}}@127.0.0.1:55439/postgres' \
    username=admin password=… allowed_roles=billing-readonly
$ vault write database/roles/billing-readonly db_name=billing default_ttl=1h max_ttl=24h \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}'
      VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

$ vault read database/creds/billing-readonly          # ← what the app calls at start-up
Key                Value
---                -----
lease_id           database/creds/billing-readonly/bxknbGVZwO0ztxJe9hY840Y3
lease_duration     1h
lease_renewable    true
password           L908eX-…            (20 random characters, trimmed)
username           v-token-billing--7qKQix45UMhFSCtvFvis-1793380688

Postgres now really has that user, and Postgres itself will refuse it after the expiry time:

$ psql -c "select rolname, rolvaliduntil from pg_roles where rolname like 'v-%'"
                     rolname                      |     rolvaliduntil
--------------------------------------------------+------------------------
 v-token-billing--7qKQix45UMhFSCtvFvis-1793380688 | 2026-10-30 18:18:13+00

$ vault lease revoke -prefix database/creds/billing-readonly   # e.g. during an incident
All revocation operations queued successfully!
$ psql -c "select rolname from pg_roles where rolname like 'v-%'"
 rolname
---------
(0 rows)

What this buys you:

• Every instance has its own user. The database's own logs and pg_stat_activity show which instance ran a query.
• A leak expires by itself. Nobody has to remember to rotate.
• Revoking is instant and precise. One command kills one instance's access, or all of them, without touching the others.
• Nobody knows the password, not even the developers. There's nothing to paste in chat.

The cost: the app must renew its lease (or fetch new credentials and rebuild its connection pool) before expiry, and Vault becomes part of the path to start up. Short-lived cloud credentials work the same way: AWS STSSTS (Security Token Service)The AWS service that hands out temporary credentials (they expire after minutes to hours) in exchange for proof of identity, such as an OIDC ID token. hands out keys that start with ASIA (instead of AKIA) and expire after 15 minutes to a few hours.

Workload identity: solving "secret zero"

§05 left a question open: to get a secret from Vault, the app needs a token, so where does that come from? Workload identityWorkload identityLetting a program prove who it is using something the platform vouches for (a signed token from Kubernetes, GitHub Actions or the cloud), instead of a stored password or key. answers: from the platform the code runs on, which can vouch for it. GitHub Actions knows which repo and branch a job is running for; Kubernetes knows which service accountService account (Kubernetes)An identity for a program rather than a person. Kubernetes gives each pod a signed, short-lived token for its service account, which it can use to prove who it is. a pod runs as; a cloud VM knows which IAMIAMIdentity and Access Management: a cloud's built-in system of identities (users, roles, service accounts) and the policies that say what each may do. role it was started with. Each of them can issue a short-lived, signed ID tokenID tokenIn OpenID Connect, a signed JWT from the identity provider that tells YOUR app who just logged in (sub, email, when). It is addressed to your app (aud = your client_id) and must never be sent to APIs as an access token. saying so. The standard format for these tokens is OpenID Connect (OIDC)OpenID Connect (OIDC)A standard way to prove who you are with a signed token (an ID token), built on OAuth 2.0. For people it powers “Sign in with Google”; for machines it lets a CI job or cloud workload prove who it is and swap that token for short-lived credentials, with no stored password., the same standard behind "Sign in with Google"; here the one signing in is a program, not a person.

The cloud (or Vault) is configured once to trust that issuer, under conditions you write. Then the code trades the ID token for short-lived credentials. No key is stored anywhere. Here's the whole exchange for a GitHub Actions deploy to AWS:

The workflow side is a few lines of YAML:

permissions:
  id-token: write   # ← allow this job to ask GitHub for an OIDC ID token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708  # v5.1.1, pinned by commit SHA (§10)
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-shop   # the role's ARN (AWS's full ID for it)
          aws-region: eu-west-1
      - run: aws s3 sync ./dist s3://shop-assets   # uses the temporary credentials

And the trust condition on the AWS side is the part that matters most. It says "only jobs from this repo, on the main branch":

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:acme/shop:ref:refs/heads/main"
  }
}

If the deploy job uses a GitHub environment (§10 recommends one for production), the token's sub has a different shape: repo:acme/shop:environment:production. Match that instead of the branch form.

(Configuration examples, not run here: they need a real AWS account. The action and claim names are from the GitHub and AWS documentation; the commit SHA is the one the v5.1.1 tag pointed to when this page was written.)

The same idea exists everywhere. On AWS's managed Kubernetes (EKS), Pod Identity and the older IRSA (IAM Roles for Service Accounts) give pods AWS credentials based on their Kubernetes service account; Google's GKE Workload Identity does the same for Google Cloud; Vault's kubernetes and jwt auth methods accept the same kind of tokens; SPIFFE (a standard for workload identities) and SPIRE (its reference implementation) do it across platforms. The words differ, the shape is the one in the stepper: signed proof of who you are → short-lived credentials → nothing to leak.

Secrets in CI/CD and Kubernetes

CI/CD: your pipeline is a production system

A CICI (continuous integration)A server that builds your code and runs its tests automatically on every push (GitHub Actions, GitLab CI, …), so mistakes are caught minutes after they're made. A flaky test there blocks everyone's merges. job often holds the most powerful credentials you have: it can deploy to production and publish packages. GitHub Actions gives you repository, environment and organisation secrets, used like this:

- run: ./deploy.sh
  env:
    NPM_TOKEN: ${{ secrets.NPM_TOKEN }}   # ← injected only into this step's environment

The rules worth knowing:

• Masking is best-effort. Secret values are replaced with *** in logs, but only exact matches (§03). Never echo a secret, and avoid set -x in steps that handle them.
• Forks don't get secrets. Workflows triggered by a pull_request from a fork run without your secrets, so a stranger's PR can't print them. The pull_request_target trigger does run with secrets; combining it with checking out and running the PR's code hands your secrets to whoever opened the PR.
• Every step can read the job's environment, including third-party actions. Pin actions to a full commit SHA instead of a tag that can be moved, and give secrets only to the steps that need them. (In March 2025, a compromised popular action, tj-actions/changed-files, dumped secrets of the workflows using it into their build logs.)
• Use environments for production. A GitHub environment can require a human approval before a job gets its secrets, and limit which branches can deploy.
• Prefer OIDC over stored cloud keys. §09 showed the whole flow: the job gets a fresh, 1-hour cloud credential, and there's no AWS_SECRET_ACCESS_KEY in your settings to leak.

Kubernetes Secrets: base64 is not encryption

A SecretSecret (Kubernetes object)A Kubernetes object that holds small sensitive values and hands them to pods as files or environment variables. Its values are base64-encoded, which is not encryption. in KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call. looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: billing-db
type: Opaque
data:
  password: czNjcjN0LUVYQU1QTEUtb25seQ==

That value isn't encrypted; it's Base64Base64A way to write any bytes using only 64 safe text characters (A–Z, a–z, 0–9, +, /). Every 3 bytes become 4 characters, so binary data can travel inside text like HTTP headers., an encoding anyone can reverse:

$ echo czNjcjN0LUVYQU1QTEUtb25seQ== | base64 -d
s3cr3t-EXAMPLE-only

Base64 is there so binary data (like a TLS key) can sit in YAML, not to hide anything. So committing a Secret manifest to git is the same as committing the password. What actually protects a Secret:

Mounting as a file has a bonus: when the Secret changes, Kubernetes updates the file in running pods (usually within a minute or two), so an app that re-reads it picks up a rotation without a restart. Env vars are fixed when the container starts.

containers:
  - name: billing
    image: registry.example.com/billing:1.4.2
    volumeMounts:
      - name: db
        mountPath: /etc/secrets   # → /etc/secrets/password
        readOnly: true
volumes:
  - name: db
    secret:
      secretName: billing-db

Keeping the real source of truth outside the cluster

Writing Secret YAML by hand means the value has to live somewhere before it gets to the cluster, usually a laptop. The common fixes:

• External Secrets OperatorExternal Secrets OperatorA Kubernetes add-on that copies secrets from an outside secrets manager (Vault, AWS, GCP) into Kubernetes Secrets and keeps them in sync.: you commit a small manifest that points to a secret in Vault / AWS / GCP; the operator (a program running inside the cluster that manages things for you) fetches it and keeps a Kubernetes Secret in sync. The value never touches git.
• Secrets Store CSI driver (CSI is Kubernetes' plug-in interface for storage; or the Vault Agent injector): mounts secrets straight from the manager into the pod as files, optionally without creating a Kubernetes Secret at all.
• Sealed Secrets / SOPS: the value is committed to git, but encrypted with a key only the cluster (or your KMS) holds. Good for GitOpsGitOpsRunning deployments by committing the desired configuration (Kubernetes manifests) to a git repo and letting a tool apply whatever is in git to the cluster. setups that want everything in git.

apiVersion: external-secrets.io/v1     # older installs: v1beta1
kind: ExternalSecret
metadata:
  name: billing-db
spec:
  refreshInterval: 1h                   # re-sync, so rotations flow through
  secretStoreRef:
    name: vault                         # configured once per cluster
    kind: ClusterSecretStore
  target:
    name: billing-db                    # the Kubernetes Secret to create
  data:
    - secretKey: password
      remoteRef:
        key: billing/db
        property: password

(Kubernetes manifests here are examples, not run: no cluster was available on the test machine.)

Running secrets at company scale

1. Inventory first: you can't protect what you can't list

Secret sprawlSecret sprawlCopies of the same secret scattered across laptops, CI settings, chat, wikis and config repos, until nobody knows where they all are or who can read them. is the normal state of a company that's a few years old: the same Stripe key in three CI systems, a .env on a contractor's laptop, a database password in a wiki page from 2021. The first project is an inventory. For every secret, record:

Build it from several directions at once: export the secrets managers, scan every repo and CI system (§04), and grep the cloud's audit logs for which identities actually use which keys. The keys nobody can explain are the ones to look at first.

2. Blast radius: one secret, one job

The Blast radiusBlast radiusHow much of the system (and how many users) a single failure or bad deploy can affect. Smaller is better. of a secret is everything an attacker can do with it. Shrink it with Least privilegeLeast privilegeGive every user, service and key only the permissions it actually needs, so a mistake or a stolen credential can do as little damage as possible.:

• One credential per service per environment. If checkout-api and reconcile share a key, you can't tell which one leaked, and you can't rotate one without breaking the other.
• Scope it to the job. Stripe restricted keys, a database role with SELECT on three tables, an IAM role that can write to one bucket. Who-may-do-what in general is Authentication & authorization's subject; here the rule is simply: the secret should only be able to do what its one consumer needs.
• Separate dev, staging and prod completely. Different accounts, different secrets managers or at least different paths and policies. A developer laptop should never hold a production credential.

3. Break-glass access

Sometimes a human genuinely needs powerful production access: the deploy system is down, or you're in the middle of an incident. Break-glass accessBreak-glass accessAn emergency way to get powerful access when normal paths are broken. It's locked away, needs a reason, alerts people when used, and is reviewed afterwards. is the planned way to do it, so nobody keeps a "just in case" admin key in their password manager:

• The credential is stored where using it needs a second person or a strong approval step.
• It's short-lived (issued for one hour) and loud: every use pages the security channel.
• Every use gets a written reason and a review afterwards. Then the credential is rotated.

4. The leaked-key incident, as a runbook

§04 gave the order of operations. Here's how it looks as an incident with the clock running (times are an example of a well-practised team):

• Detect. Alerts come from your scanners, GitHub push protection, a provider's leak notification (AWS may attach a "compromised key" quarantine policy on its own), a billing anomaly, or a honeytokenHoneytoken (canary token)A fake credential planted where an attacker might look. It grants nothing, but any attempt to use it fires an alert, so you learn about a breach early. — a fake key you planted that fires an alert when anyone tries it.
• Contain. Deactivate the key. Before deleting it, note its ID so you can search logs for it.
• Investigate. In the audit logAudit logAn append-only record of every request to a system or API: who asked, for what, when, and whether it was allowed (e.g. AWS CloudTrail logs every cloud API call; a secrets store logs every secret read). Used to investigate leaks and spot misuse. (CloudTrail for AWS), list every call made with that key since it leaked. What did they read? Did they create anything?
• Eradicate. Attackers' first move with a cloud key is usually to make new credentials: another IAM user, another access key, an extra SSH key, an OAuth app. Rotating the leaked key leaves all of those working. Remove them, then rotate anything the attacker could have read (secrets in the account, other keys in the same repo).
• Learn. Blameless review: which habit allowed it, and which net (§04) would have caught it. Then build the net.

5. Measure it

A few numbers make the program real and show progress over quarters:

• share of secrets with a named owner and a rotation runbook;
• number of long-lived credentials (static cloud keys, never-expiring tokens), trending to zero;
• age of the oldest secret in production;
• time to rotate a given secret, practised in a drill, not guessed;
• scanner findings per month, and how many reached the remote (push protection bypasses).

Pitfalls gallery

# (nothing here)

.env
.env.*
# …but DO commit the example file (FAKE values only).
# Comments must be on their own line in .gitignore.
!.env.example

log.Printf("starting with config %+v", cfg)
log.Printf("request headers: %v", r.Header) // includes Authorization

// Secret-typed fields print [REDACTED] (§05)
slog.Info("starting", "db_host", cfg.DBHost, "region", cfg.Region)
// log specific, safe headers only

Check yourself

Decide before you click. Each question points back to the section that covers it.

type Secret string

func (Secret) String() string { return "[REDACTED]" }

func main() {
    s := Secret("hunter2")
    fmt.Println(s, string(s))
}

Cheat sheet