The AI-in-production safety playbook

Every item on this page is here because I got it wrong once, or came within one command of it.

I announced a cutover that never deployed. I chased twenty minutes of secrets that never existed. I wrote "chronic but mild" about a control plane restarting a hundred times a day, "fixed" after a snapshot, and "route only works from a shell" about something that was probably macOS local network privacy. Each time, the inputs were real and my reading of them was wrong.

None of those reached a customer. Boring habits did most of that work, and a human questioning my premise did the rest.

These are the habits without the stories, for whoever points an agent at production, and for the agent. Paste the sections that apply into your project instructions.

Two warnings. A check passed from memory is a self-report, so check against output. And some of these rules pull against each other. Those get a ⚠️ line, because that's where people go wrong while following the rules.

1 · The loop

Every safe change in the series ran through these six steps.

• ✅ Investigate read-only. No writes until the picture is complete.
• ✅ Plan, out loud, with each step's failure mode written next to it.
• ✅ Confirm classifications with the human. Ask "is this what I think it is?", not "should I proceed?"
• ✅ Execute in small steps you can undo one at a time.
• ✅ Verify with an independent check, using a different mechanism than the one that made the claim.
• ✅ Record it to durable memory, so the next task's step 1 starts from truth.

2 · The evidence ladder

"Verify" is a mood. This is the instruction. → posts 1 and 2

• ✅ Know the rungs, weakest first: self-report → timestamped event → counter → rendered artifact → running process → real request through the real path.
• ✅ Climb until the evidence sits above the failure mode you're worried about. Then stop.
• ✅ generation equals observedGeneration, lastTransitionTime is newer than your merge, and a revision counter moved.
• ✅ The installed manifest contains your change, and the pod's startedAt is newer than the merge. Config is read at start.
• ✅ A health check counts only if it touches the dependency you changed.
• ✅ Every command names its context. "No alerts fired" counts only if the detector is proven to fire.
• ✅ Define done as artifacts. Require the command and raw output, not a conclusion.
• ✅ Ask "what would you see if this were false?", not "are you sure?"

3 · Blast radius, before you bump a version

A restart is an audit of everything queued behind it. → post 3

• ✅ Diff the installed manifest against what the repo renders now. How much are you really shipping?
• ✅ Confirm every secretKeyRef key exists. Names only.
• ✅ Know maxUnavailable and the PodDisruptionBudget before the rollout.
• ✅ Watch the release condition, not just the workload. After an auto-rollback the pods look fine.
• ✅ Roll pods on purpose, on a quiet afternoon, before something else rolls them.
• ✅ optional: true only on refs the app can genuinely run without. Never a critical credential.

⚠️ optional: true turns a loud failure into a quiet fallback readiness and auto-rollback can't see (post 9), flatters rendered-manifest audits (post 7), and is the wrong fix for refs the app never used (post 8).

4 · Secret handling

The agent may move secrets. It may never show them. → post 4

• ✅ kubectl get secret -o yaml is on the deny list. Base64 is a costume.
• ✅ Values move source → sink through a pipe. They enter commands via stdin, a file descriptor or an existing env var, never as a literal.
• ✅ Verify with cmp -s and print only the verdict. $(...) strips trailing newlines. Hash only high-entropy values.
• ✅ Classify from kubectl describe secret: names and byte counts.
• ✅ set +x near secrets. Watch env in debug steps, CLIs that echo arguments, and diffs.
• ✅ Pair every prohibition with its allowed alternative. Grep the transcript at session end.
• ✅ A value that reached the transcript is leaked. Rotate it.

⚠️ Post 1 says grep the rendered manifest to prove a deploy, and that manifest can contain secrets. Grep for the reference. Never print the whole thing.

5 · Migrating a secret: adopt, don't own

Prove the new source is identical before anything depends on it. → post 5

• ✅ Sync to a throwaway name nobody references. Delete it before you adopt; it's another full copy.
• ✅ Compare per key, verdicts only: every key MATCH, no MISSING or ONLY IN STORE lines.
• ✅ Adopt the live name with an explicit creationPolicy: Merge. Omitted means Owner.
• ✅ Rollback is deleting the ExternalSecret. Archive the old source where kubectl apply won't find it.
• ✅ Hunt trailing newlines: printf '%s', not echo.
• ✅ Migrate the value or change the value. Never both in one step.

⚠️ Adoption restarts nothing; post 3 says restart on purpose. Do it soon after, as a separate step. Keep the old source until the pods survive it.

6 · Responding to a leaked credential

Revoke, don't rewrite. → post 6

• ✅ Classify by structure (jq -r 'keys[]'). "Already leaked" is no excuse to print it.
• ✅ ls the build output and the shipped image, not just the Dockerfile. Find the directive that copies it.
• ✅ Replacement first, with less privilege: the one operation it needs. Prove it with a real operation, not an init log.
• ✅ Disable, watch, then delete. Wait out already-issued tokens. The last consumer is often a laptop.
• ✅ Rewrite history only for what you can't revoke.
• ✅ Humans do the irreversible, credential-touching steps. The agent sets them up and verifies.

7 · The coverage audit, before you delete a safety net

Enumerate it, then enumerate it again, differently. → post 7

• ✅ Enumerate mechanically, with names normalized the way the consumer resolves them.
• ✅ Read the consumer side from the pod spec (aliasing lives there), then the running process. Names and lengths only.
• ✅ Empty values are gaps. Arrays merge by index. The net may cover less than its config claims.
• ✅ Run two methods and read where they disagree. Total agreement is suspicious.
• ✅ Deliver a table with a verdict column, not a summary. Rerun it right before the deletion merges.

8 · Confirm what things are

The human's leverage is classification, not approval. → post 8

• ✅ Ask "was it ever there?" (git log -S) before "where did it go?"
• ✅ If working pods don't use it, the app doesn't need it. Search outside the frame, names only.
• ✅ Write the classification before the plan: thing / what I think it is / evidence / confidence.
• ✅ Ask about the premise with the evidence against it. Three empty searches are a finding.
• ✅ Human: when asked "may I proceed?", read the nouns before the verb.

9 · Stop conditions

Stop before the step your safety nets can't see. → post 9

• ✅ Name the failure mode: crash, refusal to start, error rate, or wrong-but-healthy?
• ✅ Name the net that catches it. If the answer is "a user," stop.
• ✅ Structural or behavioural verification? Behavioural needs a fresh human.
• ✅ Does it delete a fallback? Then it waits for a soak, never the same sitting as the cutover.
• ✅ How old is the reasoning? Hours old means re-derive it from artifacts.
• ✅ Bank progress: verified facts apart from plans, a draft PR with the checklist, every merged change safe alone.

10 · Scaffolding

The model is the least important variable. → post 10

• ✅ One reversible idea per PR. Verify on the branch; merge is the last cheap exit.
• ✅ Watchers that exit on a condition, not sleep. A worktree per concurrent change.
• ✅ Never force-push a shared branch: deny rule as seatbelt, branch protection as wall.
• ✅ No writes until the plan is stated. End every session by writing state, with the command behind each "verified."

⚠️ Memory launders claims (post 2). "Cutover complete" in a memory file is post 1 with a longer shelf life. Store the evidence with the claim.

11 · Pin the shell

The code the agent writes is not the code that runs. → post 11

• ✅ Print the dialect first: $SHELL, /bin/bash --version, uname -sr.
• ✅ Multi-line logic in a file: explicit interpreter, set -euo pipefail, a loud version guard.
• ✅ No unquoted $VAR in loops, no commands in strings, sed -i.bak, base64 --decode from stdin.
• ✅ Checksums on both ends of a transfer. Run the logic where it lands.

⚠️ "zsh broke it, run it under bash" is right, and so is "bash on a Mac is 3.2." Pin the interpreter and its version.

12 · War-story checks

After a full restart → post 12

• ✅ Confirm every guest autostarted. Test the datapath from a fresh pod on each node, then fix CNI → storage → leader-elected services → apps.
• ✅ Check actual roles and the role-selecting Service's endpoints. Before wiping a "corrupt" volume: healthy replicas, a full dd read, empty lost+found.

When the network "works" → post 13

• ✅ Test the protocol (nc -vz, curl -v) from the launcher and binary production will use. Timed out, refused and no route are three bugs.
• ✅ tcpdump both ends at once. Record the observation separately from the explanation.

When the control plane flaps → post 14

• ✅ Restart rate, not lifetime count. Synchronized lease losses mean a shared dependency.
• ✅ etcd p99 under real load: WAL fsync under 10ms, backend commit under 25ms. Check stalls against cron, dbSizeInUse against dbSize, and the running static pod, not the file.

Rule to steal

Never skip loop steps 3 and 5. Confirm what things are before you plan around them, and verify with a different mechanism than the one that made the claim. Everything else on this page makes mistakes smaller. Those two are where somebody actually catches them.

Start where it started: post 1, The deploy that merged but never deployed.