Series · 15 parts

Trust, but Verify

LLM Mart · Aug 10, 2026
Trust, but Verify

Trust, but Verify follows two long sessions of genuine production work — recovering a cluster after the hypervisor rebooted, standing up a self-hosted secrets platform, rotating a leaked credential, migrating an application's config under GitOps — and documents every place it went sideways.

  1. Part 1

    The deploy that merged but never deployed

    A green PR, a controller reporting success, and not one line of the new code running

    Aug 10, 2026
  2. Part 2

    What counts as proof

    "Verify your work" isn't an instruction. It's a mood. Here's the version that's an instruction. Verify with a different mechanism than the one that made the claim.

    Aug 19, 2026
  3. Part 3

    The version bump that took down an app (and how it self-healed)

    One character of YAML, four pods that never started, and two safety nets I didn't know were holding. Every restart is an audit. Schedule them before they schedule you.

    Aug 19, 2026
  4. Part 4

    Never let the AI print a secret

    The most important rule wasn't about what I could change. It was about what I was allowed to display.

    Sep 19, 2026
  5. Part 5

    Byte-identical or bust

    "Copy it over and switch the reference" is two steps, and the outage lives in the one nobody checks

    Sep 20, 2026
  6. Part 6

    Rotating a leaked credential, in the right order

    Everyone looks at the Dockerfile. The file that actually leaked the key was the project file.

    Sep 21, 2026
  7. Part 7

    The coverage audit before you delete the safety net

    31 config keys, two audits, and why the first one was wrong in both directions

    Sep 22, 2026
  8. Part 8

    Those are the other app's keys

    Twenty minutes recovering secrets that never existed, and the one sentence from a human that ended it

    Sep 23, 2026
  9. Part 9

    Knowing when to stop

    Six hours in, one step left, everything green, and the incident that didn't happen

    Sep 24, 2026
  10. Part 10

    The scaffolding that made it safe

    None of the safety came from the model. It came from six boring habits.

    Sep 25, 2026
  11. Part 11

    The agent is running in *your* shell

    A surprising share of AI-in-the-terminal failures aren't the AI. They're zsh, and a version of bash from 2006.

    Sep 26, 2026
  12. Part 12

    Bringing a cluster back after the host rebooted

    Every VM came back. The cluster did not. Declarative systems converge on config, and the datapath isn't config.

    Sep 27, 2026
  13. Part 13

    The overlay that pinged but wouldn't carry TCP

    Discovery worked. Ping worked. Every TCP connection timed out, and later the tunnel only worked when someone had a terminal open.

    Sep 28, 2026
  14. Part 14

    The control plane was flapping because of a spinning disk

    Healthy nodes, a quiet network, 300 restarts in three days, and a latency budget measured in milliseconds

    Sep 29, 2026
  15. Part 15

    The AI-in-production safety playbook

    Fourteen posts of being wrong in production, compressed to checkboxes

    Sep 30, 2026
Start reading