#incident-response
The control plane was flapping because of a spinning disk
Healthy nodes, a quiet network, 300 restarts in three days, and a latency budget measured in milliseconds
The overlay that pinged but wouldn't carry TCP
Discovery worked. Ping worked. Every TCP connection timed out, and later the tunnel only worked when someone had a terminal open.
Bringing a cluster back after the host rebooted
Every VM came back. The cluster did not. Declarative systems converge on config, and the datapath isn't config.
Knowing when to stop
Six hours in, one step left, everything green, and the incident that didn't happen
The version bump that took down an app (and how it self-healed)
One character of YAML, four pods that never started, and two safety nets I didn't know were holding. Every restart is an audit. Schedule them before they schedule you.