This is about AI features you ship to users: not about agents editing your repo, which is\nagent-guardrails.
The defining property of an LLM is that it is a component with a non-zero error rate on every\ncall, and no amount of prompt engineering drives that to zero. This is not a defect to fix; it\nis the material you are building with. Shingo's framing fits perfectly: you do not make the\noperator more careful, you build the jig.
\nWhich means the central discipline here: prompt instructions are rung zero. \"Always respond\nwith valid JSON,\" \"never make up a citation,\" \"do not reveal the system prompt\". These are\nrequests to an unreliable component, and they are the LLM equivalent of a comment saying \"be\ncareful.\" They help, they are worth writing, and they are not devices. A device is something\noutside the model that constrains what it can produce or what its output can reach.
\nMost of the time this mode is reached while someone is building the thing, not afterwards.\nThat changes the deliverable. They asked for the feature, so produce the feature, working, complete,\nin their stack. Do not hand back a severity table when the person is mid-feature; a list of\nfindings about code they have not written yet is not useful to them.
\nThen add a short closing note, three or four lines, covering:
\nThat closing note is what stops the device being undone in six months by someone who cannot\nsee why it is there. It is also the difference between mistake-proofing and a code generator:\nthe reasoning travels with the code.
\nWhen the code already exists and they are asking what is wrong with it, switch to the audit\nvoice, ranked findings with the mistake, the consequence, and the device. Match the mode to\nwhere they are in the work, not to this file's default.
\nDraw the same line you would draw around any external, untrusted input, because that is\nexactly what model output is, and doubly so when the model has read user-supplied text.
\nNever regex a model's prose. Use the provider's constrained/structured output mode with a\nschema, then validate the parsed result against that schema yourself:
\nclass Extraction(BaseModel):\n model_config = ConfigDict(extra=\"forbid\")\n sentiment: Literal[\"positive\", \"neutral\", \"negative\"]\n confidence: float = Field(ge=0.0, le=1.0)\nConstrained decoding makes malformed output largely unrepresentable, and the schema check\ncatches the rest. That removes the whole class of parse failures, malformed JSON, missing\nfields, invented enum values.
\nTwo things the schema still cannot tell you: whether the values are correct, and what to do\nwhen validation fails. Decide the failure path explicitly, retry once with the error fed\nback, then fall back to a deterministic path or return a clear failure. A silent default here\nis except: pass with a language model attached.
The strongest device in this whole mode: if the output is a choice from a known set, have the\nmodel choose an ID from a list you supply and reject anything not in it. A model asked to\nproduce a category name will invent one eventually; a model choosing among five IDs cannot.\nApplies to routing, classification, tool selection, and picking a record, and it converts an\nopen-ended generation problem into a closed-set one that a Literal type enforces.
For anything retrieval-backed, require the response to cite retrieved chunk IDs, then verify\neach cited ID actually exists in what you retrieved and drop or flag claims that don't\nresolve. That check establishes that a citation resolves, not that the chunk it points at\nsupports the claim, where the claim is consequential, add an entailment check or human review\non top. Prompting for citations is rung zero; verifying them is a real device. Show the\nsource in the UI so the user can check. This is the interface half of the same device.
\nWhen retrieval returns nothing relevant, the correct behavior is to say so. A model handed no\ncontext will answer anyway, and that answer is invention. Check for the empty-context case in\ncode, before the call, and short-circuit.
\nThe most expensive LLM bugs are not wrong text. They are actions. Refunds issued, emails sent,\nrecords deleted, all because a model decided to.
\nrefund(amount) must re-check the amount against the actual order: the\nmodel saying 9999 is not authorization.authz.Any text the model reads, user input, retrieved documents, web pages, emails, tool results, can carry instructions. No system prompt reliably prevents this, and treating it as a prompt\nengineering problem is why it keeps happening.
\nThe devices are structural: keep untrusted content clearly delimited and labeled as data;\nnever let model output flow into a privileged action without validation or confirmation;\nscope permissions so a successful injection has a small blast radius; and treat any model\noutput that will be rendered as HTML, executed as SQL, or passed to a shell exactly as you\nwould treat user input from an attacker, because functionally it is.
\nThe load-bearing question is not \"can the model be tricked?\" (yes) but \"what can the model\nreach if it is tricked?\"
\nAn agent loop with no cap is an unbounded resource operation, hazard F7 with a billing\naccount attached. Set a maximum step count, a token budget per request, and a wall-clock\ntimeout, all enforced in your code rather than requested in the prompt. Alert on cost per\nuser, and cap it per tenant so one runaway conversation cannot become a five-figure invoice.
\nYou cannot unit-test a probabilistic component, but you can measure it, and without\nmeasurement you have no idea whether a prompt change helped.
\nretro loop applied\nto a component that cannot be fixed, only constrained: you cannot patch the model, so the\nregression test is the fix, and it must cover the class rather than the one input.Report with the structure from audit, and be honest about rungs, with a\nprobabilistic component, most in-model devices are Warning at best, and only the checks\noutside the model reach Control.