Prompt testing: build a lightweight eval set

A prompt can look excellent in a chat window and fall apart the moment the input changes. One good answer is a data point, not a quality system. If a prompt sits underneath a workflow anyone depends on, test it against the inputs it will actually receive.

The good news is that the useful version of this is small. Five cases in a spreadsheet beats an elaborate harness nobody runs.

What a case actually contains

Each case needs four things: an input, the expected behavior, pass criteria, and any reference data the grader needs.

The expected behavior is the field people skip, and it's the one that makes the set work. A strict extraction task has one right answer and a schema. A drafting task has a range of acceptable answers and a set of things that must never appear. Writing that down forces you to decide what you're testing before you're staring at an output you want to like.

Record the run alongside it — prompt version, model, model settings, tools available, and date. Without that, a result is an anecdote.

Start with five cases

Not fifty. Five, chosen deliberately:

Case 5 earns its place fastest. Most prompts fail it silently — they pick something plausible and present it with the same confidence as a correct answer, and you never find out until a user does.

Pull the inputs from production if you can. Real messiness is more specific than invented messiness: contradictory instructions, mixed languages, a PDF that pasted badly, a form where someone put the phone number in the name field.

Then grow the set from failures, not from imagination.

Define pass and fail before you look at output

A rubric is only useful if two people scoring the same output land in the same place. That means observable criteria:

• Extracts all required fields.
• Invents no value that isn't in the source.
• Returns valid JSON with the required keys.
• Flags ambiguity instead of choosing silently.
• Stays within the length target.

Score 0/1/2 when binary is too blunt — 0 for fail, 1 for usable after edits, 2 for shippable. Track the edit distance if the output gets edited before use; "passed" and "passed after I rewrote half of it" are different results.

Keep a severe-error flag on its own axis, separate from the score. A fabricated number, a leaked instruction, or an action taken without authorization isn't a low score. It's a different category of failure, and averaging it away is how a dangerous regression ships looking fine.

Change one thing at a time

If you change the prompt, the model, and the preprocessing together and the score improves, you've learned nothing you can act on. OpenAI's evals guidance follows the same shape — define a task, run it against test data, analyze, iterate — and the structure works identically in a spreadsheet.

The failure this catches is the expensive one: a prompt change that improves the common case and quietly breaks a rare one. You will not notice that by re-running the case you were working on.

Categorize failures, because the category is the fix

A pile of failed cases tells you the prompt is bad. Categorized failures tell you what to do:

That last row is worth dwelling on. If a prompt can be talked into an action it shouldn't take, the fix is almost never a better instruction — it's a control outside the model. Prompt-shaped fixes for authority-shaped problems are the most common wasted afternoon in this work, and a security case is not a quality case.

Keep the regressions

Run the set after every meaningful change: prompt, model, model settings, retrieval data, tools, or input preprocessing. All six change behavior; only one of them is the prompt.

Every real failure earns a permanent case. That's the whole regression set — mistakes you have decided never to make twice. It grows slowly and it's the part of the suite that pays for itself.

For anything tool-using, test the boring infrastructure failures too: a missing document, stale data, a permission boundary, a tool that returns an error. Those paths are where confident wrong answers live.

NIST's AI Risk Management Framework frames evaluation as continuous measurement rather than a launch gate, which is the right instinct here. A prompt feeding a daily workflow is a system in production, whether or not it looks like one.

Evals turn a vague worry into a repeatable test with a threshold. Start with five representative cases, write down what good means before you look at an output, categorize what breaks, and keep every regression. The goal isn't a perfect score. It's fewer surprises tomorrow.