Claude Skill

uig

Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query ma

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Download calibre-labs-reforge-ai-evals-skills_uig-d787d1c.zip · 6 KB

Install

skills CLI npx skills add https://github.com/Calibre-Labs/reforge-ai-evals/tree/main/skills/uig
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install calibre-labs-reforge-ai-evals@llmmart
Git git clone https://github.com/Calibre-Labs/reforge-ai-evals.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole calibre-labs/reforge-ai-evals collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

User Input Grid Skill

A User Input Grid (UIG) — also called a Synthetic Query Matrix (SQM) — is a structured framework for ensuring eval coverage of an AI product reflects the real diversity of users, content domains, input types, and input styles the system will face in production. It comes from the Reforge AI Evals curriculum and a few teams at Amplitude, Linear, and Notion have published versions of it.

The core idea: pick 3–5 dimensions that drive meaningful diversity in your product, define 4–7 values per dimension, and treat the cartesian product (pruned for plausibility) as your eval sourcing matrix. A 4-dimension grid with average ~5 values per dimension gives ~625 theoretical combinations — practical eval sets land at 30–80 high-coverage inputs after pruning.

The reason this matters: most AI eval datasets are built from one of three weak sources — engineer intuition (overfits to features the engineer knows about), real-user logs (overfits to common easy cases users actually try), or synthetic generation without a framework (random uneven coverage). A UIG forces you to be deliberate about what you're testing, makes gaps visible, and gives you a shared vocabulary with the product team for talking about coverage.

When to use this skill

This skill triggers in three modes — be explicit about which one you're in before starting:

  1. Greenfield UIG — designing a UIG for a new AI product/feature with no existing dataset. Workflow ends at "framework + example inputs to seed the dataset."

  2. Audit existing dataset(s) — there are one or more eval datasets and you want to know what they're missing. Workflow includes evaluating each dataset against the UIG with STRONG/PARTIAL/WEAK coverage ratings, surfacing gaps, and recommending additions.

  3. Both — design the UIG informed by patterns in the existing dataset, then audit the dataset against the UIG. This is the most common case when a team has been building evals ad-hoc and wants to put structure around them.

If the mode isn't specified, ask in one short question, then proceed.

Workflow

Step 1 — Understand the product

Before designing dimensions, you need to know: what does this AI product do, who uses it, what data does it have access to, what tools does it call, and what does success look like? Don't skip this — a UIG built without product understanding produces generic dimensions that don't catch real failure modes.

If the user has shared product docs, sample inputs, system prompts, or tool definitions, read them first. If you don't have enough context, ask 2–3 specific questions — not a generic interview. Examples: "What actions can the AI take? Are there topics it explicitly can't handle? What are the most common user goals?"

Step 2 — Read the dataset(s) if any exist

If there are existing datasets, read them before designing the UIG. Real input data is the best signal you have. Look for:

  • Persona signals — are there user-type labels in metadata? Do inputs imply different user contexts (new user, power user, admin, customer vs. employee)?
  • Domain coverage — what topic areas do inputs touch? What's missing from the inputs that the product clearly supports?
  • Input type distribution — how many action instructions vs. information requests vs. multi-step tasks vs. ambiguous inputs?
  • Style patterns — are inputs well-specified, vague, count-only, edge cases?
  • Scoring criteria quality — are pass/fail criteria deterministic, or vague LLM-judge prompts? Watch for OR conditions ("X or asks for clarification") that make the test always pass.

This "dataset-first reading" surfaces dimensions you'd never invent from first principles.

Step 3 — Design the dimensions

Pick 3–5 dimensions. Fewer than 3 and the grid doesn't drive enough diversity; more than 5 and the cartesian product explodes and the dimensions start overlapping conceptually.

Two dimensions are nearly universal for any AI product. Use these as defaults, adapted to context:

Input Type — what the user is asking the AI to do

Value Definition Example
Action instruction Direct the AI to perform an operation "Update the Nike opportunity to Closed Won"
Information request Ask the AI to retrieve or explain something "What's the status of case #12345?"
Aggregation Compute or summarize across a set "How many open P1 cases do I have this week?"
Multi-step Chain of operations in one input "Find at-risk accounts and draft outreach emails"
Clarification / follow-up Continue or refine a prior turn "Actually, make the tone more formal"
Ambiguous / under-specified Intent unclear, AI must resolve "Handle this" with no supporting context
Edge / out-of-scope Outside the AI's topics or capabilities A request the system is not designed to fulfill

For products with workflow-triggered inputs (no user text — an automated record or event fires the AI): add Automated trigger as a value and ensure your eval dataset includes both human-initiated and system-initiated inputs.

For AI products with traditional search/query interfaces, "Action instruction" may be rare and "Lookup" a better fit — adapt the values to what users actually send.

Input Style — how well-formed or specified the input is

Value Definition Example
Well-specified Complete, unambiguous, all context present Clear intent, named entities, no missing fields
Under-specified Missing context or intent the AI must infer Vague instruction, implied subject, no record ID
Multi-constraint Multiple conditions that must all be satisfied "Empathetic but firm, under 100 words, no jargon"
Jargon-heavy Domain- or org-specific terminology Internal product codes, CRM field names, acronyms
Edge / out-of-scope Impossible, self-contradicting, or out of bounds Request the AI cannot or should not fulfill

Two dimensions are always domain-specific and need to be derived from the product:

  • User Persona — different user types whose context changes what a good response looks like. Don't fall back to "novice / intermediate / expert" — that's a smell. Real personas are defined by their goals and context (e.g. "sales rep closing deals" vs. "service agent handling escalations" vs. "customer self-serving on a portal").
  • Data/Content Domain — the distinct topic areas, capability surfaces, or data categories the product covers. This is the dimension you most need the dataset to inform — first-principles drafts almost always miss something.

For agent products, consider a fifth dimension: Action/Output Type — what the AI produces (drafted email, updated record, routing decision, summary, action refusal). This catches output format and action correctness bugs that input-diversity dimensions alone won't surface.

For each dimension, define 4–7 values. Each value needs a short justification (one sentence on why it's distinct from the others) and a characteristic example input showing what it generates.

Step 4 — Write the UIG markdown file

Ask the user where to save the output. Default: <product-name>-uig.md in the current working directory.

Structure the file as follows:

# [Product Name] User Input Grid
Brief intro paragraph — what product, what the UIG covers, target dataset size.

## Dimensions Overview
Summary table: Dimension | Values | Notes

## [Dimension 1] — [Name]
Table: Value | Definition | Example Input | Why distinct
Notes on judgment calls made.

## [Dimension 2] — [Name]
...

## Example Input Combinations
Table of 8–12 rows sampling the grid: one row per combination, with a concrete example input.
This forces you to verify each combination produces a sensible, testable input.

## Dataset Evaluation  ← only in modes 2 and 3
Per-dataset coverage tables and overall summary.

## Recommendations
Prioritized gap list with example inputs.

The example inputs section is important — it makes the abstract grid concrete and forces you to verify that each combination produces a sensible input. If a combination feels nonsensical when you try to write an input for it, that's a signal the grid has an implausible region you can prune.

Step 5 — Evaluate datasets against the UIG (if mode 2 or 3)

For each dataset, rate coverage on each dimension as STRONG / PARTIAL / WEAK:

  • STRONG — most values represented, with clear examples
  • PARTIAL — some values present, others noticeably missing
  • WEAK — only one or two values covered

For each (dataset × dimension) cell, write a one-paragraph note covering: what's present, what's missing, and the practical implication of the gap. The implication matters more than the gap itself — "no multi-step inputs" is a fact; "the AI could score 95% on this dataset and still be broken when users chain two operations in a single message" is the point.

Seven common failure patterns to watch for:

  1. Persona collapse — all inputs written from one user type's perspective; the dataset silently assumes a single user context even though the product serves multiple roles.
  2. Instruction skew — too many freeform instructions, missing information requests, aggregations, or structured/triggered inputs that the product clearly handles.
  3. Advice-without-criteria — eval rows that test for "good response" but provide no falsifiable scoring criterion. If two evaluators would disagree on pass/fail, the criterion isn't ready.
  4. OR-condition trap — scoring criteria phrased as "output contains X or asks for clarification." This always passes because a confused AI can always ask for clarification. Split into two separate eval rows.
  5. Real-input feedback loop — dataset built entirely from real user logs, which overfit to the common easy cases actual users send. Rare but high-stakes inputs never appear.
  6. Missing comparison/trend — time-period and multi-entity inputs are consistently absent because they're harder to write; they're also where AI products frequently fail.
  7. Output-quality vs. input-diversity confusion — hallucination tests (output quality checks) mixed into a coverage dataset (input diversity checks). These need separate eval strategies.

After per-dataset tables, write an overall coverage summary combining all datasets, then a recommendations section prioritized by impact:

  1. Input types that are entirely absent — highest-priority gaps; nothing in the eval suite catches these failure modes.
  2. Cross-dimension combinations that span two gap areas — catch coordination bugs.
  3. Persona-differentiated inputs where the persona context meaningfully shapes the right response.
  4. Out-of-scope and edge-case inputs that test graceful decline or scope clarification.
  5. Output-quality / hallucination tests for high-stakes domains.

For each recommendation, give 3–6 example inputs with dimension labels and what the test should verify. Don't just say "add multi-step inputs" — write the actual inputs.

Step 6 — Expand the dataset (if requested)

If asked to "expand the dataset" or "fill the gaps", generate a fresh markdown file with 30–80 new inputs organized by the gap they fill, each tagged with persona / domain / input type / style. For every input, include an evaluable scoring criterion.

The OR-condition trap: Never write a scoring criterion as "output does X or asks for clarification." That criterion always passes — any AI can deflect with a clarifying question. Instead, split into two rows: one where input is clear and the AI must act, one where input is genuinely ambiguous and graceful clarification is the right response.

Don't generate from random combinations. Walk through the gaps in priority order and write inputs deliberately. Aim for inputs that look like what a real user would actually send — including partial information, casual phrasing, frustration, and incomplete context when that's how real users communicate.

Output Format

Default to markdown. Ask the user where to save; if unspecified, save to <product-name>-uig.md in the current working directory.

For UIGs with rich tables that benefit from visual layout, an HTML file is also acceptable as a secondary format. Markdown is the default.

Working Notes

  • Don't skip the dataset reading step. First-principles UIGs miss things real data would surface. Even 15 minutes reading a sample of real inputs will reshape the grid.

  • Resist the urge to make dimensions orthogonal at all costs. Some overlap is fine if it produces meaningful coverage. "Input Type" and "Input Style" partially overlap (an ambiguous input is often also under-specified) but they catch different failure modes — ambiguous tests whether the AI resolves intent correctly, while under-specified tests whether it asks the right clarifying question.

  • A UIG with 4 dimensions averaging 5 values gives 625 combinations. You don't need eval coverage of all 625 — you need coverage of the dimensions, not the full cartesian product. 30–80 deliberately chosen inputs that hit each dimension value at least 3 times is much more valuable than 500 random combinations.

  • Tables of dimension values should be readable on their own. Someone glancing at the UIG summary table without reading the body should immediately understand the framework. Use short value names (1–3 words) and put the longer descriptions in the per-dimension sections.

  • The goal isn't "coverage of every value" — it's catching failure modes the team didn't anticipate. When you write the recommendations section, lead with what would have been caught if these inputs had been in the eval set. That framing turns abstract coverage gaps into concrete reasons to invest.

  • For conversational / multi-turn agents, include at least a few multi-turn sequences in the dataset — not just isolated inputs. A clarification / follow-up row is only testable in context of the prior turn. Flag these explicitly in the dataset with a multi_turn: true tag.

Files (reforge-ai-evals)
  • agents
    • openai.yaml 239 B
      Loading…
  • SKILL.md 14.8 KB
    ---
    name: uig
    description: >
      Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets
      against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone
      asks to design an eval framework, audit a test set, build a "user input grid", "synthetic
      query matrix" or "SQM", improve dataset coverage, or assess whether their offline evals
      reflect what real users will actually send. Also trigger when they share a product domain
      + sample dataset and want to know what the dataset is missing, or when they're setting up
      evals from scratch for a new AI feature. Output is a markdown file (UIG framework + dataset
      evaluation + recommendations) saved to the current project directory.
    ---
    
    # User Input Grid Skill
    
    A User Input Grid (UIG) — also called a Synthetic Query Matrix (SQM) — is a structured framework
    for ensuring eval coverage of an AI product reflects the real diversity of users, content domains,
    input types, and input styles the system will face in production. It comes from the Reforge AI
    Evals curriculum and a few teams at Amplitude, Linear, and Notion have published versions of it.
    
    The core idea: pick 3–5 dimensions that drive meaningful diversity in your product, define 4–7
    values per dimension, and treat the cartesian product (pruned for plausibility) as your eval
    sourcing matrix. A 4-dimension grid with average ~5 values per dimension gives ~625 theoretical
    combinations — practical eval sets land at 30–80 high-coverage inputs after pruning.
    
    The reason this matters: most AI eval datasets are built from one of three weak sources — engineer
    intuition (overfits to features the engineer knows about), real-user logs (overfits to common
    easy cases users actually try), or synthetic generation without a framework (random uneven
    coverage). A UIG forces you to be deliberate about *what* you're testing, makes gaps visible, and
    gives you a shared vocabulary with the product team for talking about coverage.
    
    ## When to use this skill
    
    This skill triggers in three modes — be explicit about which one you're in before starting:
    
    1. **Greenfield UIG** — designing a UIG for a new AI product/feature with no existing dataset.
       Workflow ends at "framework + example inputs to seed the dataset."
    
    2. **Audit existing dataset(s)** — there are one or more eval datasets and you want to know what
       they're missing. Workflow includes evaluating each dataset against the UIG with STRONG/PARTIAL/WEAK
       coverage ratings, surfacing gaps, and recommending additions.
    
    3. **Both** — design the UIG informed by patterns in the existing dataset, then audit the dataset
       against the UIG. This is the most common case when a team has been building evals ad-hoc and
       wants to put structure around them.
    
    If the mode isn't specified, ask in one short question, then proceed.
    
    ## Workflow
    
    ### Step 1 — Understand the product
    
    Before designing dimensions, you need to know: what does this AI product do, who uses it, what
    data does it have access to, what tools does it call, and what does success look like? Don't skip
    this — a UIG built without product understanding produces generic dimensions that don't catch real
    failure modes.
    
    If the user has shared product docs, sample inputs, system prompts, or tool definitions, read them
    first. If you don't have enough context, ask 2–3 specific questions — not a generic interview.
    Examples: "What actions can the AI take? Are there topics it explicitly can't handle? What are the
    most common user goals?"
    
    ### Step 2 — Read the dataset(s) if any exist
    
    If there are existing datasets, read them before designing the UIG. Real input data is the best
    signal you have. Look for:
    
    - **Persona signals** — are there user-type labels in metadata? Do inputs imply different user
      contexts (new user, power user, admin, customer vs. employee)?
    - **Domain coverage** — what topic areas do inputs touch? What's missing from the inputs that
      the product clearly supports?
    - **Input type distribution** — how many action instructions vs. information requests vs.
      multi-step tasks vs. ambiguous inputs?
    - **Style patterns** — are inputs well-specified, vague, count-only, edge cases?
    - **Scoring criteria quality** — are pass/fail criteria deterministic, or vague LLM-judge prompts?
      Watch for OR conditions ("X *or* asks for clarification") that make the test always pass.
    
    This "dataset-first reading" surfaces dimensions you'd never invent from first principles.
    
    ### Step 3 — Design the dimensions
    
    Pick **3–5 dimensions**. Fewer than 3 and the grid doesn't drive enough diversity; more than 5 and
    the cartesian product explodes and the dimensions start overlapping conceptually.
    
    **Two dimensions are nearly universal** for any AI product. Use these as defaults, adapted to
    context:
    
    #### Input Type — what the user is asking the AI to do
    
    | Value | Definition | Example |
    |-------|-----------|---------|
    | Action instruction | Direct the AI to perform an operation | "Update the Nike opportunity to Closed Won" |
    | Information request | Ask the AI to retrieve or explain something | "What's the status of case #12345?" |
    | Aggregation | Compute or summarize across a set | "How many open P1 cases do I have this week?" |
    | Multi-step | Chain of operations in one input | "Find at-risk accounts and draft outreach emails" |
    | Clarification / follow-up | Continue or refine a prior turn | "Actually, make the tone more formal" |
    | Ambiguous / under-specified | Intent unclear, AI must resolve | "Handle this" with no supporting context |
    | Edge / out-of-scope | Outside the AI's topics or capabilities | A request the system is not designed to fulfill |
    
    For products with workflow-triggered inputs (no user text — an automated record or event fires the
    AI): add **Automated trigger** as a value and ensure your eval dataset includes both human-initiated
    and system-initiated inputs.
    
    For AI products with traditional search/query interfaces, "Action instruction" may be rare and
    "Lookup" a better fit — adapt the values to what users actually send.
    
    #### Input Style — how well-formed or specified the input is
    
    | Value | Definition | Example |
    |-------|-----------|---------|
    | Well-specified | Complete, unambiguous, all context present | Clear intent, named entities, no missing fields |
    | Under-specified | Missing context or intent the AI must infer | Vague instruction, implied subject, no record ID |
    | Multi-constraint | Multiple conditions that must all be satisfied | "Empathetic but firm, under 100 words, no jargon" |
    | Jargon-heavy | Domain- or org-specific terminology | Internal product codes, CRM field names, acronyms |
    | Edge / out-of-scope | Impossible, self-contradicting, or out of bounds | Request the AI cannot or should not fulfill |
    
    **Two dimensions are always domain-specific** and need to be derived from the product:
    
    - **User Persona** — different user types whose context changes what a good response looks like.
      Don't fall back to "novice / intermediate / expert" — that's a smell. Real personas are defined
      by their goals and context (e.g. "sales rep closing deals" vs. "service agent handling escalations"
      vs. "customer self-serving on a portal").
    - **Data/Content Domain** — the distinct topic areas, capability surfaces, or data categories the
      product covers. This is the dimension you most need the dataset to inform — first-principles
      drafts almost always miss something.
    
    For agent products, consider a fifth dimension: **Action/Output Type** — what the AI *produces*
    (drafted email, updated record, routing decision, summary, action refusal). This catches output
    format and action correctness bugs that input-diversity dimensions alone won't surface.
    
    For each dimension, define 4–7 values. Each value needs a **short justification** (one sentence on
    why it's distinct from the others) and a **characteristic example input** showing what it generates.
    
    ### Step 4 — Write the UIG markdown file
    
    Ask the user where to save the output. Default: `<product-name>-uig.md` in the current working
    directory.
    
    Structure the file as follows:
    
    ```
    # [Product Name] User Input Grid
    Brief intro paragraph — what product, what the UIG covers, target dataset size.
    
    ## Dimensions Overview
    Summary table: Dimension | Values | Notes
    
    ## [Dimension 1] — [Name]
    Table: Value | Definition | Example Input | Why distinct
    Notes on judgment calls made.
    
    ## [Dimension 2] — [Name]
    ...
    
    ## Example Input Combinations
    Table of 8–12 rows sampling the grid: one row per combination, with a concrete example input.
    This forces you to verify each combination produces a sensible, testable input.
    
    ## Dataset Evaluation  ← only in modes 2 and 3
    Per-dataset coverage tables and overall summary.
    
    ## Recommendations
    Prioritized gap list with example inputs.
    ```
    
    The example inputs section is important — it makes the abstract grid concrete and forces you to
    verify that each combination produces a sensible input. If a combination feels nonsensical when you
    try to write an input for it, that's a signal the grid has an implausible region you can prune.
    
    ### Step 5 — Evaluate datasets against the UIG (if mode 2 or 3)
    
    For each dataset, rate coverage on each dimension as **STRONG / PARTIAL / WEAK**:
    
    - **STRONG** — most values represented, with clear examples
    - **PARTIAL** — some values present, others noticeably missing
    - **WEAK** — only one or two values covered
    
    For each (dataset × dimension) cell, write a one-paragraph note covering: what's present, what's
    missing, and the practical implication of the gap. The implication matters more than the gap
    itself — "no multi-step inputs" is a fact; "the AI could score 95% on this dataset and still be
    broken when users chain two operations in a single message" is the point.
    
    **Seven common failure patterns to watch for:**
    
    1. **Persona collapse** — all inputs written from one user type's perspective; the dataset silently
       assumes a single user context even though the product serves multiple roles.
    2. **Instruction skew** — too many freeform instructions, missing information requests, aggregations,
       or structured/triggered inputs that the product clearly handles.
    3. **Advice-without-criteria** — eval rows that test for "good response" but provide no falsifiable
       scoring criterion. If two evaluators would disagree on pass/fail, the criterion isn't ready.
    4. **OR-condition trap** — scoring criteria phrased as "output contains X *or* asks for clarification."
       This always passes because a confused AI can always ask for clarification. Split into two separate
       eval rows.
    5. **Real-input feedback loop** — dataset built entirely from real user logs, which overfit to the
       common easy cases actual users send. Rare but high-stakes inputs never appear.
    6. **Missing comparison/trend** — time-period and multi-entity inputs are consistently absent because
       they're harder to write; they're also where AI products frequently fail.
    7. **Output-quality vs. input-diversity confusion** — hallucination tests (output quality checks)
       mixed into a coverage dataset (input diversity checks). These need separate eval strategies.
    
    After per-dataset tables, write an **overall coverage summary** combining all datasets, then a
    **recommendations section** prioritized by impact:
    
    1. Input types that are entirely absent — highest-priority gaps; nothing in the eval suite catches
       these failure modes.
    2. Cross-dimension combinations that span two gap areas — catch coordination bugs.
    3. Persona-differentiated inputs where the persona context meaningfully shapes the right response.
    4. Out-of-scope and edge-case inputs that test graceful decline or scope clarification.
    5. Output-quality / hallucination tests for high-stakes domains.
    
    For each recommendation, give 3–6 example inputs with dimension labels and what the test should
    verify. Don't just say "add multi-step inputs" — write the actual inputs.
    
    ### Step 6 — Expand the dataset (if requested)
    
    If asked to "expand the dataset" or "fill the gaps", generate a fresh markdown file with
    **30–80 new inputs** organized by the gap they fill, each tagged with persona / domain / input
    type / style. For every input, include an evaluable scoring criterion.
    
    **The OR-condition trap:** Never write a scoring criterion as "output does X *or* asks for
    clarification." That criterion always passes — any AI can deflect with a clarifying question.
    Instead, split into two rows: one where input is clear and the AI must act, one where input is
    genuinely ambiguous and graceful clarification is the right response.
    
    Don't generate from random combinations. Walk through the gaps in priority order and write inputs
    deliberately. Aim for inputs that look like what a real user would actually send — including
    partial information, casual phrasing, frustration, and incomplete context when that's how real
    users communicate.
    
    ## Output Format
    
    **Default to markdown.** Ask the user where to save; if unspecified, save to
    `<product-name>-uig.md` in the current working directory.
    
    For UIGs with rich tables that benefit from visual layout, an HTML file is also acceptable as a
    secondary format. Markdown is the default.
    
    ## Working Notes
    
    - **Don't skip the dataset reading step.** First-principles UIGs miss things real data would
      surface. Even 15 minutes reading a sample of real inputs will reshape the grid.
    
    - **Resist the urge to make dimensions orthogonal at all costs.** Some overlap is fine if it
      produces meaningful coverage. "Input Type" and "Input Style" partially overlap (an ambiguous
      input is often also under-specified) but they catch different failure modes — ambiguous tests
      whether the AI resolves intent correctly, while under-specified tests whether it asks the right
      clarifying question.
    
    - **A UIG with 4 dimensions averaging 5 values gives 625 combinations.** You don't need eval
      coverage of all 625 — you need coverage of the dimensions, not the full cartesian product.
      30–80 deliberately chosen inputs that hit each dimension value at least 3 times is much more
      valuable than 500 random combinations.
    
    - **Tables of dimension values should be readable on their own.** Someone glancing at the UIG
      summary table without reading the body should immediately understand the framework. Use short
      value names (1–3 words) and put the longer descriptions in the per-dimension sections.
    
    - **The goal isn't "coverage of every value" — it's catching failure modes the team didn't
      anticipate.** When you write the recommendations section, lead with what *would have been
      caught* if these inputs had been in the eval set. That framing turns abstract coverage gaps
      into concrete reasons to invest.
    
    - **For conversational / multi-turn agents**, include at least a few multi-turn sequences in the
      dataset — not just isolated inputs. A clarification / follow-up row is only testable in context
      of the prior turn. Flag these explicitly in the dataset with a `multi_turn: true` tag.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.