Claude Skill

ticket-to-eval

Strip PII from a customer support ticket or eval trace and convert it into eval dataset rows. Produces two outputs: a regression dataset row (close to the original input, tagged with failure_mode) and a generalized dataset row (abstracted for broader coverage). Works with local C

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Download calibre-labs-reforge-ai-evals-skills_ticket-to-eval-d787d1c.zip · 3 KB

Install

skills CLI npx skills add https://github.com/Calibre-Labs/reforge-ai-evals/tree/main/skills/ticket-to-eval
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install calibre-labs-reforge-ai-evals@llmmart
Git git clone https://github.com/Calibre-Labs/reforge-ai-evals.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole calibre-labs/reforge-ai-evals collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Ticket → Eval Conversion

You are converting a customer support ticket or eval trace into two eval dataset rows: a regression row (exact input, tagged with failure_mode) and a generalized row (abstracted for broader coverage). Work through the phases below.


Phase 0: Dataset Discovery

Ask the user where their eval datasets live:

  • Local CSV — glob for CSV files whose first line contains input,metadata
  • Eval platform (Braintrust, LangSmith, etc.) — ask for the project name and list available datasets
  • If unclear, ask: "Should I look for datasets in a local CSV or an eval platform?"

Identify two targets:

  • Regression target — the dataset tracking real failures (look for "regression" in the name)
  • Coverage target — the dataset for generalized coverage rows (the broader dev/test set)

Ask the user to confirm both before proceeding.

Store: regression_target and coverage_target as either {type: "local", path: "<path>"} or {type: "platform", name: "<dataset name>", project: "<project>"}.


Phase 1: Extract & Identify

Parse the input — it may be a CSV row, free-form complaint text, or trace JSON.

Extract:

  • original_input: the exact text or structured input the user sent to the AI
  • complaint_summary: what went wrong (1 sentence)
  • failure_mode: classify using the taxonomy below

Failure Mode Taxonomy

These are common AI failure modes. The list below uses a Market Map agent as a concrete example of each — adapt the labels to your product's failure patterns:

Failure mode What it means Market Map example
intent_misparse Agent misread or ignored a key term in the input Jargon term (PLG, bootstrapped) treated as freeform text
stale_data Agent used outdated information Ranked a company that is bankrupt or acquired
temporal_override Historical/future input answered with current data "Market in 2008" answered with today's data
scope_ignored A constraint in the input was ignored Geographic or segment constraint silently dropped
unit_mismatch Metrics returned in wrong units or mixed without disclosure Revenue in EUR mixed with USD with no conversion note
hallucinated_citation Source does not exist or doesn't support the claim URL 404s or publication doesn't mention the claim
implausible_metric A numeric figure is off by >5x from any credible estimate ARR figure that's 10x the company's actual size
refused_valid_input Agent declined to handle something it should be able to Refused a well-formed request within its stated scope
latency Response time complaint (not a content failure; flag separately) —
format_violation Wrong count, broken structure, incomplete output Wrong number of results, broken table
other Describe in a note field —

Phase 2: Strip PII

Scan the original input AND any surrounding context for PII. Replace in-place with typed placeholders. Do NOT alter words that are not PII.

PII type Placeholder
Person name [USER_NAME]
Email address [USER_EMAIL]
Company / employer [USER_COMPANY]
Session / trace ID [SESSION_ID]
Internal dollar figure [INTERNAL_FIGURE]

Rule: domain-specific content (product names, market terms, company names the user is asking about) is rarely PII — preserve it exactly unless it contains the user's own name or employer submitted in a private context.

Show the cleaned input and ask the user to confirm before proceeding.


Phase 3: Map to Dimensions

Tag the cleaned input using the UIG dimensions from the project's eval dataset. If none are defined, fall back to these generic fields:

{
  "input_type": "<action_instruction | information_request | aggregation | multi_step | clarification | edge_out_of_scope>",
  "domain": "<adapt to your product's domain dimension>",
  "style": "<well_specified | under_specified | multi_constraint | jargon_heavy | edge_out_of_scope>",
  "temporal": "<current | historical | future>",
  "edge_case": true | false
}

Regression-specific additions (Row A only):

{
  "failure_mode": "<from Phase 1>",
  "source_ticket": "<ticket ID if known, else null>",
  "regression": true
}

Phase 4: Generate Both Rows

Row A — Regression Row

  • input: PII-stripped input (exact)
  • metadata: full dimension tags + regression fields
  • expected: blank unless user provides a reference response
  • tags: ["regression"]
  • id: generate with python3 -c "import uuid; print(uuid.uuid4())"
  • target: regression_target from Phase 0

Row B — Generalized Row

Rewrite the input to remove specifics that make it a regression test, keeping the failure-relevant pattern. Goal: organic-looking coverage diversity.

Generalisation examples:

  • "UPI payment apps in India" (scope_ignored) → different geography + category not already in the dataset
  • "PLG-first B2B devtools under $500M ARR" (intent_misparse) → same jargon pattern, different market
  • "social networking in 2008" (temporal_override) → only add if the domain isn't already covered

Before finalising Row B, check what's already in the coverage target. If a similar row already exists, pick a different generalisation.

  • metadata: dimension tags only (no failure_mode, regression, source_ticket)
  • tags: []
  • id: generate a new UUID
  • target: coverage_target from Phase 0

Phase 5: Output & Append

Present both rows clearly labeled with their target:

ROW A → <regression target name>:
<formatted row>

ROW B → <coverage target name>:
<formatted row>

Ask: "Should I append both, just one, or neither?"

Appending to local CSV

Use the Edit tool to append the new line after the last row. Do not rewrite the file.

Appending to an eval platform

Use the platform's SDK or MCP tool to insert the row. Generic pattern:

# Adapt to your platform's SDK
dataset.insert(
    input="<cleaned input>",
    expected=None,
    metadata={"input_type": "...", "failure_mode": "...", ...},
    tags=["regression"]
)

Handling Latency Tickets

Latency complaints can't be tested with a content eval:

  1. Still generate Row A for the regression dataset (useful for load testing).
  2. For Row B, skip the content dataset. Instead check if perf-test-queries.txt exists in the datasets directory; if not, create it. Append the PII-stripped input.
  3. Tell the user: "This failure mode needs a latency monitor, not an LLM judge."

Batch Mode

If the user passes multiple tickets, process all through Phases 1–4 first, then present all rows grouped by target file before asking for a single confirmation.

Files (reforge-ai-evals)
  • agents
    • openai.yaml 228 B
      Loading…
  • SKILL.md 7.3 KB
    ---
    name: ticket-to-eval
    description: >
      Strip PII from a customer support ticket or eval trace and convert it into eval dataset rows.
      Produces two outputs: a regression dataset row (close to the original input, tagged with
      failure_mode) and a generalized dataset row (abstracted for broader coverage). Works with
      local CSV files or any eval platform. Use this skill whenever someone shares a support ticket,
      user complaint, or raw trace and wants to turn it into eval data. Trigger on: "add this ticket
      to the dataset", "convert this trace to an eval", "strip PII from this complaint", "turn this
      into a test case".
    ---
    
    # Ticket → Eval Conversion
    
    You are converting a customer support ticket or eval trace into two eval dataset rows: a
    regression row (exact input, tagged with failure_mode) and a generalized row (abstracted for
    broader coverage). Work through the phases below.
    
    ---
    
    ## Phase 0: Dataset Discovery
    
    Ask the user where their eval datasets live:
    - **Local CSV** — glob for CSV files whose first line contains `input,metadata`
    - **Eval platform** (Braintrust, LangSmith, etc.) — ask for the project name and list available datasets
    - If unclear, ask: "Should I look for datasets in a local CSV or an eval platform?"
    
    Identify two targets:
    - **Regression target** — the dataset tracking real failures (look for "regression" in the name)
    - **Coverage target** — the dataset for generalized coverage rows (the broader dev/test set)
    
    Ask the user to confirm both before proceeding.
    
    Store: `regression_target` and `coverage_target` as either `{type: "local", path: "<path>"}` or
    `{type: "platform", name: "<dataset name>", project: "<project>"}`.
    
    ---
    
    ## Phase 1: Extract & Identify
    
    Parse the input — it may be a CSV row, free-form complaint text, or trace JSON.
    
    Extract:
    - **original_input**: the exact text or structured input the user sent to the AI
    - **complaint_summary**: what went wrong (1 sentence)
    - **failure_mode**: classify using the taxonomy below
    
    ### Failure Mode Taxonomy
    
    These are common AI failure modes. The list below uses a Market Map agent as a concrete example
    of each — adapt the labels to your product's failure patterns:
    
    | Failure mode | What it means | Market Map example |
    |---|---|---|
    | `intent_misparse` | Agent misread or ignored a key term in the input | Jargon term (PLG, bootstrapped) treated as freeform text |
    | `stale_data` | Agent used outdated information | Ranked a company that is bankrupt or acquired |
    | `temporal_override` | Historical/future input answered with current data | "Market in 2008" answered with today's data |
    | `scope_ignored` | A constraint in the input was ignored | Geographic or segment constraint silently dropped |
    | `unit_mismatch` | Metrics returned in wrong units or mixed without disclosure | Revenue in EUR mixed with USD with no conversion note |
    | `hallucinated_citation` | Source does not exist or doesn't support the claim | URL 404s or publication doesn't mention the claim |
    | `implausible_metric` | A numeric figure is off by >5x from any credible estimate | ARR figure that's 10x the company's actual size |
    | `refused_valid_input` | Agent declined to handle something it should be able to | Refused a well-formed request within its stated scope |
    | `latency` | Response time complaint (not a content failure; flag separately) | — |
    | `format_violation` | Wrong count, broken structure, incomplete output | Wrong number of results, broken table |
    | `other` | Describe in a note field | — |
    
    ---
    
    ## Phase 2: Strip PII
    
    Scan the original input AND any surrounding context for PII. Replace in-place with typed
    placeholders. Do NOT alter words that are not PII.
    
    | PII type | Placeholder |
    |---|---|
    | Person name | `[USER_NAME]` |
    | Email address | `[USER_EMAIL]` |
    | Company / employer | `[USER_COMPANY]` |
    | Session / trace ID | `[SESSION_ID]` |
    | Internal dollar figure | `[INTERNAL_FIGURE]` |
    
    **Rule**: domain-specific content (product names, market terms, company names the user is
    *asking about*) is rarely PII — preserve it exactly unless it contains the user's own name
    or employer submitted in a private context.
    
    Show the cleaned input and ask the user to confirm before proceeding.
    
    ---
    
    ## Phase 3: Map to Dimensions
    
    Tag the cleaned input using the UIG dimensions from the project's eval dataset. If none are
    defined, fall back to these generic fields:
    
    ```json
    {
      "input_type": "<action_instruction | information_request | aggregation | multi_step | clarification | edge_out_of_scope>",
      "domain": "<adapt to your product's domain dimension>",
      "style": "<well_specified | under_specified | multi_constraint | jargon_heavy | edge_out_of_scope>",
      "temporal": "<current | historical | future>",
      "edge_case": true | false
    }
    ```
    
    Regression-specific additions (Row A only):
    ```json
    {
      "failure_mode": "<from Phase 1>",
      "source_ticket": "<ticket ID if known, else null>",
      "regression": true
    }
    ```
    
    ---
    
    ## Phase 4: Generate Both Rows
    
    ### Row A — Regression Row
    
    - **input**: PII-stripped input (exact)
    - **metadata**: full dimension tags + regression fields
    - **expected**: blank unless user provides a reference response
    - **tags**: `["regression"]`
    - **id**: generate with `python3 -c "import uuid; print(uuid.uuid4())"`
    - **target**: `regression_target` from Phase 0
    
    ### Row B — Generalized Row
    
    Rewrite the input to remove specifics that make it a regression test, keeping the
    failure-relevant pattern. Goal: organic-looking coverage diversity.
    
    Generalisation examples:
    - `"UPI payment apps in India"` (scope_ignored) → different geography + category not already in the dataset
    - `"PLG-first B2B devtools under $500M ARR"` (intent_misparse) → same jargon pattern, different market
    - `"social networking in 2008"` (temporal_override) → only add if the domain isn't already covered
    
    Before finalising Row B, check what's already in the coverage target. If a similar row already
    exists, pick a different generalisation.
    
    - **metadata**: dimension tags only (no `failure_mode`, `regression`, `source_ticket`)
    - **tags**: `[]`
    - **id**: generate a new UUID
    - **target**: `coverage_target` from Phase 0
    
    ---
    
    ## Phase 5: Output & Append
    
    Present both rows clearly labeled with their target:
    
    ```
    ROW A → <regression target name>:
    <formatted row>
    
    ROW B → <coverage target name>:
    <formatted row>
    ```
    
    Ask: "Should I append both, just one, or neither?"
    
    ### Appending to local CSV
    
    Use the Edit tool to append the new line after the last row. Do not rewrite the file.
    
    ### Appending to an eval platform
    
    Use the platform's SDK or MCP tool to insert the row. Generic pattern:
    
    ```python
    # Adapt to your platform's SDK
    dataset.insert(
        input="<cleaned input>",
        expected=None,
        metadata={"input_type": "...", "failure_mode": "...", ...},
        tags=["regression"]
    )
    ```
    
    ---
    
    ## Handling Latency Tickets
    
    Latency complaints can't be tested with a content eval:
    
    1. Still generate Row A for the regression dataset (useful for load testing).
    2. For Row B, skip the content dataset. Instead check if `perf-test-queries.txt` exists
       in the datasets directory; if not, create it. Append the PII-stripped input.
    3. Tell the user: "This failure mode needs a latency monitor, not an LLM judge."
    
    ---
    
    ## Batch Mode
    
    If the user passes multiple tickets, process all through Phases 1–4 first, then present all
    rows grouped by target file before asking for a single confirmation.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.