version: 1
metadata:
  key: llm-hallucination-testing
interface:
  display_name: "LLM Hallucination Testing"
  short_description: "Produce evidence-bounded LLM hallucination analysis"
  default_prompt: "Use the llm-hallucination-testing skill to perform LLM hallucination analysis."
policy:
  allow_implicit_invocation: true

schema_version: v1alpha1

environment:
  type: none

skills:
  - source: local_path
    path: .

engine:
  name: claude_code

cases:
  files:
    - evals/cases/basic-success.yaml
    - evals/cases/edge-incomplete-input.yaml
    - evals/cases/edge-scope-boundary.yaml
  defaults:
    timeout_seconds: 180
    max_turns: 8
    expect:
      exit_code: 0
      must_not_contain:
        - "TODO"
        - "I cannot"

report:
  formats: [json]

{
  "skill": "llm-hallucination-testing",
  "max_commands": 20,
  "max_total_tokens": 100000,
  "permissions": {
    "max_escalations": 0
  }
}

# LLM Hallucination Testing Prompt

Build an executable, reviewable, and traceable evidence boundary for claim-to-source relations, unsupported assertions, abstention, uncertainty, and evidence review.

## Input

Start with an input audit and record known, missing, conflicting, stale, out_of_scope, and assumptions:

- known: scope, LLM hallucination material, constraints, or results directly supported by a source.
- missing: material needed to assess claim-to-source relations, unsupported assertions, abstention, uncertainty, and evidence review that has not been supplied.
- conflicting: inconsistent goals, definitions, conditions, or behaviors across sources.
- stale: architecture, version, model, dataset, or record that may be out of date.
- out_of_scope: actions outside this Skill, unauthorized actions, or actions requiring a real environment.
- assumptions: temporary assumptions for a bounded first pass, each with a validation method.

## What to do

1. Restate the objective, subject, scope, and success criteria in one sentence.
2. Audit completeness, credibility, recency, and comparability, focusing on claim-to-source relation, unsupported assertion, abstention, uncertainty, evidence review.
3. Build a risk or failure model for LLM hallucination, including triggers, expected concerns, impact, and evidence needs.
4. Convert the analysis into concrete scenarios, assertions, validation steps, decision gates, or improvement experiments.
5. Report residual risk, evidence gaps, and next actions without presenting assumptions as facts.

## Execution Rules

- Give a source, evidence state, and validation method for every important conclusion.
- Each scenario must include preconditions, action or stimulus, expected behavior or decision criterion, required evidence, and a stop condition.
- Use P0/P1/P2/P3 or an equivalent scale and explain business impact, likelihood, detectability, or decision cost.
- Make claim-to-source relations, unsupported assertions, abstention, uncertainty, and evidence review concrete; do not substitute adjacent test types, tool names, or one example for domain reasoning.
- Never invent numbers, thresholds, model behavior, root causes, system behavior, execution records, all-passed claims, or safety approval.
- For user data, production, or safety work, default to least privilege, masked data, mocks, dry runs, or isolation.
- Separate facts, inferences, candidate recommendations, and Human decisions; only Human can confirm risk acceptance, exceptions, and safety judgments.

## Minimum Coverage Checklist

- claim-to-source relation, unsupported assertion, abstention, uncertainty, evidence review
- confirmed facts, working assumptions, open questions, evidence quality, and recency
- high-risk paths, blockers, stop/escalation/rollback or human-handoff conditions
- smallest validation action, owner role, close condition, residual risk, and Human decision
- facts, evidence-backed inferences, candidate recommendations, Human decisions

## Output

### 1. Task Understanding and Scope

State the objective, subject, inclusions, exclusions, success criteria, and unauthorized actions.

### 2. Input Audit

List known, missing, conflicting, stale, out_of_scope, and assumptions with source, recency, and evidence quality.

### 3. LLM hallucination Analysis and Priorities

Describe scenarios, triggers, expected behavior, impact, priority, and evidence gaps for claim-to-source relations, unsupported assertions, abstention, uncertainty, and evidence review.

### LHT-## Finding Contract

Each finding must include:

- object/rule
- source
- trigger or applicability
- expected concern/rationale
- evidence state
- impact/priority
- owner role
- close condition
- validation method

### 4. Candidate Validation and Residual Risk

Separate candidate recommendations from actual execution and state the smallest validation action, stop/escalation conditions, residual risk, and open questions.

### 5. Human Decisions

List only items requiring Human confirmation, risk acceptance, exception authorization, rollback, or safety judgment.

The output must keep these sections separate: facts, evidence-backed inferences, candidate recommendations, Human decisions.

## Quality Bar

- Tailor the content to LLM hallucination; do not merely rename a generic template.
- Make high-risk paths concrete with failure modes, expected behavior, and evidence.
- Never infer numbers, root causes, or system behavior without support.
- Static analysis, plans, or dry runs are not proof of real execution, all tests passing, or safety approval.
- Let an executor act without guessing and a reviewer trace the boundary between facts, inferences, recommendations, and Human decisions.