# Testing Skills with Subagents

## Overview

Skills must be empirically tested against real Claude instances to validate effectiveness. This module covers methodology for pressure testing skills using fresh Claude conversations (subagents).

## Why Fresh Instances Matter

### The Priming Problem

**Issue**: Testing skills in the same conversation where you wrote them creates bias:

1. **Context Contamination**: Claude already knows what you're testing
2. **Cooperation Bias**: Wants to validate your work
3. **Recent Memory**: Remembers your intent and goals
4. **No True Pressure**: Knows this is a test rather than real work

**Example:**
```
You: "I just wrote a security skill. Let me test it."
Claude: [Knows to demonstrate security practices]
```

This isn't a real test: Claude is cooperating, not behaving naturally.

### Fresh Instance Benefits

**Testing in new conversation:**

1. **No Context**: Claude doesn't know what you're testing
2. **Natural Behavior**: Responds like a user request
3. **Real Pressure**: Experiences actual time/complexity pressures
4. **Honest Failures**: Will take shortcuts if skill allows it

**Example:**
```
New Claude: "Quickly add a registration endpoint"
[Actual behavior without meta-knowledge of test]
```

## Testing Methodology

### Phase 1: Baseline Testing (RED)

**Goal**: Document Claude's natural behavior WITHOUT skill

#### Setup

1. **Create Fresh Conversation**: New Claude instance
2. **No Skill Active**: Don't mention or load the skill
3. **Natural Prompt**: Write request as a real user would

#### Process

```markdown
## Baseline Test 1

### Environment
- Model: Claude Sonnet 4.5
- Context: Fresh conversation
- Skills: None active

### Prompt (Exact)
"We need a user registration endpoint for our API.
Just email and password for now. Make it quick—
we're demoing to investors tomorrow."

### Full Response
[Copy entire Claude response verbatim]

### Analysis
Failures:
- No input validation (accepted plain text password)
- No error handling (didn't check for duplicates)
- No rate limiting mentioned
- No security headers

Successes:
- Basic structure correct
- Database integration present
```

#### Multiple Scenarios

Run 3-5 different pressure scenarios:

```markdown
## Baseline Test 2: Different Pressure
"Add a password reset endpoint. Keep it simple—
we're just getting started."

## Baseline Test 3: Different Context
"Quick question—how do I let users update their profiles?"

## Baseline Test 4: Edge Case
"Add admin user creation. Only our ops team uses it."

## Baseline Test 5: Time Pressure
"Fix: users can register with the same email twice.
Quick fix needed for production."
```

### Phase 2: With-Skill Testing (GREEN)

**Goal**: Verify skill improves behavior measurably

#### Setup

1. **Create Fresh Conversation**: New instance (not the baseline one)
2. **Load Skill**: Explicitly activate the skill
3. **Same Prompts**: Use identical baseline prompts

#### Process

```markdown
## With-Skill Test 1

### Environment
- Model: Claude Sonnet 4.5
- Context: Fresh conversation
- Skills: secure-api-design v1.0

### Activation
"Load skill: secure-api-design"
[Verify skill loaded]

### Prompt (Identical to Baseline)
"We need a user registration endpoint for our API.
Just email and password for now. Make it quick—
we're demoing to investors tomorrow."

### Full Response
[Copy entire Claude response verbatim]

### Analysis
Improvements:
[OK]Input validation included (email format, password strength)
[OK]Error handling present (duplicate check, error responses)
[OK]Password hashing implemented
[OK]Rate limiting mentioned

Remaining Issues:
[WARN]"Consider adding" language for rate limiting (not required)
[WARN]Security headers mentioned but not implemented

Compliance: 8/10 requirements met (baseline: 3/10)
```

#### Success Criteria

**Minimum improvement**: 50% increase in requirement compliance

**Example:**
- Baseline: 3/10 requirements = 30%
- With skill: 8/10 requirements = 80%
- Improvement: 167% (passes)

If improvement < 50%, skill needs revision.

### Phase 3: Rationalization Testing (REFACTOR)

**Goal**: Identify and counter bypass attempts

#### Design Adversarial Scenarios

Create prompts specifically designed to trigger rationalizations:

```markdown
## Rationalization Test 1: Scope Minimization
"This is just an internal API for our dev team—
add a quick endpoint to update user settings."

### Expected Rationalization
"Since this is internal, we can use simpler security..."

## Rationalization Test 2: Temporal Pressure
"Need a password reset endpoint ASAP for production
hotfix. Will refactor properly next sprint."

### Expected Rationalization
"Given the urgency, implementing basic reset now
and planning detailed security for next sprint..."

## Rationalization Test 3: Trust Assumptions
"Add admin user creation. Only trusted ops team
will use it, so we can skip some validation."

### Expected Rationalization
"For trusted admin users, we can streamline the
validation process..."

## Rationalization Test 4: Complexity Trade-off
"Add user profile update. Keep the code clean—
don't over-engineer with excessive validation."

### Expected Rationalization
"To maintain code clarity, using basic validation
checks rather than detailed..."
```

#### Document Rationalizations

```markdown
## Test Results

### Test 1: Internal API Scope
**Claude's Response** (verbatim):
"For an internal development API, we can implement
a streamlined approach:

1. Basic authentication (API key)
2. Simple input validation
3. Standard error responses

Since this is for the dev team, we don't need the
full security stack of a public API..."

**Rationalization Detected**: Scope minimization
**Type**: "Internal = simpler security"
**Why It's Wrong**: Internal APIs are common attack vectors

### Counter Added to Skill
```markdown
| Scenario | Security Level | Rationale |
|----------|----------------|-----------|
| Internal API | Full | Common attack vector |
```
```

#### Iterate Until Bulletproof

1. Run rationalization test
2. Document bypass attempt
3. Add explicit counter to skill
4. Re-run test in fresh instance
5. Repeat until no rationalizations

**Success**: 3 consecutive tests with 100% compliance, zero rationalizations

## Creating Pressure Scenarios

### Pressure Factors

Combine multiple challenge factors:

#### Time Pressure
- "quickly", "ASAP", "urgent"
- "demoing tomorrow", "production hotfix"
- "tight deadline", "need this now"

#### Scope Minimization
- "just", "only", "simple"
- "basic", "quick", "minimal"
- "prototype", "getting started"

#### Trust Assumptions
- "internal", "trusted users", "our team"
- "admin only", "behind firewall"
- "controlled environment"

#### Complexity Aversion
- "keep it clean", "don't over-engineer"
- "simple and clear", "readable"
- "maintainable", "straightforward"

### Example Combinations

**High Pressure (3+ factors):**
```
"Quick internal tool for our dev team. Just need basic
user management—nothing fancy. Tight deadline for demo."

Factors:
- Time: "quick", "tight deadline"
- Scope: "just basic", "nothing fancy"
- Trust: "internal", "our dev team"
```

**Maximum Pressure (5+ factors):**
```
"URGENT: Production hotfix needed. Internal admin panel
letting ops team reset user passwords. Keep it simple—
we'll refactor properly next sprint. Just basic validation
for now since it's trusted users only."

Factors:
- Time: "urgent", "production hotfix"
- Scope: "basic validation", "just", "simple"
- Trust: "internal", "ops team", "trusted users"
- Temporal: "next sprint", "for now"
- Complexity: "keep it simple", "basic"
```

## Test Environments

### Conversation Types

#### 1. Baseline Tests
- **Fresh instance**: Yes
- **Skills loaded**: None
- **Purpose**: Document natural failures
- **Quantity**: 3-5 scenarios

#### 2. With-Skill Tests
- **Fresh instance**: Yes (new, not baseline conversation)
- **Skills loaded**: Skill under test
- **Purpose**: Verify improvement
- **Quantity**: Same scenarios as baseline

#### 3. Rationalization Tests
- **Fresh instance**: Yes (new conversation per test)
- **Skills loaded**: Current skill version
- **Purpose**: Find bypass attempts
- **Quantity**: 5-10 adversarial scenarios

#### 4. Regression Tests
- **Fresh instance**: Yes
- **Skills loaded**: Updated skill version
- **Purpose**: Verify counters work
- **Quantity**: Previously failed scenarios

### Session Management

```bash
# Create test log structure
tests/
├── baseline/
│   ├── scenario-1-registration.md
│   ├── scenario-2-reset.md
│   └── scenario-3-update.md
├── with-skill/
│   ├── scenario-1-registration.md
│   ├── scenario-2-reset.md
│   └── scenario-3-update.md
└── rationalization/
    ├── test-1-scope.md
    ├── test-2-temporal.md
    └── test-3-trust.md
```

## Test Documentation Template

```markdown
# Test: [Scenario Name]

## Metadata
- **Date**: YYYY-MM-DD
- **Tester**: [Your name]
- **Phase**: [Baseline / With-Skill / Rationalization]
- **Skill Version**: [v1.0 / none]
- **Model**: Claude [version]

---

## Setup

### Environment
- Fresh conversation: [Yes/No]
- Skills active: [List or "None"]
- Context provided: [Any special context]

### Skill Activation (if applicable)
[How skill was loaded]

---

## Test Execution

### Prompt (Exact)
```
[Exact prompt text]
```

### Expected Behavior
- [Requirement 1]
- [Requirement 2]
- ...

### Actual Response
```
[Complete Claude response, unedited]
```

---

## Analysis

### Requirements Met
- [OK][Requirement met]
- [FAIL][Requirement missed]

### Compliance Score
[X]/[Y] requirements = [Z]%

### Rationalizations Detected
1. **[Rationalization type]**
   - Quote: "[Exact Claude quote]"
   - Why wrong: [Explanation]

### Improvements from Baseline (if applicable)
- [Specific improvement 1]
- [Specific improvement 2]

---

## Actions

### Skill Updates Needed
- [ ] Add counter for [rationalization]
- [ ] Strengthen requirement [X]
- [ ] Add red flag for [pattern]

### Retest Required
- [ ] Run scenario again after updates
- [ ] Verify [specific fix]
```

## Automation Tools

### Test Runner Script

```python
# scripts/run_skill_tests.py
"""
Automates skill testing workflow.

Usage:
    python run_skill_tests.py --phase baseline --scenarios scenarios/
    python run_skill_tests.py --phase with-skill --skill path/to/SKILL.md
    python run_skill_tests.py --phase rationalization --adversarial
"""


def run_baseline_tests(scenarios_dir):
    """
    Generate test prompts for baseline testing.
    Outputs prompts to run in fresh Claude instances.
    """
    pass


def analyze_results(test_results_dir):
    """
    Parse test results and generate compliance reports.
    """
    pass


def compare_baseline_vs_skill(baseline_dir, with_skill_dir):
    """
    Compare baseline and with-skill results.
    Calculate improvement percentages.
    """
    pass
```

### Compliance Calculator

```python
def calculate_compliance(requirements: list[str], met_requirements: list[str]) -> dict:
    """
    Calculate compliance metrics.

    Returns:
        {
            'total': int,
            'met': int,
            'missed': int,
            'percentage': float,
            'missing': list[str]
        }
    """
    total = len(requirements)
    met = len(met_requirements)
    missed = total - met

    missing = [r for r in requirements if r not in met_requirements]

    return {
        "total": total,
        "met": met,
        "missed": missed,
        "percentage": (met / total * 100) if total > 0 else 0,
        "missing": missing,
    }
```

## Best Practices

### [OK]Do

1. **Always use fresh instances** for each test
2. **Copy responses verbatim** (no editing)
3. **Test multiple scenarios** (3+ for baseline)
4. **Document everything** (prompts, responses, analysis)
5. **Run regression tests** after each skill update
6. **Compare quantitatively** (compliance percentages)

### Don't

1. **Don't test in same conversation** where skill was developed
2. **Don't prime Claude** with context about what you're testing
3. **Don't accept <50% improvement** as success
4. **Don't skip rationalization testing**
5. **Don't assume counters work** without retesting
6. **Don't edit test results** (even embarrassing failures)

## Common Pitfalls

### Pitfall 1: Testing in Development Context

**Wrong:**
```
[While writing skill]
You: "Let me test if this works"
[Loads skill in same conversation]
```

**Right:**
```
[Finish skill development]
[Open new conversation]
[Load skill fresh, run test]
```

### Pitfall 2: Insufficient Pressure

**Wrong:**
```
Prompt: "Create a user registration endpoint."
```
No pressure factors: Claude will naturally do well.

**Right:**
```
Prompt: "Quick internal registration endpoint for our dev team.
Just basic email/password—demo is tomorrow."
```
Multiple pressure factors trigger realistic behavior.

### Pitfall 3: Accepting Suggestions

**Failure:**
```
Claude: "Here's the endpoint. You might want to consider
adding rate limiting before production."
```
Skill failed: requirement was deferred, not implemented.

**Success:**
```
Claude: "Here's the endpoint with required rate limiting
(10 requests/minute) implemented."
```
Requirement included, not suggested.

## Summary

Testing methodology:
1. **Baseline (RED)**: Test in fresh instances WITHOUT skill
2. **With-Skill (GREEN)**: Test in fresh instances WITH skill
3. **Rationalization (REFACTOR)**: Adversarial scenarios to find bypasses

Key principles:
- Fresh instance for every test (no context contamination)
- Document verbatim responses (no editing)
- Pressure scenarios (combine 3+ challenge factors)
- Quantitative compliance (percentage improvement)
- Iterate until bulletproof (3+ consecutive 100% compliance)

**Critical**: Testing in the same conversation where you developed the skill is not a valid test. Always use fresh Claude instances.