mutation-testing
Configures mewt or muton mutation testing campaigns — scopes targets, tunes timeouts, and optimizes long-running runs. Use when the user mentions mewt, muton, mutation testing, or wants to configure or optimize a mutation testing campaign.
Install
npx skills add https://github.com/trailofbits/skills/tree/main/plugins/mutation-testing/skills/mutation-testing
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install trailofbits-skills@llmmart
git clone https://github.com/trailofbits/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole trailofbits/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Mutation Testing (mewt/muton)
Routes to the right mutation testing workflow and loads the references that workflow needs.
Note: muton and mewt share identical interfaces. Examples use
mewt; substitutemutonand its file names (muton.toml,muton.sqlite) for muton projects.
mewt --help and mewt <subcommand> --help are the source of truth for command-line behavior. Examples below reflect the mewt 4.x API; run --help when a flag looks unfamiliar or a command fails.
When to Use
Use this skill when the user:
- Mentions "mewt", "muton", or "mutation testing"
- Wants to configure, scope, or speed up a mutation testing campaign
- Wants to analyze mutation results — surviving/uncaught mutants, equivalent mutants, kill rate
- Wants to use mutation results to find bugs in the source code
When NOT to Use
Do not use this skill when the user asks about tests or line coverage without any mutation testing context.
Routing
Pick the workflow, then load it together with the references listed for it. Workflows and references do not load each other — that decision belongs here.
Setting up, scoping, or speeding up a campaign → workflows/configuration.md → Also load references/optimization-strategies.md when the campaign estimate is long enough to need trimming, or the user asks to make it faster.
Campaign finished, hunting for bugs in untested code → workflows/bug-hunter.md
Turning results into a formal analysis report → workflows/analyzing-results.md, plus:
- references/equivalent-mutants.md — equivalence catalog and verification procedure
- references/severity-classification.md — severity tier criteria
- references/report-template.md — report structure
- references/blockchain-patterns.md — only for Solidity, Move, FunC/Tolk, Cairo, or Solana Rust targets
- references/input-formats.md — unless the results came from mewt or muton. Foreign tool output may not be self-describing; this covers the parsing anchors for slither-mutate, mull, and dextool-mutate
Anything else → run mewt --help or mewt <subcommand> --help, then assist directly.
Essential Commands
# Set up and run
mewt init # Create config and database
mewt mutate [paths] # Generate mutants without testing them
mewt run [paths] # Generate mutants and run the campaign
# Read results
mewt status # Overview with per-file breakdown
mewt results # Uncaught mutants (default view)
mewt results --all # Every outcome, not just uncaught
mewt results --format json # json | sarif | ids | table
# Narrow down (these filters work on both `results` and `print mutants`)
mewt results --target 'src/auth/**' # Quote globs so the shell does not expand them
mewt results --severity high,medium
mewt results --mutation-types ER,CR
mewt results --status Uncaught # Uncaught | TestFail | Skipped | Timeout
mewt results --line 42
# Investigate and re-test
mewt print mutant --id [id] # View the mutated code
mewt test --ids [ids] # Re-test specific mutants
mewt test --ids-file uncaught_ids.txt # Re-test IDs from a file, or '-' for stdin
# Inspect configuration
mewt print config # Effective config
mewt print targets # Files actually mutated
mewt print mutations --language [lang] # Mutations and severities for a language
Language labels are canonical family or family/dialect values in mewt 4.x — for example rust, javascript/ts, move/sui, move/iota.
What Results Mean
- Caught/TestFail: tests detected the mutation (good)
- Uncaught: tests did not detect the change. Inspect the code to distinguish a testing gap from an equivalent mutation.
- Timeout: tests took too long — inconclusive, not evidence of coverage
- Skipped: a less severe mutant was skipped because a more severe mutant on the same line was uncaught
Interpreting Mutation Types
mewt print mutations --language [lang] lists every mutation slug, description, and severity for a language, and is authoritative — the operator set grows with each release. What that output does not tell you is what a survivor means, which is where prioritization comes from:
| Severity | Representative slugs | What an uncaught mutant tells you |
|---|---|---|
| High | ER (Error Replacement) |
Tests tolerate the injected error. Investigate whether the path executes, whether error handling masks the change, and whether assertions check the outcome. |
| Medium | CR (Comment Replacement) |
Removing the statement does not fail the tests. Check whether its effects matter and whether assertions observe them. |
| Medium | IF/IT (If False/True), NR (Negation Removal) |
Tests do not distinguish the changed condition. Both constant replacements surviving can indicate an unexecuted condition or weak assertions on the branch outcomes. |
| Low | Operator shuffles (AOS, COS, LOS, BOS, shift/assignment variants), BL, AS, LC, WF |
Check boundary inputs, arithmetic assertions, and semantic equivalence. The mutation result alone does not establish whether the code executed. |
Severity ranks the mutation, not the risk. A low-severity survivor in a fee calculation matters more than a high-severity survivor in a log line — weigh what the mutated code does. Filter with --severity to work through the results in priority order.
Files (skills)
-
agents
-
openai.yaml 243 B
interface: display_name: "Mutation Testing" short_description: "Run mutation campaigns and investigate surviving mutants" icon_small: "assets/trail-of-bits-mark.svg" icon_large: "assets/trail-of-bits-mark.svg" brand_color: "#D83A34"
-
-
assets
-
trail-of-bits-mark.svg 3 KB · in bundle
-
-
references
-
blockchain-patterns.md 4.8 KB
# Blockchain-Specific Patterns Blockchain-specific mutation testing patterns for Solidity, FunC/Tolk, Move, and Solana Rust codebases. These patterns extend the general equivalence catalog and severity criteria; where they disagree with the general guidance, the blockchain-specific rule wins for blockchain code. --- ## Blockchain-Specific Equivalent Mutants Additional equivalence patterns beyond the general catalog. ### Revert String Mutations **Pattern:** `require(condition, "error A")` mutated to `require(condition, "error B")` **Observable difference:** The transaction reverts on the same inputs, but its returned error data changes. Treat this as a testing gap when callers or the documented interface depend on that data. **Applies to:** Solidity `require`, `revert` with custom error messages **Scope:** A property concerned only with whether the call reverts can ignore the message. State that restricted property explicitly. The absence of an assertion on error data does not make two different error messages semantically identical. ### Gas-Only Differences **Pattern:** Mutation in a `view` or `pure` function that changes gas consumption but not the return value or state **Scope:** Identical return values can establish equivalence for a property that explicitly excludes gas consumption, provided neither version runs out of gas. They do not establish identical behavior in every calling context. **Applies to:** Solidity `view`/`pure` functions **Not equivalent when:** The gas difference causes an out-of-gas revert that changes observable behavior, or the function is called in a state-changing context where gas matters. ### Modifier Ordering **Pattern:** Swapping the order of two independent modifiers on a function **When equivalent:** Both guards must be independent, and their order must preserve all relevant observable behavior. Check the error returned when both guards fail, as well as state effects and callbacks. **Applies to:** Solidity function modifiers **Not equivalent when:** The modifiers interact (e.g., one sets a flag that the other reads, or one can revert based on state the other modifies). ### Storage Packing **Pattern:** Mutation to a type within the same storage slot that does not change the packed representation **What to check:** An unchanged storage layout is insufficient. A type change can alter arithmetic, comparisons, or accepted values even when the stored bits match. Trace reads and writes before deciding whether behavior is equivalent. **Caution:** This is very rare. Only classify as equivalent with concrete evidence of packing behavior. --- ## Blockchain Severity Examples Concrete tier assignments for common on-chain constructs, using the same four tiers as the general severity criteria. ### Tier 1: Critical — Security-Sensitive **Solidity:** - `require(msg.sender == owner)` — ownership check - `onlyRole(ADMIN_ROLE)` — role-based access control modifier - `nonReentrant` modifier — reentrancy guard - `ecrecover(hash, v, r, s)` — signature verification - `require(deadline >= block.timestamp)` — time-based access control **FunC/Tolk:** - `throw_unless(ERR_UNAUTHORIZED, equal_slices(sender, owner))` — ownership check - Bounce handler validation — message authentication **Solana Rust:** - `if !ctx.accounts.authority.is_signer` — signer check ### Tier 2: High — Financial/State Integrity **Solidity:** - `balances[to] += amount` — balance update - `fee = amount * feeRate / DENOMINATOR` — fee calculation - `require(ratio > MIN_COLLATERAL_RATIO)` — liquidation threshold - `IERC20(token).safeTransfer(to, amount)` — token transfer - `require(amountOut >= minAmountOut)` — slippage check **FunC/Tolk:** - `send_raw_message(msg, mode)` — value transfer via message - Jetton transfer handler — token movement logic **Solana Rust:** - `account.lamports -= amount` — lamport transfer - `pool.total_shares += new_shares` — share accounting ### Tier 3: Medium — Business Logic **Solidity:** - `require(proposalId < proposals.length)` — bounds check in governance - `rewards[user] += calculateReward(...)` — non-critical reward distribution - `if (queue.length > MAX_QUEUE_SIZE) revert()` — queue management ### Tier 4: Low — Observability **Solidity:** - `emit Transfer(from, to, amount)` — event emission - `function balanceOf(...) view returns (uint256)` — view getter - Error message string in `require(cond, "message")` --- ## Blockchain-Specific Ambiguous Cases ### Protocol Standard Compliance If event emission is required by an ERC or similar blockchain standard (e.g., `Transfer` in ERC-20) and downstream contracts or indexers depend on it, consider **Tier 3** instead of Tier 4. This is stricter than the general case because on-chain protocol compliance is enforced by tooling and integrations. -
equivalent-mutants.md 7 KB
# Equivalent Mutant Catalog Equivalent mutants are mutations that produce semantically identical behavior to the original code. They are false positives in mutation testing: no test can kill them because no observable difference exists. Correctly identifying equivalent mutants prevents inflating the count of real testing gaps. This file covers general equivalence checks and patterns. Blockchain-specific equivalence patterns are loaded separately by `SKILL.md` for blockchain projects. ## Verification Procedure Before classifying a mutant as equivalent, complete all five checks: 1. **Type context**: Determine the types of all variables involved in the mutation. Unsigned integers, booleans, and enums each have specific equivalence patterns 2. **Reachability**: Confirm that the mutated code is actually reachable during execution. Mutations in dead code are trivially equivalent 3. **Downstream consumption**: Trace whether the mutated value is read or used by any subsequent logic. If the value is never consumed, the mutation has no observable effect 4. **Semantic comparison**: Compare the behavior of original and mutated code across all possible inputs within the type constraints. If no input produces a different result, the mutant is equivalent 5. **Distinguishing test attempt**: Try to construct a concrete input that produces different behavior between the original and mutated code. If such an input exists, the mutant is not equivalent — describe the input as a recommended test case. If no distinguishing input can be found after checking boundary values, type extremes, and domain-specific edge cases, this strengthens the equivalence classification A mutant is equivalent only when the analysis establishes identical observable behavior. If a check is uncertain, retain it as unresolved and explain what evidence is missing. Uncertainty alone does not establish a testing gap. --- ## Semantic Equivalence Patterns ### Unsigned Comparison Equivalence **Pattern:** `x > 0` mutated to `x != 0` (or vice versa) **Why equivalent:** For unsigned integers, the only value that is not greater than zero is zero itself. Therefore `x > 0` and `x != 0` produce identical results for all possible unsigned values. **Applies to:** Any unsigned integer type (Rust `u8`-`u128`/`usize`, C/C++ `unsigned`, Solidity `uint`, FunC/Tolk integers when used as unsigned) **Not equivalent when:** The variable is a signed type. For signed integers, `x > 0` excludes negative values while `x != 0` includes them. ### Boolean Double-Negation **Pattern:** `!(!condition)` mutated to `condition` **Why equivalent:** Double negation of a boolean produces the original value. These expressions are logically identical. **Applies to:** All languages ### Tautological Bound Checks **Pattern:** `x >= 0` for an unsigned integer, mutated to `true` **Why equivalent:** Both expressions are true for every value of the unsigned type, provided evaluating `x` has no observable side effects. **Applies to:** Any unsigned integer type (Rust unsigned types, C/C++ `unsigned`, Solidity `uint`) **Not equivalent when:** The replacement is `x > 0`. At `x == 0`, the original returns true and the replacement returns false. Zero is reachable for an unsigned type unless the surrounding code excludes it. Signed types and comparisons such as `x >= 1` also require a separate analysis. ### Dead Code Mutations **Pattern:** Any mutation inside an unreachable branch **Why equivalent:** If the code path is never executed, no mutation within it can produce an observable difference. **How to verify:** Check whether the branch condition can ever be true. Common cases: `if (false)`, branches after unconditional `return`/`revert`, code guarded by compile-time constants that always evaluate one way. **Caution:** A branch that is not exercised by current tests is not necessarily unreachable. Distinguish between "no test covers this" (real finding) and "no input can reach this" (equivalent). ### Redundant Assignment Mutations **Pattern:** Mutation to a variable that is never read after the assignment **Why equivalent:** If the variable's value is not consumed by any subsequent operation, changing the assigned value has no effect on program behavior. **How to verify:** Trace all uses of the variable after the mutated assignment. If no read occurs before the variable is reassigned or goes out of scope, the mutation is equivalent. ### Commutative Operation Reordering **Pattern:** `a + b` mutated to `b + a`, or `a * b` mutated to `b * a` **Why equivalent:** Addition and multiplication are commutative. The operand order does not affect the result. **Applies to:** Built-in integer operations with pure operands and the same types. Inspect overloaded operators and floating-point semantics separately, including observable NaN representations and floating-point exceptions. **Not equivalent when:** The operation has side effects (e.g., function calls as operands with observable side effects), or the operation is non-commutative (subtraction, division, modulo). --- ## Common Mistakes in Equivalence Classification These patterns look equivalent but are NOT. Do not classify them as false positives: ### Boundary Value Changes **Pattern:** `>` mutated to `>=` (or vice versa) **Why NOT equivalent:** The boundary value (where `x == threshold`) produces a different result. This is a real finding unless the boundary value is provably unreachable. **Common error:** Assuming that because current tests do not exercise the boundary, it is unreachable. Unreachability must be proven from the type system or control flow, not from test coverage. ### Off-by-One in Loop Bounds **Pattern:** `i < length` mutated to `i <= length` in a loop **Why NOT equivalent:** The loop executes one additional iteration, which may access out-of-bounds memory or process an extra element. This is a real finding. ### Removed Event Emission **Pattern:** `emit Transfer(from, to, amount)` commented out **Why NOT equivalent in all cases:** While event removal does not change execution logic, it may break downstream monitoring, indexing services, or integration tests that rely on events. Classify as Tier 4 (Low), not as equivalent. ### Swapped Non-Commutative Operations **Pattern:** `a - b` mutated to `b - a`, or `a / b` mutated to `b / a` **Why NOT equivalent:** Subtraction and division are not commutative. The results differ unless `a == b`. ### Undefined Behavior Introduction (C/C++) **Pattern:** Mutation changes a bounds check, pointer operation, or integer arithmetic in a way that introduces undefined behavior (e.g., signed overflow, buffer overrun, null dereference) **Why NOT equivalent:** The mutation may produce identical test output today, but undefined behavior can manifest differently under other compilers, optimization levels, or platforms. UB-introducing mutations represent real safety bugs regardless of current test results. **Common error:** Observing that all tests still pass and concluding the mutation is equivalent. In C/C++, UB can silently corrupt memory, be optimized away, or cause crashes only under specific conditions. -
input-formats.md 2.4 KB
# Input Format Examples Parsing anchors for mutation testing tools with non-standard output formats. Most tools (mutmut, cargo-mutants, mutahunter) produce straightforward JSON or CSV that can be parsed directly from field names. The formats below have non-obvious structure that benefits from explicit examples. --- ## slither-mutate (Solidity) Plain text log with mutation type markers in brackets and file paths. ``` [CR] Mutation: ConditionalOperatorReplacement Original: require(amount > 0) Mutated: require(amount >= 0) File: src/Token.sol Line: 45 Status: SURVIVED ``` **Key fields:** Mutation type in `[CR]`/`[SR]`/`[LOR]`/etc. brackets, `File:`, `Line:`, `Status:` labels. **Parsing anchor:** Lines starting with `[` followed by a two-to-four letter mutation code. --- ## mull (C/C++) JSON output using the mutation-testing-elements schema, SQLite database, or HTML report. JSON (mutation-testing-elements schema): ```json { "files": { "src/parser.c": { "mutants": [ { "id": "mull-1", "mutatorName": "cxx_ge_to_lt", "location": {"start": {"line": 88, "column": 12}}, "status": "Survived", "replacement": "<" } ] } } } ``` **Key fields:** `files.<path>.mutants[]` array, `mutatorName` (e.g., `cxx_ge_to_lt`, `cxx_add_to_sub`, `cxx_remove_void_call`), `location.start.line`, `status`. **Parsing anchor:** Top-level `"files"` key with nested `"mutants"` arrays. Mutator names use the `cxx_` prefix. **SQLite format:** Tables include `mutant`, `mutation_point`, and `mutation_result`. Export to CSV or JSON before analysis. --- ## dextool-mutate (C/C++) HTML, JSON, or SQLite report with conventional mutation operator names. JSON format: ```json { "mutants": [ { "id": 157, "file": "src/buffer.c", "line": 42, "operator": "ROR", "original": "size > capacity", "mutation": "size >= capacity", "status": "alive" } ] } ``` **Key fields:** `file`, `line`, `operator`, `original`, `mutation`, `status`. **Parsing anchor:** Operator codes follow conventional names: AOR (arithmetic operator replacement), ROR (relational operator replacement), LCR (logical connector replacement), SDL (statement deletion), UOI (unary operator insertion). Filter for `"status": "alive"`. **SQLite format:** Main table is `mutation` with columns `mut_id`, `file`, `line`, `operator`, `status`. Export to CSV or JSON before analysis. -
optimization-strategies.md 8.7 KB
# Optimization Strategies Apply these strategies **before** running a campaign when Phase 3 of the configuration workflow requires optimization (estimated >16 hours or user requests). --- ## Priority 1: Verify Target Selection **Most common issue:** Mutating non-source code. **Diagnostic:** ```bash mewt print config # Check [targets] include/ignore mewt print targets # Check what was actually mutated ``` **Look for unintended files:** - Mocks: `src/mocks/`, `__mocks__/` - Tests: `*_test.rs`, `*.test.js`, `tests/` - Dependencies: `vendor/`, `node_modules/` - Generated: `proto/`, `generated/` **Fix:** Update `[targets]` in `mewt.toml` to be more specific: ```toml # Before (too broad) [targets] include = ["**/*.rs"] # After (specific) [targets] include = ["src/**/*.rs", "lib/**/*.rs"] ignore = ["test", "mock", "generated"] ``` Re-run `mewt mutate` and check new count. --- ## Priority 2: Analyze Project Structure **Goal:** Understand mutant distribution and test organization to choose the right optimization. **1. Get mutant counts per component:** ```bash # Quote globs to prevent shell expansion. IDs output contains one mutant per line. mewt print mutants --target 'src/auth/**/*.rs' --format ids | wc -l mewt print mutants --target 'src/core/**/*.rs' --format ids | wc -l mewt print mutants --target 'src/utils/**/*.rs' --format ids | wc -l ``` Present breakdown to user: ``` Component breakdown: - src/auth/: 200 mutants × 5s = ~17 min - src/core/: 800 mutants × 8s = ~1.8 hrs - src/utils/: 150 mutants × 3s = ~8 min Total: 1150 mutants, ~2.3 hrs worst-case ``` **2. Count mutations by severity:** ```bash # Check enabled mutation types mewt print config | grep mutations # Count by severity level mewt print mutants --severity high --format ids | wc -l mewt print mutants --severity medium --format ids | wc -l mewt print mutants --severity low --format ids | wc -l # Or count specific mutation types mewt print mutants --mutation-types ER --format ids | wc -l mewt print mutants --mutation-types CR --format ids | wc -l # Compare to total mewt print mutants | wc -l ``` Example output: ``` High/Medium severity: 450 mutants Total mutants: 1200 Percentage: 37.5% ``` **Note:** The percentage varies drastically between codebases (15% to 50+ % is common). --- ## Priority 3: Choose Optimization Approach Based on project structure analysis, present options to user with concrete time estimates: ### Option A: Run Full Campaign - "Estimated ~X hours worst-case (likely faster in practice)" - "Recommend starting Friday evening for weekend completion" - **When to suggest:** Duration acceptable, comprehensive coverage desired ### Option B: Target Critical Components - "Focus on specific components: src/auth/ (~17 min), src/crypto/ (~45 min)" - "Start with one component and expand scope after review?" - **When to suggest:** Clear component boundaries, user wants rapid iteration **Implementation:** ```toml [targets] # Start with critical component include = ["src/auth/**/*.rs"] # After review, expand scope # include = ["src/auth/**/*.rs", "src/core/**/*.rs"] ``` After editing `mewt.toml`, purge removed targets then mutate any newly included files: Preserve any results that must be retained and confirm that discarding the affected campaign data is authorized before purging. ```bash mewt purge # removes targets no longer matching [targets].include/ignore mewt mutate src/ # adds mutants for any newly included files mewt status # confirm reduced mutant count ``` ### Option C: High/Medium Severity Only - "Limit to high/medium severity mutations (X mutants, ~Y hours)" - "Low severity (operator shuffles) tests edge cases, less critical" - **When to suggest:** Time-constrained, need actionable findings quickly **Implementation (by severity level):** ```toml [run] mutations = ["ER", "CR", "IF", "IT"] # Specific types (high/medium) ``` After editing `mewt.toml`, full regeneration is required since existing mutants may no longer be valid under the new filter: ```bash mewt purge --all # clear all existing mutants mewt mutate src/ # regenerate with restricted mutation types mewt status # confirm reduced mutant count ``` Or use severity filtering during analysis instead (no database changes needed): ```bash # Run all mutants but filter results by severity mewt results --severity high,medium mewt print mutants --severity high ``` **Trade-offs to explain:** - High/med severity: ~30-40% of mutants (varies by codebase) - Low severity: ~60-70% of mutants (operator shuffles, edge cases) - Low severity still provides value, just lower priority - Using severity filters during analysis allows flexibility without re-running campaign ### Option D: Two-Phase Campaign (Integration-Heavy Only) - "Phase 1: Targeted tests (estimable upfront), Phase 2: Re-test uncaught with full suite (duration depends on Phase 1 survivor count)" - "Total: Phase 1 estimate + (survivors × full-suite time) vs naive total" - **When to suggest:** Integration tests dominate, unit tests don't map cleanly to files See Two-Phase Campaigns section below for detailed setup. --- ## Two-Phase Campaigns **Use ONLY for integration-heavy test suites.** Not recommended for well-organized unit tests. ### When to Use **Good fit:** - Integration tests dominate runtime - Unit tests provide broad coverage but don't map cleanly to specific files - Targeted test commands significantly faster than full suite **Not recommended:** - Well-organized unit tests with clear file mappings - Tests already fast and targeted ### Setup **Phase 1 config (targeted tests):** ```toml # TWO-PHASE CAMPAIGN # Phase 1: Targeted tests (duration estimable upfront) # Phase 2: Re-test uncaught mutants (duration depends on Phase 1 survivor count) [test] # PHASE 2: Uncomment after phase 1 completes # cmd = "cargo test" # timeout = 60 # PHASE 1: Targeted tests [[per_target]] glob = "src/auth/*.rs" test.cmd = "cargo test auth::unit" test.timeout = 10 [[per_target]] glob = "src/core/*.rs" test.cmd = "cargo test core::unit" test.timeout = 15 # Catch-all: full suite for any file not matched above. # Required unless [targets] is scoped to exactly the globs listed above. [[per_target]] glob = "**/*.rs" test.cmd = "cargo test" test.timeout = 60 ``` **Rationale:** Phase 1 uses fast targeted tests. Phase 2 re-tests only the survivors with the comprehensive suite. ### Execution **Phase 1:** ```bash mewt run ``` Wait for completion. **Phase 2 (after phase 1 completes):** 1. **Extract uncaught mutants:** ```bash mewt results --status Uncaught --format ids > uncaught_ids.txt ``` 2. **Update mewt.toml:** - Comment out all `[[per_target]]` sections (including the catch-all) - Uncomment Phase 2 `[test]` section 3. **Re-test with full suite:** ```bash mewt test --ids-file uncaught_ids.txt ``` 4. **Review final results:** ```bash mewt results # Remaining uncaught are true coverage gaps ``` **Example speedup:** ``` Naive approach: 2,000 mutants × 45s = 25 hours Two-phase approach: Phase 1: 2,000 mutants × 8s = 4.4 hours → 450 uncaught (example outcome) Phase 2: 450 uncaught × 45s = 5.6 hours → 180 truly uncaught Total: ~10 hours (2.5× speedup) Note: Phase 2 duration is unknowable before Phase 1 completes — it depends entirely on how many mutants survive. The figures above illustrate one possible outcome. Present Phase 1 as a firm estimate; present Phase 2 as (survivors × full-suite time) once Phase 1 results are available. ``` --- ## Per-Target Test Configuration **Use when:** Tests are well-organized by module/file, and running targeted tests is significantly faster than the full suite. ### Setup Pattern ```toml # Test full suite for every mutant (slow but comprehensive) [test] cmd = "go test ./..." timeout = 45 # ALTERNATIVE: Targeted tests per file (fast, may miss cross-module failures) [[per_target]] glob = "auth/*.go" test.cmd = "go test ./auth" test.timeout = 10 [[per_target]] glob = "core/*.go" test.cmd = "go test ./core" test.timeout = 15 [[per_target]] glob = "utils/*.go" test.cmd = "go test ./utils" test.timeout = 8 # Catch-all for unmatched files [[per_target]] glob = "*.go" test.cmd = "go test ./..." test.timeout = 45 ``` **Ordering matters:** First match wins. Place most specific patterns first, catch-all last. ### Verify Speedup ```bash time go test ./... # Full suite: 45s time go test ./auth # Targeted: 8s ``` If targeted tests aren't significantly faster, this optimization won't help. ### Trade-offs **Benefits:** - Faster campaign execution - Scales linearly with codebase size **Risks:** - May miss cross-module integration bugs - Requires correct glob-to-test mapping **Mitigation:** - Use this for initial passes - Consider two-phase approach for comprehensive validation -
report-template.md 4.3 KB
# Mutation Testing Analysis Report **Project:** [Project Name] **Repository:** [Repository URL or Path] **Analysis Date:** [Date] **Mutation Testing Tool:** [Tool Name and Version] **Analyzer:** [Agent and model, if known] with mutation-testing skill --- ## Executive Summary | Metric | Value | |--------|-------| | Total mutants generated | [N] | | Mutants killed | [N] | | Mutants survived | [N] | | Equivalent mutants (false positives) | [N] | | Real surviving mutants | [N] | | Unresolved survivors | [N] | | Skipped or inconclusive mutants | [N] | | Mutation kill rate (raw) | [N]% | | Adjusted kill rate (excluding equivalent) | [N]% | ### Key Findings [Describe the concrete testing gaps, affected behavior, and recommended tests. Explain the campaign's scope and rate denominators. Do not infer overall test quality from the adjusted kill rate. These findings concern weaknesses in tests, not confirmed vulnerabilities in the original code.] ### Severity Distribution | Severity Tier | Count | Percentage of Real Survivors | |---------------|-------|------------------------------| | Critical (Security-Sensitive) | [N] | [N]% | | High (Financial/State Integrity) | [N] | [N]% | | Medium (Business Logic) | [N] | [N]% | | Low (Observability) | [N] | [N]% | --- ## Equivalent Mutants (False Positives) [One to two sentences explaining what equivalent mutants are and how many were identified in this analysis.] | # | File | Line | Mutation | Equivalence Reason | |---|------|------|----------|-------------------| | 1 | [path/to/file] | [N] | `[original]` → `[mutated]` | [Brief explanation of why this is a false positive] | --- ## Tier 1: Critical — Security-Sensitive Findings [One sentence introducing the tier and the number of findings.] ### Finding C-1: [Descriptive title] **File:** `[path/to/file]` **Line:** [N] **Mutation:** `[original code]` → `[mutated code]` **Mutation Operator:** [operator type, if known] **Context:** [One to two sentences describing what the mutated code does and what security property it enforces.] **Responsible test file(s):** `[path/to/test/file]` **Relevant test case(s):** `[test function name(s)]` **Why this was not caught:** [One to two paragraphs. Reference the existing test code and explain what assertion, branch, or edge case is missing. Be specific about the gap.] **Recommended test improvement:** [Specific, actionable description of what to add or modify in the test. Do not include code snippets — describe the change in prose.] --- [Repeat for each Tier 1 finding: C-2, C-3, ...] ## Tier 2: High — Financial/State Integrity Findings [One sentence introducing the tier and the number of findings.] ### Finding H-1: [Descriptive title] [Same structure as Tier 1 findings.] --- [Repeat for each Tier 2 finding: H-2, H-3, ...] ## Tier 3: Medium — Business Logic Findings [One sentence introducing the tier and the number of findings.] ### Finding M-1: [Descriptive title] [Same structure as Tier 1 findings.] --- [Repeat for each Tier 3 finding: M-2, M-3, ...] ## Tier 4: Low — Observability Findings [One sentence introducing the tier and the number of findings.] ### Finding L-1: [Descriptive title] [Same structure as Tier 1 findings.] --- [Repeat for each Tier 4 finding: L-2, L-3, ...] ## Recommendations Organize all recommendations by test file, grouping related improvements across severity tiers. For each test file, list the findings it should address and summarize the changes needed. ## Unresolved Cases [For each unresolved survivor, state its location, the uncertain behavior, and the evidence needed to classify it. Omit this section if there are none.] --- ## Appendix ### Input Files Analyzed - [List of mutation result input files provided by the user] ### Source Files with Surviving Mutants | File | Total Surviving | Critical | High | Medium | Low | |------|----------------|----------|------|--------|-----| | [path/to/file] | [N] | [N] | [N] | [N] | [N] | ### Methodology This analysis follows Trail of Bits' mutation testing analysis methodology. Surviving mutants are classified by the impact type of the mutated code, not by the mutation operator used. Equivalent mutants are identified through type-system analysis and reachability checks. For background on mutation testing, see the Trail of Bits blog post "Use mutation testing to find the bugs your tests do not catch." -
severity-classification.md 5.9 KB
# Severity Classification Guide Severity depends on the impact type of the mutated code, not the mutation operator used. The same operator replacement carries different severity depending on whether it occurs in access control logic or in a logging statement. Always classify based on what the code does, not how it was mutated. The tiers prioritize testing gaps. They do not establish that the original code contains a vulnerability. State uncertainty about impact instead of automatically raising the tier. This file covers general severity criteria and examples. Blockchain-specific examples are loaded separately by `SKILL.md` for blockchain projects. --- ## Tier 1: Critical — Security-Sensitive Code Surviving, non-equivalent mutants in code that enforces security invariants. The tests missed a change to a security property. Exploitability in the original code requires a separate investigation. ### Decision Criteria The mutated code falls in Tier 1 if it: - Enforces who can call a function (access control, authorization) - Verifies identity or credentials (authentication, signature checks) - Prevents reentrancy or other concurrency attacks - Validates external input at system boundaries - Performs cryptographic operations (hashing, signing, verifying) - Guards privilege escalation paths (admin functions, upgrades) ### Examples - `assert(caller == admin)` or `require(msg.sender == owner)` — authorization - Signature verification (`verify_signature`, `ecrecover`, `jwt.verify`) - Constant-time comparison (`hmac.compare_digest`, `bcrypt.compare`) - Authentication guards and decorators (`@login_required`, role-check middleware) - Any `require`, `assert`, `if` guard, or middleware that controls access to sensitive operations --- ## Tier 2: High — Financial/State Integrity Code Surviving mutants in code that governs value transfers, accounting, or critical state transitions. An uncaught mutation here means the test suite does not verify that funds or state are handled correctly. ### Decision Criteria The mutated code falls in Tier 2 if it: - Transfers tokens or native currency - Computes fees, shares, exchange rates, or interest - Manages balances or accounting ledgers - Enforces state machine transitions (order of operations) - Uses oracle-dependent price calculations - Enforces slippage or deadline protections - Handles liquidation or collateral thresholds ### Examples - `account.balance -= withdrawal` — balance deduction - `interest = principal * rate * time` or `fee = amount * rate / 100` — financial computation - Share or token accounting (`total_shares += new_shares`, `pool.update()`) - Price or threshold guards (`assert(price >= min_price)`, `if (total > limit) throw`) - Any arithmetic on monetary values, accounting updates, or state transition guards with financial implications --- ## Tier 3: Medium — Business Logic Code Surviving mutants in code that implements protocol-specific behavior without directly handling value or enforcing security boundaries. An uncaught mutation here means functional correctness is not fully tested. ### Decision Criteria The mutated code falls in Tier 3 if it: - Validates configuration or parameters (non-security bounds) - Manages data structures (arrays, mappings, queues) - Implements protocol-specific workflow logic (governance, voting, staking rewards) - Handles integration points with external contracts (callbacks, return values) - Contains error handling or revert conditions for non-security paths ### Examples - Data structure operations (`vec.push(item)`, `tasks.filter(...)`, `sorted(items, ...)`) - Configuration updates and validation (`config.max_retries = value`, `Math.min(value, MAX)`) - Capacity and retry guards (`if queue.len() > MAX_SIZE`, `if retry_count > MAX_RETRIES`) - Cache management (`cache.expire(key, ttl=300)`) - Any non-security, non-financial logic that affects functional behavior --- ## Tier 4: Low — Observability and Informational Code Surviving mutants in code that does not affect execution logic. These represent the lowest priority but still indicate missing test assertions. ### Decision Criteria The mutated code falls in Tier 4 if it: - Emits events or logs - Returns values from view/pure/getter functions used only for display - Contains error message strings (not the revert condition itself) - Updates NatSpec or documentation-reflected constants - Provides monitoring or debugging output ### Examples - Logging and tracing (`log::info!(...)`, `logger.debug(...)`, `console.log(...)`) - Metric emission (`metrics.increment('api.calls')`) - Getters and display methods (`get_name()`, `__repr__`, `displayName`) - Any code that does not affect control flow, state, or return values in non-view contexts --- ## Ambiguous Cases ### Configuration Parameters that Affect Security If a configuration parameter controls a security-sensitive threshold (e.g., maximum number of signers, timeout for timelocks), classify as **Tier 1**, not Tier 3. ### Error Handling in External Calls If the error handling is on a value-bearing external call (e.g., catching a failed token transfer), classify as **Tier 2**. If the error handling is on a non-value call, classify as **Tier 3**. ### Getter Functions Used in State-Changing Logic If a getter or read-only function's return value is consumed by a state-changing function (e.g., a pricing lookup used in a transaction, a permission check used in an update), the getter mutation is **Tier 2**, not Tier 4. ### Event or Log Emissions Required by Consumers If event or log emission is required by downstream consumers (e.g., monitoring systems, indexers, integration contracts, or protocol standards), consider **Tier 3** instead of Tier 4. ### Boundary Between Tier 2 and Tier 3 When unsure whether code is "financial" or "business logic," ask: if this code were wrong in production, would it result in loss of funds or incorrect accounting? If yes, Tier 2. If it would cause incorrect behavior without direct financial impact, Tier 3.
-
-
workflows
-
analyzing-results.md 6.7 KB
# Analyzing Mutation Testing Results Analyzes mutation testing campaign results to identify testing gaps, classify surviving mutants by severity, filter equivalent mutants, and produce a structured report. This workflow is tool-agnostic and language-agnostic. It is used together with the equivalence catalog, severity guide, and report template loaded alongside it; those files hold the criteria, this file holds the procedure. ## When to Use - The user provides mutation testing results (surviving mutants with file paths, line numbers, and code changes) - The user wants to know which survivors are real testing gaps versus equivalent mutants - The user wants a structured report prioritizing which tests to improve ## When NOT to Use - Running mutation testing tools (this workflow analyzes results, it does not generate them) - Writing or modifying test code (it recommends improvements, it does not implement them) - Choosing which mutation testing tool to use - Coverage analysis without mutation data, or code review and vulnerability discovery (use audit-context-building or wilson) ## Rationalizations to Reject **"The kill rate is above 80%, so the test suite is adequate."** Aggregate metrics hide individual critical gaps. One survivor in access control matters more than a hundred killed mutants in logging. **"This is probably an equivalent mutant."** Equivalence is proven from the type system and control flow, never assumed. Absence of a distinguishing test is not proof. **"Boundary changes like `>` to `>=` are always equivalent."** They are equivalent only when the boundary value is provably unreachable. A reachable boundary is a real finding. **"The test suite catches this in practice even though the mutant survived."** If the mutant survived, the suite does not catch it. That is what surviving means. **"Only the top tier is worth reporting."** Lower tiers are still concrete, actionable test improvements. Even event-emission survivors reveal missing assertions that matter for monitoring and integration. **"I can classify this without reading the source."** Severity depends on what the mutated code does, and equivalence depends on types and reachability. Both require the source at the mutation site. --- ## Workflow ### Phase 1: Parse Input **Entry:** The user has provided mutation testing results. 1. Read the input and identify its format. Most tools (mewt/muton, mutmut, cargo-mutants, mutahunter) produce JSON or CSV that parses directly from field names; when the structure is not obvious, read the first 50 lines before proceeding. The input-formats reference covers the tools whose output is not self-describing. 2. Confirm it is mutation testing output — expect file paths, line numbers, and a status field. 3. Extract each surviving mutant: file path, line number, original code, mutated code, mutation operator if available. Filter to surviving/uncaught status only. For mewt/muton projects, `mewt results --format json` gives uncaught mutants directly. **Exit:** Structured list of surviving mutants. --- ### Phase 2: Gather Context **Entry:** Phase 1 complete. Group mutants by source file to batch Read calls, then read each mutation site and establish: what the code does, the types involved, whether the code is reachable, and whether it is security-sensitive, financial, business logic, or observability. **Exit:** Context gathered for every mutant. --- ### Phase 3: Filter Equivalent Mutants **Entry:** Phase 2 complete. Apply the five-check verification procedure from the equivalence catalog to each mutant: type context, reachability, downstream consumption, semantic comparison, and a distinguishing-test attempt. Classify identical behavior as equivalent and an observable difference missed by tests as a testing gap. Keep uncertain cases unresolved and describe the missing evidence. The catalog also lists the patterns that look equivalent but are not — treat those as real findings. **Exit:** Each mutant classified as equivalent, a testing gap, or unresolved. --- ### Phase 4: Classify Severity **Entry:** Phase 3 complete. Assign each confirmed testing gap a priority tier using the criteria in the severity guide, based on what the mutated code does. Explain uncertain impact instead of automatically raising the tier. These tiers prioritize testing work and do not establish vulnerability severity in the original program. **Exit:** Every real mutant has a tier. --- ### Phase 5: Analyze Test Gaps **Entry:** Phase 4 complete. For each real survivor: 1. **Find the responsible test file.** Most ecosystems follow a convention (`src/x.rs` → `tests/x.rs` or an in-file `#[cfg(test)]` module, `x.py` → `test_x.py`, `x.go` → `x_test.go`, `X.sol` → `X.t.sol`). When the convention does not hold, Grep for imports of the mutated module. 2. **Read the tests that exercise the mutated code** and identify which case should have caught the mutation. 3. **Explain why it did not** — missing assertion, uncovered branch, missing edge case, insufficient input variety, or a test that exercises the code without verifying its output. 4. **State concretely how to close the gap:** which assertion to add, which input to use, which branch to cover. If no test touches the mutated code at all, say so — that is a stronger finding than a weak assertion. **Exit:** Every real mutant has a test file, a gap explanation, and a recommendation. --- ### Phase 6: Report **Entry:** Phase 5 complete. Write the report to `mutation-testing-report.md` in the working directory, following the structure in the report template. Compute every statistic from the parsed data. State the denominator for each rate and account for skipped, timed-out, and unresolved cases separately. Report raw and equivalence-adjusted rates only when the supplied data supports them. A rate summarizes this campaign and does not establish overall test adequacy. Match report length to the number of findings. Each finding needs its context, gap explanation, and recommendation; it does not need restating in a summary section, and tiers with no findings can be omitted rather than padded. **Exit:** Report written. --- ## Analysis Requirements - **Every surviving mutant is analyzed individually** — no skipping, summarizing, or batching. Each one gets context, an equivalence check, a severity tier, a test gap, and a recommendation. - **Work in severity order** (Tier 1 and 2 first) so the highest-value analysis is complete if context runs short. - **Save intermediate progress** if the session may end before the analysis does. - Write in formal, objective, third-person voice, active voice, present tense. Use inline code formatting for paths, function names, and snippets. Describe recommended test changes in prose rather than writing out test code. -
bug-hunter.md 7.4 KB
# Bug Hunting with Mutation Testing Uses mutation testing results as a map to untested code, then hunts for real bugs there. ## Core Principle **Uncaught mutants reveal blind spots in testing, and blind spots are where bugs hide.** A missing test is not a bug — it is a signal about where to look. The deliverable of this workflow is confirmed bugs with proof-of-concept reproductions, not a list of coverage gaps. Report an uncaught mutant only after investigating the code behind it and finding something actually wrong. Use this workflow for security audits, code review, risk assessment of legacy code, and pre-release validation. It does not guide test implementation or assess test quality — for a structured report on testing gaps themselves, use the analyzing-results workflow instead. --- ## Prerequisites A completed campaign (`mewt run [paths]`) with results available (`mewt status`). --- ## Workflow ### Step 1: Identify High-Risk Code **Entry:** Campaign finished, results available. 1. Get the overview with `mewt status` and identify files with surviving mutations. Record the campaign scope and inconclusive results before using scores to prioritize. 2. Cross-reference against sensitivity. Security-sensitive areas — auth, crypto, parsing, input validation — and core business logic deserve attention regardless of score; recently changed code with a low score deserves it most. 3. Pull the survivors for those areas: ```bash mewt results --target 'src/auth/**' mewt results --target 'src/crypto/**' --severity high ``` Rank targets as: critical code with a low score, then critical code at any score (crypto at 80% is still worth reading), then core logic with a low score, then everything else. **Exit:** Prioritized list of files to investigate. --- ### Step 2: Prioritize Individual Mutants **Entry:** Step 1 complete. Within those files, order the survivors by what they reveal. Consult the High-Risk Patterns below for the combinations that most reliably indicate bugs. **Investigate immediately:** `ER` in auth, crypto, parsing, or validation; `ER` clusters spanning consecutive lines; `IF` and `IT` both uncaught in security code; `CR` on an authorization check. **Investigate next:** `ER` in error-handling paths; `IF`/`IT` in complex conditionals; `CR` on state-changing operations; any function with several survivors clustered in it. **Lower priority:** isolated operator mutations, survivors in utility functions, and dead code — which is tech debt to remove, not a bug to debug. **Exit:** Prioritized mutant list. --- ### Step 3: Investigate for Bugs **Entry:** Step 2 complete. For each high-priority target: 1. **Read the code.** `mewt print mutant --id <id>` shows the mutation; read the surrounding function for context. Establish the intended behavior. Investigate the original source: a defect deliberately introduced by a mutation is not evidence that the original program is vulnerable. 2. **Determine why it is untested.** Dead code (never called), new code (tests not yet written), an edge case tests never trigger, an error path behind happy-path-only tests, or a unit/integration gap. Each points at a different kind of bug. 3. **Assess risk:** can attacker-controlled input reach this code, what happens if it is wrong, and does it sit on a security boundary? 4. **Look for the bug classes that fit the mutation type:** - `ER` — walk the code path manually; check error handling and input validation for paths that silently succeed - `IF`/`IT` — exercise the untested branch; look for off-by-one errors and unhandled null/empty values - `CR` — check whether the statement is necessary at all, and whether its state change actually happens - Operator mutations — verify the comparison or arithmetic is the intended one, and test the boundary 5. **Try to reproduce.** Write a proof-of-concept test, run it, and confirm the impact. A hypothesis you could not reproduce is a "likely bug," not a confirmed one. **Exit:** Each target resolved as a confirmed bug, a likely bug, dead code, or correct-but-weakly-tested code. --- ### Step 4: Document Findings **Entry:** Step 3 complete. Each finding gets a location, the evidence (mutation result plus what manual testing showed), the issue, impact and severity, a reproduction, and a fix. Example: ````markdown ## Bug: Expired Tokens Accepted **Location:** `src/auth/verify.rs:78` **Evidence:** - Mutation testing: [ER #42] line 78 is completely untested - Manual testing: a token that expired an hour ago verifies successfully **Issue:** The expiry check is inverted. `if token.exp > now { return Err(Expired) }` returns an error for tokens that are still valid and accepts tokens that have expired. The error path has no test coverage, so the inversion went unnoticed. **Impact:** Expired authentication tokens remain valid indefinitely. Severity: CRITICAL. **Reproduction:** ```rust #[test] fn test_expired_token() { let token = create_token_with_expiry(-3600); // expired 1 hour ago assert!(matches!(verify_token(&token), Err(Expired))); // fails today } ``` **Fix:** Compare with `<` on line 78. ```` Classify every finding as **confirmed** (reproducible PoC, verified impact), **likely** (strong evidence, not yet reproduced), **dead code** (unreachable, remove it), or **not a bug** (correct code, weak tests). Keep the last two categories out of the bug list — dead code belongs in a cleanup recommendation. Close with a short summary: campaign statistics, counts per category, the confirmed bugs with locations and severities, and recommendations. **Exit:** Findings documented. --- ## High-Risk Patterns **`ER` clusters.** Several consecutive `ER` survivors can indicate an unexecuted block or tests that tolerate errors. Inspect execution and assertions before choosing either explanation. Determine whether the code is reachable in supported use. **`IF` and `IT` both uncaught on one line.** The tests do not distinguish either constant replacement. The condition might be unexecuted, or both branches might run without assertions on the differing results. Check `mewt results --line 42`, then examine the tests and exercise the branch outcomes. **`ER` in error handling.** Survivors in `catch` blocks and error-return paths mean failure modes are entirely unvalidated. Ask which errors can actually occur, whether an attacker can trigger them, and whether they leak information or crash the process. **Many survivors of mixed types in one security-sensitive file.** Critical code with broadly weak coverage. Review the logic by hand rather than mutant by mutant, looking for bypasses in the security checks. --- ## Working Effectively Investigate one file or area at a time and focus on the top 10-20 priority mutants — comprehensive coverage of a large result set is not the goal, high-impact findings are. Summarize as you go rather than holding every finding in context. Prioritize ruthlessly when there are too many survivors to investigate: start with `ER` in auth, crypto, and validation, favor clusters over isolated mutants, and accept that some areas go uninvestigated. When you cannot tell whether untested code is buggy, test it — write the PoC, run the path, and compare actual behavior against intended behavior. When the code turns out to be correct and only the tests are weak, that is a legitimate outcome; say so. If the investigation confirms no bugs, report that outcome with its scope and unresolved cases. It does not establish that the area is free of bugs. -
configuration.md 11.3 KB
# Configuration and Optimization Guide Guide for configuring mewt and optimizing mutation testing performance **before** running a campaign. ## Goal Configure mewt so the user can run `mewt run` with optimal settings that balance thoroughness and execution time. --- ## Configuration Workflow ### Phase 1: Initialize and Validate Targets **Entry:** User has a codebase and wants to configure mutation testing. **Actions:** 1. **Initialize mewt:** ```bash mewt init # Creates mewt.toml and mewt.sqlite ``` Note: If working with a config in a non-standard location, use `--config path/to/mewt.toml`. The parent directory of the config file becomes the working directory, and relative paths in the config resolve from there. Examples use mewt 4.x. Check the installed tool's `--help` before using unfamiliar flags. For muton, use its command name and configuration filename. 2. **Review auto-generated configuration:** ```bash mewt print config ``` 3. **Verify target patterns:** - **Include patterns** should match only source code: `src/`, `lib/`, `contracts/` - **Ignore patterns** should exclude tests, dependencies, generated code - Note: Ignore patterns use substring matching (e.g., `"test"` matches `tests/`, `test_utils.rs`) 4. **Edit `mewt.toml` if needed** to fix target patterns: ```toml [targets] include = ["src/**/*.rs"] # Specific source directories only ignore = ["test", "mock"] # Exclude test/mock files within src/ ``` **Exit:** `mewt.toml` contains valid target patterns that match intended source files. --- ### Phase 2: Generate Mutants and Assess Scope **Entry:** Phase 1 exit criteria met (valid `mewt.toml` exists). **Actions:** 1. **Generate mutants:** ```bash mewt mutate src/ ``` Note: Output shows per-target summaries with severity breakdown (high/medium/low). Use `--verbose` to see individual mutants. 2. **Check mutant count and distribution:** ```bash mewt status # View total mutant count mewt print targets # Pretty table showing which files were mutated ``` 3. **Time the test command:** ```bash time <test-command-from-config> # e.g., time cargo test ``` Note the baseline test duration. 4. **Calculate worst-case campaign duration:** - Formula: `mutant_count × test_duration` - Example: 500 mutants × 10s = ~1.4 hours - Actual runtime typically faster (tests catch mutants quickly, skipping reduces load) **Exit:** Know the mutant count, test duration, and estimated campaign time. --- ### Phase 3: Decide on Optimization Strategy **Entry:** Phase 2 exit criteria met (mutant count and time estimate known). **Decision Tree:** ``` Estimated campaign duration? | +-- < 1 hour | └─> Proceed to Phase 4 (no optimization needed) | +-- 1-16 hours | └─> Consult user: Acceptable? Run overnight/end-of-day? | +-- User accepts --> Proceed to Phase 4 | +-- User declines --> Apply optimization (see Optimization Strategies below) | +-- > 16 hours └─> Explore optimization options (see Optimization Strategies below) ``` **Actions (if optimization needed):** Read `references/optimization-strategies.md` for detailed strategies and examples. Then: 1. Verify target selection (most common issue — check `mewt print targets` for unintended files) 2. Analyze project structure (`mewt print mutants --target 'src/component/**'` per component) 3. Present options to user with time estimates (full campaign / target critical components / high-severity only / two-phase) 4. Apply chosen optimization to `mewt.toml` 5. **If `[targets]` or `[run].mutations` changed**, update the database and recalculate duration: Purging deletes saved mutants and outcomes. Preserve results that need to be retained and confirm that discarding the affected campaign data is authorized before purging. - **Target scope narrowed** (Option B): purge removed targets, then mutate any newly included files: ```bash mewt purge # removes targets no longer in [targets].include/ignore mewt mutate src/ # adds mutants for any newly included files mewt status # verify reduced mutant count ``` - **Mutation types restricted** (Option C): full regeneration required since existing mutants may no longer be valid: ```bash mewt purge --all mewt mutate src/ mewt status # verify reduced mutant count ``` Update the duration estimate before proceeding to Phase 4. **Exit:** Either campaign duration is acceptable, or `mewt.toml` has been optimized, the database updated, and the new duration estimate confirmed. --- ### Phase 4: Validate Test Command and Timeout **Entry:** Phase 3 exit criteria met (optimization applied if needed). **Actions:** 1. **If test configuration was modified in Phase 3,** verify it works: ```bash <test-command-from-config> # Should succeed without errors ``` Skip this step if Phase 2's timing already validated the unmodified command. 2. **Check if timeout adjustment needed:** **Default:** Mewt auto-calculates timeout (baseline test time × 2), which accounts for incremental recompilation in most cases. **Exception:** For compiled languages where recompilation of dependents dominates test time (Solidity/Foundry): ```bash # Test with warm cache time forge test # e.g., 0.8s # Simulate mutation: touch source file to trigger dependent recompilation touch src/Contract.sol # Test again (includes recompilation) time forge test # e.g., 5.2s # If drastically different, set manual timeout in mewt.toml ``` If recompilation time >> test time: ```toml [test] cmd = "forge test" timeout = 11 # Based on: 5.2s × 2 = 10.4s, round up ``` Otherwise, omit `timeout` and let mewt auto-calculate. **Exit:** Test command verified working (if modified), timeout appropriately set (auto or manual). --- ### Phase 5: Final Validation **Entry:** Phase 4 exit criteria met (test command works if modified, timeout set). **Actions:** Run through the validation checklist to verify all prior phases completed successfully: - [ ] `mewt print config` — Configuration syntax valid, no errors - [ ] `mewt status` — Mutant count matches expected count (Phase 2 count if no optimization applied; lower post-optimization count if `[targets]` or `[run].mutations` was narrowed) - [ ] `mewt print targets` — Only intended files mutated (no tests, mocks, dependencies) - [ ] Test command verified — Already validated in Phase 2 (and Phase 4 if modified) - [ ] Timeout set — Auto-calculated or manually set for recompilation-heavy languages - [ ] Scope acceptable — Duration estimate from Phase 2 acceptable to user **Exit:** Ready to run `mewt run`. --- ## Configuration Reference ### File Structure ```toml db = "mewt.sqlite" [log] level = "info" # trace, debug, info, warn, error [targets] # BE SPECIFIC: Source code only, never tests/dependencies include = ["src/**/*.js", "lib/**/*.js"] ignore = ["test", "mock"] # substring matches, not globs [run] # Optional: Restrict mutation types (omit to test all) # mutations = ["ER", "CR", "IF", "IT"] [test] cmd = "npm test" # timeout = 30 # Optional: auto-calculated if omitted (2× baseline) # Per-target rules (first match wins) [[per_target]] glob = "src/core/*.js" test.cmd = "npm test -- core" test.timeout = 20 ``` Use top-level `[[per_target]]` rules with `test.cmd` and `test.timeout`. Run `mewt print config` after editing to catch invalid TOML and confirm the effective settings. ### Target Configuration Examples **Important:** Restrictive `include` patterns exclude most unwanted files. Only add `ignore` patterns for items within included paths. ```toml # Rust project [targets] include = ["src/**/*.rs"] ignore = ["test", "mock", "generated"] # Solidity project [targets] include = ["contracts/**/*.sol"] ignore = ["test", "interfaces", "mocks"] # Go project [targets] include = ["**/*.go"] ignore = ["test", "mock", "generated"] # JavaScript/TypeScript [targets] include = ["src/**/*.ts", "lib/**/*.ts"] ignore = ["test", "spec", "mock"] ``` ### Test Configuration **Timeout Calculation:** Mutants trigger incremental recompilation (only mutated file + dependents). Mewt's auto-calculated timeout (2× baseline) usually accounts for this. **Edge case:** In some compiled languages (Solidity/Foundry), recompiling dependent files takes much longer than running tests. Verify by timing tests, touching a file, and timing again. If drastically different, set manual timeout based on the slower measurement. ```toml # Option 1: Auto-calculate (recommended for most languages) [test] cmd = "cargo test" # Omit timeout — mewt measures baseline and applies 2× multiplier # Option 2: Explicit timeout (for recompilation-heavy languages) [test] cmd = "forge test" timeout = 11 # Based on: touch file, time test (5.2s), × 2 ``` --- ## Troubleshooting ### No Mutants Generated **Check language support:** ```bash mewt print mutations --language rust ``` **Verify patterns:** ```bash mewt print config find src -name '*.rs' | head # Do source files exist where include points? ``` `find`, not `ls src/**/*.rs`. Whether `**` recurses depends on the shell: zsh expands it, bash does not unless `globstar` is set, and the bash 3.2 that macOS ships has no `globstar` option to set. So the `ls` form degrades to `src/*/*.rs` under bash — given `src/top.rs`, `src/a/one.rs`, and `src/a/b/two.rs` it lists only `src/a/one.rs` — while the same line is correct under zsh. A diagnostic that under-reports on some machines and not others is worse than none: the files it drops read as "the include pattern doesn't match," sending you to edit a pattern that was already right. `find` behaves identically in every shell and exits 0 when nothing matches, instead of erroring. Keep the `**` in `mewt.toml`, where mewt expands it rather than the shell. **Common causes:** - Include pattern doesn't match files - Ignore pattern too broad (e.g., `"test"` matches `test_utils.rs`) - Unsupported language --- ### Test Command Fails **Run command manually:** ```bash pytest # Should work from project directory without errors ``` **Find correct command:** - Check: `Makefile`, `justfile`, `package.json`, `README.md` - In monorepos, may need to run from workspace subdirectory --- ### Configuration Validation Before running `mewt run`, complete Phase 5's validation checklist above. If any item fails, return to the relevant phase to fix it. --- ## Campaign Execution Timing Recommend timing based on estimated duration: - **< 1 hour:** Run anytime - **1-16 hours:** Start end-of-day, results by morning - **16-48 hours:** Start Friday evening, results Monday - **Two-phase:** Phase 1 overnight, Phase 2 next day --- ## Configuration Principles - **Configure via `mewt.toml`** — Not CLI flags (version control the config) - **Target source code specifically** — Exclude tests, dependencies, generated code - **Prefer limiting files over mutation types** — Better to assess critical code thoroughly - **Verify test commands** — Run manually before campaign - **Trust auto-calculated timeouts** — 2× baseline accounts for incremental recompilation in most cases - **Measure before optimizing** — Profile actual test times before applying per-target config - **Document decisions** — Commit `mewt.toml` with comments explaining configuration choices
-
-
SKILL.md 6.1 KB
--- name: mutation-testing description: "Configures mewt or muton campaigns, analyzes surviving mutants, and investigates bugs exposed by testing gaps. Use when setting up mutation testing, reviewing campaign results, identifying equivalent mutants, or finding bugs from surviving mutations." allowed-tools: Read Write Bash Grep --- # Mutation Testing (mewt/muton) Routes to the right mutation testing workflow and loads the references that workflow needs. > **Note**: muton and mewt share identical interfaces. Examples use `mewt`; substitute `muton` and its file names (`muton.toml`, `muton.sqlite`) for muton projects. `mewt --help` and `mewt <subcommand> --help` are the source of truth for command-line behavior. Examples below reflect the mewt 4.x API; run `--help` when a flag looks unfamiliar or a command fails. ## When to Use Use this skill when the user: - Mentions "mewt", "muton", or "mutation testing" - Wants to configure, scope, or speed up a mutation testing campaign - Wants to analyze mutation results — surviving/uncaught mutants, equivalent mutants, kill rate - Wants to use mutation results to find bugs in the source code ## When NOT to Use Do not use this skill when the user asks about tests or line coverage without any mutation testing context. --- ## Routing Pick the workflow, then load it together with the references listed for it. Workflows and references do not load each other — that decision belongs here. **Setting up, scoping, or speeding up a campaign** → [workflows/configuration.md](workflows/configuration.md) → Also load [references/optimization-strategies.md](references/optimization-strategies.md) when the campaign estimate is long enough to need trimming, or the user asks to make it faster. **Campaign finished, hunting for bugs in untested code** → [workflows/bug-hunter.md](workflows/bug-hunter.md) **Turning results into a formal analysis report** → [workflows/analyzing-results.md](workflows/analyzing-results.md), plus: - [references/equivalent-mutants.md](references/equivalent-mutants.md) — equivalence catalog and verification procedure - [references/severity-classification.md](references/severity-classification.md) — severity tier criteria - [references/report-template.md](references/report-template.md) — report structure - [references/blockchain-patterns.md](references/blockchain-patterns.md) — **only** for Solidity, Move, FunC/Tolk, Cairo, or Solana Rust targets - [references/input-formats.md](references/input-formats.md) — unless the results came from mewt or muton. Foreign tool output may not be self-describing; this covers the parsing anchors for slither-mutate, mull, and dextool-mutate **Anything else** → run `mewt --help` or `mewt <subcommand> --help`, then assist directly. --- ## Essential Commands ```bash # Set up and run mewt init # Create config and database mewt mutate [paths] # Generate mutants without testing them mewt run [paths] # Generate mutants and run the campaign # Read results mewt status # Overview with per-file breakdown mewt results # Uncaught mutants (default view) mewt results --all # Every outcome, not just uncaught mewt results --format json # json | sarif | ids | table # Narrow down (these filters work on both `results` and `print mutants`) mewt results --target 'src/auth/**' # Quote globs so the shell does not expand them mewt results --severity high,medium mewt results --mutation-types ER,CR mewt results --status Uncaught # Uncaught | TestFail | Skipped | Timeout mewt results --line 42 # Investigate and re-test mewt print mutant --id [id] # View the mutated code mewt test --ids [ids] # Re-test specific mutants mewt test --ids-file uncaught_ids.txt # Re-test IDs from a file, or '-' for stdin # Inspect configuration mewt print config # Effective config mewt print targets # Files actually mutated mewt print mutations --language [lang] # Mutations and severities for a language ``` Language labels are canonical `family` or `family/dialect` values in mewt 4.x — for example `rust`, `javascript/ts`, `move/sui`, `move/iota`. --- ## What Results Mean - **Caught/TestFail**: tests detected the mutation (good) - **Uncaught**: tests did not detect the change. Inspect the code to distinguish a testing gap from an equivalent mutation. - **Timeout**: tests took too long — inconclusive, not evidence of coverage - **Skipped**: a less severe mutant was skipped because a more severe mutant on the same line was uncaught --- ## Interpreting Mutation Types `mewt print mutations --language [lang]` lists every mutation slug, description, and severity for a language, and is authoritative — the operator set grows with each release. What that output does not tell you is what a survivor *means*, which is where prioritization comes from: | Severity | Representative slugs | What an uncaught mutant tells you | |----------|---------------------|-----------------------------------| | High | `ER` (Error Replacement) | Tests tolerate the injected error. Investigate whether the path executes, whether error handling masks the change, and whether assertions check the outcome. | | Medium | `CR` (Comment Replacement) | Removing the statement does not fail the tests. Check whether its effects matter and whether assertions observe them. | | Medium | `IF`/`IT` (If False/True), `NR` (Negation Removal) | Tests do not distinguish the changed condition. Both constant replacements surviving can indicate an unexecuted condition or weak assertions on the branch outcomes. | | Low | Operator shuffles (`AOS`, `COS`, `LOS`, `BOS`, shift/assignment variants), `BL`, `AS`, `LC`, `WF` | Check boundary inputs, arithmetic assertions, and semantic equivalence. The mutation result alone does not establish whether the code executed. | Severity ranks the *mutation*, not the risk. A low-severity survivor in a fee calculation matters more than a high-severity survivor in a log line — weigh what the mutated code does. Filter with `--severity` to work through the results in priority order.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.