find-untested-sources
MANDATORY for static requests to find, identify, or list untested source files or modules, sources without tests, source-to-test pairing, test-gap worklists, or suggested test locations. Invoke even for a tiny package; do not substitute manual globbing. Uses Roslyn for C#/.NET an
Install
npx skills add https://github.com/dotnet/skills/tree/main/plugins/dotnet-test/skills/find-untested-sources
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dotnet-skills@llmmart
git clone https://github.com/dotnet/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dotnet/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Find Untested Sources
Purpose
Coverage tools answer "which lines were executed?" — they require a green build and a passing test run, which is minutes-to-tens-of-minutes on a real repo. The question this skill answers is different and much cheaper:
Which source files have no test file referencing any of their declared types/symbols?
That's the question an agent asks before writing a new test — and it can be answered statically in a few seconds by parsing source files, with no build, no dependency resolution, and no compilation. The output is a deterministic test-pairing map that lets the agent pick the next file to test without reading the entire codebase first.
Two engines — pick one
This skill ships two interchangeable analyzers with a compatible JSON contract:
| Engine | Script | Use when |
|---|---|---|
| Roslyn (C#) | scripts/Find-UntestedSources.cs |
The repo is .NET-only. Parses every .cs file with the Roslyn syntax API and does strict namespace disambiguation, so it is materially more accurate on duplicated short names like Settings or Context. |
| tree-sitter (polyglot) | scripts/find_untested_sources.py |
The repo is not exclusively C#, or you want one tool across C#, Python, TypeScript/JavaScript, Go, Java, Rust, Ruby, Kotlin, Swift, PowerShell, and C++. |
For a .NET-only repository, prefer the Roslyn engine — its namespace-aware pairing beats the polyglot engine's identifier overlap.
Required workflow
- Use the narrowest repository or package root named by the caller. Do not scan a parent workspace when the request identifies a subdirectory.
- Execute the appropriate analyzer once. Do not replace analyzer execution with
manual globbing, filename matching, or visual inspection.
For polyglot analysis, pass
--include-testedwhen the answer must distinguish paired sources from unpaired sources. "Static pairing only" prohibits compiling the target repository and running its tests; it does not prohibit launching this skill's parse-only analyzer. State that distinction briefly when the caller also says "do not build." Treat analyzer dependencies as environment prerequisites: do not install packages, try the wrong engine, build the repository, or fall back to a manual scan when an analyzer invocation fails. Report the prerequisite failure instead. - Base the result on the analyzer's JSON. Preserve its paired/unpaired classification and suggested relative path; do not guess a different path.
- When the caller named a subdirectory, prefix analyzer-relative paths with that subdirectory so reported paths are workspace-relative.
- Report the requested result plus the static-pairing coverage caveat. Do not append build, package-install, test-run, or coverage commands. When paired sources exist, name their covering test files so the unpaired classification is auditable.
When to Use
- User asks "where should I add tests based on source pairing?", "which files have no tests?", "find unpaired source files", or "give me a static test gap list".
- Before invoking a test-generation agent, to produce a source-pairing worklist.
- After generating tests, to verify each new test file pairs to a source file.
- To enumerate "weakly paired" source files (only one referring test) for follow-up depth checks.
When Not to Use
- Line/branch coverage — use
coverage-analysis. - Priorities derived from real coverage data — use
coverage-analysis. - CRAP-score / risk hotspots — use
coverage-analysis. - Are existing tests strong? — use
test-gap-analysis(mutation reasoning) orassertion-quality.
Roslyn engine (C#)
Prerequisites
- .NET SDK that supports file-based apps (
dotnet run script.cs). Pinned in the repo'sglobal.json(SDK 11 preview or later). - No internet access required beyond the initial NuGet restore of
Microsoft.CodeAnalysis.CSharpon first run.
Usage
# From the skill folder
dotnet run scripts/Find-UntestedSources.cs -- <repo-root> [--top N]
# Save the report
dotnet run scripts/Find-UntestedSources.cs -- <repo-root> > pairing.json
# Iterate the untested list, highest-API-surface first
$report = Get-Content pairing.json | ConvertFrom-Json
$report.untested | Select-Object -First 10 source, decl_count, suggested_test_path
Diagnostics go to stderr; JSON goes to stdout.
Output schema
{
"repo": "<absolute path>",
"elapsed_ms": 8883,
"counts": {
"source_files": 3036,
"test_files": 867,
"untested_files": 1852,
"paired_files": 1184
},
"untested": [
{
"source": "src/Foo/Bar.cs",
"decl_count": 8, // # of type declarations in the file
"suggested_test_path": // mirror of source under a discovered test project
"tests/Foo.Tests/Bar/BarTests.cs"
}
],
"source_to_tests": {
"src/Foo/Baz.cs": [
"tests/Foo.Tests/BazTests.cs",
"tests/Foo.IntegrationTests/Scenarios/BazScenarios.cs"
]
}
}
How it works
- File discovery — recursive walk pruning
bin/,obj/,node_modules/,.git/,.vs/,packages/, and any dotted subdir. Skips generated files (.g.cs,.Designer.cs,.AssemblyInfo.cs). - Test vs source classification — walks up to the nearest
.csprojand marks it a test project if the project name ends in.Tests,.Test,.UnitTests,.IntegrationTests,.E2E,.EndToEnd,.Spec,.Specs, or the content referencesMicrosoft.NET.Test.Sdk,MSTest.Sdk,Microsoft.Testing.Platform,xunit,NUnit,TUnit, or<IsTestProject>true</IsTestProject>. - Source index (parallel) — parse each source file with
CSharpSyntaxTree.ParseText(syntax only, no compilation); record everyBaseTypeDeclarationSyntax/DelegateDeclarationSyntaxas(ShortName, EnclosingNamespace, FilePath). - Test scan (parallel) — parse each test file, collect
usingdirectives + enclosing namespace, walk everyIdentifierToken, look it up in the short-name index, and disambiguate strictly: an identifier is attributed only if the declaration's namespace matches one of the test file'susingdirectives, the enclosing namespace, or a prefix of them. This avoids noise where common names likeSettingsorContextmatch every project. - Pairing & suggestion — invert into
source → [tests]. Build a production-to-test project map from<ProjectReference>entries; for each untested source, mirror its in-project relative path under the referencing test project to suggest a path. - JSON emit — ordered by declaration count desc, then alphabetical.
Polyglot engine (tree-sitter)
Prerequisites
- Python 3.10+.
pip install tree-sitter-language-pack(single self-contained wheel that bundles parsers for 300+ languages and the high-levelprocess()API). No native build, no per-language grammar install.
Usage
# From the skill folder
python scripts/find_untested_sources.py <repo-root>
# Restrict to a language (repeatable)
python scripts/find_untested_sources.py <repo-root> --lang python --lang typescript
# Truncate the report (top 20 by declared API surface)
python scripts/find_untested_sources.py <repo-root> --limit-untested 20 > pairing.json
# Iterate, highest-API-surface first
$report = Get-Content pairing.json | ConvertFrom-Json
$report.untested_sources | Select-Object -First 10 path, declaration_count, suggested_test_path
Pass --include-tested to additionally emit tested_sources (omitted by
default to keep the payload small for LLM consumption). Diagnostics go to
stderr; JSON goes to stdout.
Output schema
{
"repo_root": "<absolute path>",
"summary": {
"source_files": 3138,
"test_files": 761,
"tested_source_files": 1419,
"untested_source_files": 1719,
"orphan_test_files": 15,
"languages": ["csharp"]
},
"untested_sources": [
{
"path": "src/Foo/Bar.cs",
"language": "csharp",
"declaration_count": 8,
"declarations": ["Bar", "BarOptions", "IBar", "..."],
"suggested_test_path": "src/Foo/BarTests.cs"
}
],
"orphan_tests": [
{ "path": "tests/SomeIntegrationTest.cs", "language": "csharp" }
]
}
How it works
File discovery — recursive walk pruning common build/vendor dirs (
bin,obj,node_modules,target,dist,build,vendor,__pycache__,.venv,.git, …) and generated files (.d.ts,.g.cs,.Designer.cs,_pb2.py,*.min.js,AssemblyInfo.cs, …).Language detection —
detect_language_from_pathmaps the extension to a supported language; unknown extensions are skipped.Test-vs-source classification — per-language path heuristics:
Language Test rule Python path contains tests//test/; or filename starts withtest_or ends_test.py; orconftest.py.JS/TS/TSX path contains __tests__,tests,test,spec,e2e; or filename contains.test./.spec..Go filename ends _test.go.Java path contains test/tests; or filename endsTest.java/Tests.java.Rust path contains tests//benches/.C# path contains tests/; or project segment ends.Tests/.Test/.UnitTests/.IntegrationTests; or filename endsTests/Test.Ruby path contains spec//test/; or filename ends_spec.rb/_test.rb.Kotlin path contains test//tests//spec/; or filename endsTest.kt/Tests.kt/Spec.kt.Swift path contains test//tests//uitests//integrationtests/(case-insensitive); or filename endsTest.swift/Tests.swift.PowerShell path contains test//tests//pester/; or filename ends.Tests.ps1/.Test.ps1.C++ path contains test//tests//testing/; or filename startstest_or ends_test.cpp/_tests.cpp.Per-file extraction —
process(text, ProcessConfig(structure, imports, symbols))returns declared items, raw import statements, and a flat declared -name list.Pairing — for each test file, union import resolution (per language, e.g. Python
from pkg.mod import x→pkg/mod.py; Javaimport a.b.C;→a/b/C.java; C#usingis namespace-not-file, so a no-op) with identifier overlap (word-like tokens, length ≥ 4, matched against declared names).JSON emit —
untested_sourcesordered by declaration count descending.
Limitations (be honest with the agent)
Both engines are static, parse-only heuristics that trade a little accuracy for orders-of-magnitude lower cost than coverage. Known gaps:
- Reflection / DI-resolved types referenced only via a string name or container resolution won't be detected — the type's short name never appears in the test source.
- Extension methods invoked as instance methods (C#): the declaring static class is not named, so its file is not credited.
var, target-typednew(), pattern matching lose the type token; the file-level union usually still catches it through other references.- Short identifier names (polyglot, < 4 chars) are dropped to avoid noisy
pairings on names like
id,db,Tag. - Monorepo path aliases (TS path mapping, Java module-info) are not resolved; a suffix-match fallback may pick the wrong source if two files share a trailing path segment.
For these cases, run actual coverage (coverage-analysis) on the unpaired
candidates the agent has already triaged.
Always label the final result as a static pairing heuristic, not evidence of line or branch coverage. Include that caveat even when every requested source file has an obvious matching or missing test.
Outputs the agent should consume
untested[*].source/untested_sources[*].path— pick the next source file to test (highest declaration count first).*.suggested_test_path— drop-in target for the new test file; the Roslyn engine honors the test project that already<ProjectReference>s the source's project, sodotnet sln addis not needed. The polyglot engine may suggest a co-located test when no test root is discoverable. When a source sibling is already paired, its test directory is the established convention and must be reused for the missing sibling rather than falling back to source co-location.source_to_tests(Roslyn) /--include-testedtested_sources(polyglot) — verify a newly written test file lands in the list for the intended source.orphan_tests(polyglot) — tests that don't reference any same-language source file; useful for triaging stale or integration-only tests.
Files (skills)
-
scripts
-
Find-UntestedSources.cs 15.9 KB · in bundle
-
find_untested_sources.py 26.7 KB
"""Polyglot source-to-test pairing analyzer using tree-sitter. Given a repo root, identifies which source files are NOT covered by any test file via two complementary heuristics: 1. Identifier overlap: a test file declares (or references) the same name a source file declares as a top-level symbol. 2. Import resolution: a test file imports a module/path that resolves to a source file. Supports any language tree-sitter-language-pack can parse and classify (Python, TypeScript, JavaScript, Go, Java, Rust, C#, Ruby, Kotlin, Swift, PowerShell, C++, ...). Output: JSON to stdout matching the schema used by the C# `find-untested-sources` skill, so the same prompt patterns can consume both tools. Dependencies: pip install tree-sitter-language-pack """ from __future__ import annotations import argparse import json import re import sys from dataclasses import dataclass, field from pathlib import Path, PurePosixPath from typing import Iterable try: from tree_sitter_language_pack import ( ProcessConfig, detect_language_from_path, process, ) except ImportError as e: print( "ERROR: tree-sitter-language-pack is not installed.\n" "Run: pip install tree-sitter-language-pack", file=sys.stderr, ) raise SystemExit(2) from e # Languages we'll process. Anything not in this set is skipped even if the # pack can parse it, because we have no test-detection heuristic for it. SUPPORTED_LANGUAGES = { "python", "typescript", "tsx", "javascript", "go", "java", "rust", "csharp", "ruby", "kotlin", "swift", "powershell", "cpp", } # Directories we never descend into. Lowercase match on segment name. PRUNE_DIRS = { ".git", ".hg", ".svn", ".vs", ".vscode", ".idea", "node_modules", "bower_components", "vendor", "third_party", "dist", "build", "out", "target", # rust + java "bin", "obj", "packages", "__pycache__", ".pytest_cache", ".mypy_cache", ".tox", ".venv", "venv", "env", ".nuget", "TestResults", "coverage", ".next", ".nuxt", ".cache", ".gradle", ".terraform", "site-packages", } # Filename patterns to skip (generated, minified, declaration files). SKIP_FILENAME_PATTERNS = ( re.compile(r"\.min\.(js|css)$", re.IGNORECASE), re.compile(r"\.d\.ts$", re.IGNORECASE), re.compile(r"\.designer\.cs$", re.IGNORECASE), re.compile(r"\.g\.cs$", re.IGNORECASE), re.compile(r"\.g\.i\.cs$", re.IGNORECASE), re.compile(r"\.generated\.cs$", re.IGNORECASE), re.compile(r"AssemblyInfo\.cs$"), re.compile(r"AssemblyAttributes\.cs$"), re.compile(r"GlobalUsings?\.cs$"), re.compile(r"_pb2\.py$"), re.compile(r"_pb\.go$"), re.compile(r"\.pb\.go$"), ) def is_test_path(rel: PurePosixPath, lang: str) -> bool: """Best-effort per-language test classification using path/filename.""" parts = [p.lower() for p in rel.parts] name = rel.name.lower() stem = rel.stem.lower() if lang == "python": if any(p in ("tests", "test") for p in parts): return True if name.startswith("test_") or stem.endswith("_test"): return True if "conftest.py" in name: return True return False if lang in ("typescript", "tsx", "javascript"): if any(p in ("__tests__", "tests", "test", "spec", "e2e") for p in parts): return True if any(s in stem for s in (".test", ".spec")): return True return False if lang == "go": return stem.endswith("_test") if lang == "java": if "test" in parts or "tests" in parts: return True return name.endswith("test.java") or name.endswith("tests.java") if lang == "rust": if any(p in ("tests", "benches") for p in parts): return True return False if lang == "csharp": if any(p in ("tests", "test") for p in parts): return True for p in parts: if p.endswith(".tests") or p.endswith(".test") or p.endswith(".unittests") or p.endswith(".integrationtests"): return True # Tokenize the original (un-lowered) stem on PascalCase boundaries # and treat it as a test file when the final word is "Test" / "Tests". # This matches the .NET convention (`UserServiceTests`, `MyTest`) # without misclassifying non-test files whose lower-cased stem # coincidentally ends in "test"/"tests" (e.g. "Contest.cs", # "Latest.cs", "Manifest.cs"). words = re.findall(r"[A-Z][a-z]*|[a-z]+|[0-9]+", rel.stem) if words and words[-1] in ("Test", "Tests"): return True return False if lang == "ruby": if any(p in ("spec", "test", "tests") for p in parts): return True return stem.endswith("_spec") or stem.endswith("_test") if lang == "kotlin": if any(p in ("test", "tests", "spec", "specs") for p in parts): return True words = re.findall(r"[A-Z][a-z]*|[a-z]+|[0-9]+", rel.stem) return bool(words and words[-1] in ("Test", "Tests", "Spec", "Specs")) if lang == "swift": if any(p in ("test", "tests", "uitests", "integrationtests") for p in parts): return True words = re.findall(r"[A-Z][a-z]*|[a-z]+|[0-9]+", rel.stem) return bool(words and words[-1] in ("Test", "Tests")) if lang == "powershell": if any(p in ("test", "tests", "pester") for p in parts): return True return name.endswith(".tests.ps1") or name.endswith(".test.ps1") if lang == "cpp": if any(p in ("test", "tests", "testing") for p in parts): return True return ( stem.startswith("test_") or stem.endswith("_test") or stem.endswith("_tests") ) return False @dataclass class FileInfo: path: Path rel: PurePosixPath lang: str is_test: bool declarations: set[str] = field(default_factory=set) referenced_identifiers: set[str] = field(default_factory=set) imports: list[str] = field(default_factory=list) def __hash__(self) -> int: return id(self) def __eq__(self, other: object) -> bool: return self is other def should_skip_filename(name: str) -> bool: return any(p.search(name) for p in SKIP_FILENAME_PATTERNS) def walk_files(root: Path, lang_filter: set[str] | None = None) -> Iterable[Path]: """Yield files under root, pruning common build/vendor directories.""" stack = [root] while stack: current = stack.pop() try: children = list(current.iterdir()) except (PermissionError, OSError): continue for child in children: try: if child.is_symlink(): continue except OSError: continue if child.is_dir(): if child.name.lower() in PRUNE_DIRS: continue stack.append(child) continue if not child.is_file(): continue if should_skip_filename(child.name): continue lang = detect_language_from_path(str(child)) if lang is None: continue if lang not in SUPPORTED_LANGUAGES: continue if lang_filter is not None and lang not in lang_filter: continue yield child # Word-character regex used to harvest identifiers from a test file's source # for the "identifier overlap" pairing strategy. Tree-sitter's `symbols` output # only gives us declared names; references (`new Foo()`, `Foo.bar()`) won't # appear, so we fall back to a simple word scan over the file body. IDENTIFIER_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") def harvest_identifiers(text: str, lang: str) -> set[str]: if lang == "powershell": return set(re.findall(r"[A-Za-z_][A-Za-z0-9-]*", text)) return set(IDENTIFIER_RE.findall(text)) def harvest_declarations(text: str, lang: str) -> set[str]: """Recover common declarations when the language-pack symbol view is sparse.""" if lang == "powershell": return set(re.findall(r"(?im)^\s*function\s+([A-Za-z_][A-Za-z0-9-]*)", text)) if lang == "cpp": return set( re.findall( r"\b(?:class|struct|enum(?:\s+class)?)\s+([A-Za-z_][A-Za-z0-9_]*)", text, ) ) return set() def parse_file(path: Path, root: Path) -> FileInfo | None: rel_str = path.relative_to(root).as_posix() rel = PurePosixPath(rel_str) lang = detect_language_from_path(str(path)) if lang is None or lang not in SUPPORTED_LANGUAGES: return None try: text = path.read_text(encoding="utf-8", errors="replace") except OSError: return None is_test = is_test_path(rel, lang) info = FileInfo(path=path, rel=rel, lang=lang, is_test=is_test) try: cfg = ProcessConfig(language=lang, structure=True, imports=True, symbols=True) result = process(text, cfg) except Exception as exc: # Emit a diagnostic so users can see when tree-sitter parsing silently # degrades results (e.g. 0-declaration sources because the parser # blew up on an unusual construct). Keep it short — one line per file. print(f"WARN: tree-sitter parse failed for {rel_str} ({lang}): {exc}", file=sys.stderr) return info # Top-level declarations: union of structure + symbols. The two views are # complementary depending on the language — e.g. for Go, `structure` lists # functions/methods but not `type` declarations, while `symbols` lists the # types. We filter `module`/`namespace` kinds (they're packaging, not # declarations) to avoid false-positive pairings on package names. excluded_kinds = {"module", "namespace"} def _kind_str(item: object) -> str: k = getattr(item, "kind", None) return str(k).lower() if k is not None else "" for item in getattr(result, "structure", None) or []: if _kind_str(item) in excluded_kinds: continue name = getattr(item, "name", None) if name: info.declarations.add(name) for sym in getattr(result, "symbols", None) or []: if _kind_str(sym) in excluded_kinds: continue name = getattr(sym, "name", None) if name: info.declarations.add(name) info.declarations.update(harvest_declarations(text, lang)) # Imports: keep the raw `source` field; we'll normalize per language. if getattr(result, "imports", None): for imp in result.imports: src = getattr(imp, "source", None) if src: info.imports.append(src) # For test files only, scan all identifier-like tokens — caller uses these # to pair with source declarations by name. if is_test: info.referenced_identifiers = harvest_identifiers(text, lang) return info # --- Per-language import-to-path resolution -------------------------------- def _strip_quoted(value: str) -> str: """Return the first quoted substring or the original string trimmed.""" m = re.search(r"""['"`]([^'"`]+)['"`]""", value) if m: return m.group(1) return value.strip() def _resolve_relative_js(test_rel: PurePosixPath, target: str) -> set[PurePosixPath]: """Resolve ./ or ../ relative import paths to candidate source paths.""" if not (target.startswith("./") or target.startswith("../") or target.startswith("/")): return set() base = PurePosixPath(target) if target.startswith("/"): joined = PurePosixPath(target.lstrip("/")) else: joined_str = (test_rel.parent / base).as_posix() # Collapse any number of "<seg>/../" pairs, including chained ones # like "a/b/../../c" → "c". The previous regex only collapsed a single # occurrence so chained "../../foo" imports stayed partially # un-normalized and failed to match indexed paths. prev = None while prev != joined_str: prev = joined_str joined_str = re.sub(r"(?:^|/)[^/]+/\.\./", "/", joined_str) joined_str = joined_str.lstrip("/") joined = PurePosixPath(joined_str) # Try various extensions and /index suffix. candidates = set() stems = [str(joined)] if str(joined).endswith("/index"): stems.append(str(joined)[: -len("/index")]) for stem in stems: for ext in (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"): candidates.add(PurePosixPath(stem + ext)) candidates.add(PurePosixPath(stem + "/index.ts")) candidates.add(PurePosixPath(stem + "/index.tsx")) candidates.add(PurePosixPath(stem + "/index.js")) return candidates def _extract_python_module(import_text: str) -> str | None: """Extract `pkg.mod` from `from pkg.mod import x` or `import pkg.mod`.""" s = import_text.strip() m = re.match(r"from\s+([\.\w]+)\s+import", s) if m: return m.group(1) m = re.match(r"import\s+([\.\w]+)", s) if m: return m.group(1) return None def _python_module_to_path( module: str, test_rel: PurePosixPath | None = None ) -> set[PurePosixPath]: """Resolve a Python import target to candidate source paths. Leading dots (PEP 328 relative imports) are resolved against `test_rel`'s directory: one dot = same package as the test file, each extra dot walks one directory up. Without a `test_rel` context we cannot resolve them safely, so we return an empty set rather than risk pairing the test with an unrelated `utils.py` at the repo root. """ leading_dots = len(module) - len(module.lstrip(".")) rest = module[leading_dots:] if leading_dots > 0: if test_rel is None or not rest: # Either we have no context to resolve against, or this is a # `from . import x` form where the regex captured only the dots # and we don't know the imported name. Skip rather than guess. return set() base_dir = test_rel.parent for _ in range(leading_dots - 1): base_dir = base_dir.parent parts = rest.split(".") base_path = base_dir.joinpath(*parts) if base_dir.parts else PurePosixPath(*parts) return { PurePosixPath(str(base_path) + ".py"), PurePosixPath(str(base_path) + "/__init__.py"), } parts = rest.split(".") candidates: set[PurePosixPath] = set() base = "/".join(parts) candidates.add(PurePosixPath(base + ".py")) candidates.add(PurePosixPath(base + "/__init__.py")) return candidates def _extract_java_fqcn(import_text: str) -> str | None: m = re.match(r"import\s+(?:static\s+)?([\w\.]+)", import_text.strip()) if not m: return None fqcn = m.group(1).rstrip(";").rstrip(".*") return fqcn def _java_fqcn_to_path(fqcn: str) -> PurePosixPath: return PurePosixPath(fqcn.replace(".", "/") + ".java") def _extract_csharp_using(import_text: str) -> str | None: m = re.match(r"using\s+(?:static\s+)?([\w\.]+)", import_text.strip()) if not m: return None return m.group(1).rstrip(";") def _extract_rust_use(import_text: str) -> set[str]: """Extract crate-relative paths from `use foo::bar::Baz;`.""" s = import_text.strip().rstrip(";") m = re.match(r"use\s+(.+)", s) if not m: return set() body = m.group(1).strip() # Strip `as` aliases; ignore grouped imports for simplicity. body = body.split(" as ")[0].strip() return {body} def _extract_go_import_targets(import_text: str) -> set[str]: s = import_text.strip() targets: set[str] = set() for m in re.finditer(r'"([^"]+)"', s): targets.add(m.group(1)) return targets # --- Pairing engine -------------------------------------------------------- def _build_indexes(sources: list[FileInfo]) -> dict: """Build lookup indexes used to resolve test imports to source files.""" by_rel: dict[str, FileInfo] = {s.rel.as_posix().lower(): s for s in sources} by_decl: dict[str, list[FileInfo]] = {} for s in sources: for d in s.declarations: by_decl.setdefault(d, []).append(s) # For Java/C#: index by FQCN-like trailing path. by_path_suffix: dict[str, list[FileInfo]] = {} # For Go: index by basename so import resolution is O(1) per target # instead of O(#sources) per import target. by_filename: dict[str, list[FileInfo]] = {} for s in sources: p = s.rel.as_posix().lower() parts = p.split("/") for i in range(len(parts)): suffix = "/".join(parts[i:]) by_path_suffix.setdefault(suffix, []).append(s) by_filename.setdefault(s.rel.name.lower(), []).append(s) return { "by_rel": by_rel, "by_decl": by_decl, "by_path_suffix": by_path_suffix, "by_filename": by_filename, } def _resolve_test_imports(test: FileInfo, indexes: dict, lang: str) -> set[FileInfo]: """Given a test file's imports, return source files those imports resolve to.""" found: set[FileInfo] = set() by_rel = indexes["by_rel"] by_path_suffix = indexes["by_path_suffix"] def add_candidate(path: PurePosixPath) -> None: key = path.as_posix().lower() if key in by_rel: found.add(by_rel[key]) return # Try matching by suffix (covers monorepo / aliased layouts). matches = by_path_suffix.get(key, []) if len(matches) == 1: found.add(matches[0]) for raw in test.imports: if lang in ("typescript", "tsx", "javascript"): target = _strip_quoted(raw) for c in _resolve_relative_js(test.rel, target): add_candidate(c) elif lang == "python": module = _extract_python_module(raw) if module is None: continue for c in _python_module_to_path(module, test.rel): add_candidate(c) elif lang == "go": by_filename = indexes["by_filename"] for tgt in _extract_go_import_targets(raw): segs = tgt.split("/") if not segs: continue last = segs[-1] key = (last + ".go").lower() for info in by_filename.get(key, ()): if not info.is_test: found.add(info) elif lang == "java": fqcn = _extract_java_fqcn(raw) if fqcn: add_candidate(_java_fqcn_to_path(fqcn)) elif lang == "rust": for use_path in _extract_rust_use(raw): segs = use_path.split("::") if not segs: continue last = segs[-1] add_candidate(PurePosixPath(last + ".rs")) if len(segs) >= 2: add_candidate(PurePosixPath(segs[-2] + ".rs")) elif lang == "csharp": ns = _extract_csharp_using(raw) if ns: # using maps to namespace, not file; we fall back to identifier overlap. pass elif lang == "ruby": target = _strip_quoted(raw) if target: add_candidate(PurePosixPath(target + ".rb")) elif lang == "kotlin": target = _strip_quoted(raw) if target: add_candidate(PurePosixPath(target.replace(".", "/") + ".kt")) elif lang in ("swift", "powershell", "cpp"): # These ecosystems commonly import modules, dot-source scripts, or # include headers rather than source files. Identifier overlap is # the reliable cross-project fallback for this analyzer. pass return found def _resolve_test_by_identifiers( test: FileInfo, indexes: dict, ) -> set[FileInfo]: """Pair test with sources whose declarations appear in the test's token set.""" found: set[FileInfo] = set() by_decl = indexes["by_decl"] # Iterate the test's referenced identifiers (typically O(hundreds)) # rather than scanning every declaration in the index (O(#decls × #tests) # across all tests), so pairing stays linear in repo size on large # codebases. for ident in test.referenced_identifiers: if len(ident) < 4: continue sources = by_decl.get(ident) if not sources: continue for s in sources: if s.lang == test.lang: found.add(s) return found def build_pairings( sources: list[FileInfo], tests: list[FileInfo], ) -> tuple[dict[FileInfo, set[FileInfo]], list[FileInfo]]: """Return (source -> covering tests) and the list of orphan test files.""" by_lang_indexes: dict[str, dict] = {} for lang in SUPPORTED_LANGUAGES: lang_sources = [s for s in sources if s.lang == lang] if lang_sources: by_lang_indexes[lang] = _build_indexes(lang_sources) source_to_tests: dict[FileInfo, set[FileInfo]] = {s: set() for s in sources} orphans: list[FileInfo] = [] for t in tests: idx = by_lang_indexes.get(t.lang) if idx is None: orphans.append(t) continue matched = _resolve_test_imports(t, idx, t.lang) | _resolve_test_by_identifiers(t, idx) if not matched: orphans.append(t) continue for s in matched: source_to_tests[s].add(t) return source_to_tests, orphans # --- Output ---------------------------------------------------------------- def _test_filename(source: FileInfo) -> str: rel = source.rel lang = source.lang stem = rel.stem if lang == "python": return f"test_{stem}.py" if lang == "go": return f"{stem}_test.go" if lang in ("typescript", "tsx"): return f"{stem}.test.{rel.suffix.lstrip('.')}" if lang == "javascript": return f"{stem}.test.js" if lang == "java": return f"{stem}Test.java" if lang == "rust": return f"{stem}_test.rs" if lang == "csharp": return f"{stem}Tests.cs" if lang == "ruby": return f"{stem}_spec.rb" if lang == "kotlin": return f"{stem}Test.kt" if lang == "swift": return f"{stem}Tests.swift" if lang == "powershell": return f"{stem}.Tests.ps1" if lang == "cpp": return f"{stem}_test.cpp" return "" def _suggest_test_path( source: FileInfo, sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath], ) -> str: filename = _test_filename(source) if not filename: return "" language_family = "typescript" if source.lang in {"typescript", "tsx"} else source.lang parent = sibling_test_dirs.get((language_family, source.rel.parent), source.rel.parent) return (parent / filename).as_posix() def build_output( sources: list[FileInfo], tests: list[FileInfo], source_to_tests: dict[FileInfo, set[FileInfo]], orphans: list[FileInfo], repo_root: Path, ) -> dict: untested: list[dict] = [] tested: list[dict] = [] sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath] = {} for paired_source, covering_tests in source_to_tests.items(): language_family = "typescript" if paired_source.lang in {"typescript", "tsx"} else paired_source.lang key = (language_family, paired_source.rel.parent) for test in covering_tests: candidate = test.rel.parent current = sibling_test_dirs.get(key) if current is None or candidate.as_posix() < current.as_posix(): sibling_test_dirs[key] = candidate for s in sources: covering = sorted(source_to_tests.get(s, set()), key=lambda t: t.rel.as_posix()) entry = { "path": s.rel.as_posix(), "language": s.lang, "declaration_count": len(s.declarations), "declarations": sorted(s.declarations), } if covering: entry["covering_tests"] = [c.rel.as_posix() for c in covering] tested.append(entry) else: entry["suggested_test_path"] = _suggest_test_path(s, sibling_test_dirs) untested.append(entry) return { "repo_root": str(repo_root), "summary": { "source_files": len(sources), "test_files": len(tests), "tested_source_files": len(tested), "untested_source_files": len(untested), "orphan_test_files": len(orphans), "languages": sorted({s.lang for s in sources} | {t.lang for t in tests}), }, "untested_sources": sorted(untested, key=lambda e: (-e["declaration_count"], e["path"])), "tested_sources": sorted(tested, key=lambda e: e["path"]), "orphan_tests": [ {"path": t.rel.as_posix(), "language": t.lang} for t in sorted(orphans, key=lambda t: t.rel.as_posix()) ], } def main() -> int: parser = argparse.ArgumentParser( description="Find untested source files in a polyglot repository using tree-sitter." ) parser.add_argument("root", type=Path, help="Path to repository root.") parser.add_argument( "--lang", action="append", choices=sorted(SUPPORTED_LANGUAGES), help="Restrict analysis to specified language(s). Repeatable.", ) parser.add_argument( "--limit-untested", type=int, default=0, help="If > 0, truncate the untested_sources list to N entries.", ) parser.add_argument( "--include-tested", action="store_true", help="Include tested_sources in the output (omitted by default to keep payload small).", ) args = parser.parse_args() root = args.root.resolve() if not root.is_dir(): print(f"ERROR: not a directory: {root}", file=sys.stderr) return 1 lang_filter = set(args.lang) if args.lang else None print(f"Scanning {root}...", file=sys.stderr) sources: list[FileInfo] = [] tests: list[FileInfo] = [] parsed_count = 0 for path in walk_files(root, lang_filter=lang_filter): info = parse_file(path, root) if info is None: continue parsed_count += 1 if info.is_test: tests.append(info) else: sources.append(info) print(f"Parsed {parsed_count} files: {len(sources)} source, {len(tests)} test.", file=sys.stderr) source_to_tests, orphans = build_pairings(sources, tests) output = build_output(sources, tests, source_to_tests, orphans, root) if args.limit_untested and args.limit_untested > 0: output["untested_sources"] = output["untested_sources"][: args.limit_untested] if not args.include_tested: output.pop("tested_sources", None) json.dump(output, sys.stdout, indent=2) sys.stdout.write("\n") return 0 if __name__ == "__main__": sys.exit(main())
-
-
SKILL.md 13.1 KB
--- name: find-untested-sources description: > MANDATORY for static source-to-test pairing: find or list source files/modules without corresponding tests, or suggest test locations from repository structure. Invoke even for a tiny package; do not substitute manual globbing. Uses Roslyn for C#/.NET and tree-sitter for Python, TS/JS, Go, Java, Rust, Ruby, Kotlin, Swift, PowerShell, and C++. DO NOT USE FOR: real line/branch/Cobertura data, coverage-backed test priorities, CRAP risk, or grading existing tests. license: MIT --- # Find Untested Sources ## Purpose Coverage tools answer "which lines were executed?" — they require a green build and a passing test run, which is minutes-to-tens-of-minutes on a real repo. The question this skill answers is different and much cheaper: > _Which source files have no test file referencing any of their declared > types/symbols?_ That's the question an agent asks **before** writing a new test — and it can be answered statically in a few seconds by parsing source files, with **no build, no dependency resolution, and no compilation**. The output is a deterministic test-pairing map that lets the agent pick the next file to test without reading the entire codebase first. ## Two engines — pick one This skill ships two interchangeable analyzers with a compatible JSON contract: | Engine | Script | Use when | |--------|--------|----------| | **Roslyn (C#)** | `scripts/Find-UntestedSources.cs` | The repo is **.NET-only**. Parses every `.cs` file with the Roslyn syntax API and does strict **namespace disambiguation**, so it is materially more accurate on duplicated short names like `Settings` or `Context`. | | **tree-sitter (polyglot)** | `scripts/find_untested_sources.py` | The repo is **not exclusively C#**, or you want one tool across C#, Python, TypeScript/JavaScript, Go, Java, Rust, Ruby, Kotlin, Swift, PowerShell, and C++. | For a .NET-only repository, **prefer the Roslyn engine** — its namespace-aware pairing beats the polyglot engine's identifier overlap. ## Required workflow 1. Use the narrowest repository or package root named by the caller. Do not scan a parent workspace when the request identifies a subdirectory. 2. Execute the appropriate analyzer once. Do not replace analyzer execution with manual globbing, filename matching, or visual inspection. For polyglot analysis, pass `--include-tested` when the answer must distinguish paired sources from unpaired sources. "Static pairing only" prohibits compiling the target repository and running its tests; it does not prohibit launching this skill's parse-only analyzer. State that distinction briefly when the caller also says "do not build." Treat analyzer dependencies as environment prerequisites: do not install packages, try the wrong engine, build the repository, or fall back to a manual scan when an analyzer invocation fails. Report the prerequisite failure instead. 3. Base the result on the analyzer's JSON. Preserve its paired/unpaired classification and suggested relative path; do not guess a different path. 4. When the caller named a subdirectory, prefix analyzer-relative paths with that subdirectory so reported paths are workspace-relative. 5. Report the requested result plus the static-pairing coverage caveat. Do not append build, package-install, test-run, or coverage commands. When paired sources exist, name their covering test files so the unpaired classification is auditable. ## When to Use - User asks "where should I add tests based on source pairing?", "which files have no tests?", "find unpaired source files", or "give me a static test gap list". - Before invoking a test-generation agent, to produce a source-pairing worklist. - After generating tests, to verify each new test file pairs to a source file. - To enumerate "weakly paired" source files (only one referring test) for follow-up depth checks. ## When Not to Use - **Line/branch coverage** — use `coverage-analysis`. - **Priorities derived from real coverage data** — use `coverage-analysis`. - **CRAP-score / risk hotspots** — use `coverage-analysis`. - **Are existing tests strong?** — use `test-gap-analysis` (mutation reasoning) or `assertion-quality`. ## Roslyn engine (C#) ### Prerequisites - .NET SDK that supports file-based apps (`dotnet run script.cs`). Pinned in the repo's `global.json` (SDK 11 preview or later). - No internet access required beyond the initial NuGet restore of `Microsoft.CodeAnalysis.CSharp` on first run. ### Usage ```powershell # From the skill folder dotnet run scripts/Find-UntestedSources.cs -- <repo-root> [--top N] # Save the report dotnet run scripts/Find-UntestedSources.cs -- <repo-root> > pairing.json # Iterate the untested list, highest-API-surface first $report = Get-Content pairing.json | ConvertFrom-Json $report.untested | Select-Object -First 10 source, decl_count, suggested_test_path ``` Diagnostics go to stderr; JSON goes to stdout. ### Output schema ```jsonc { "repo": "<absolute path>", "elapsed_ms": 8883, "counts": { "source_files": 3036, "test_files": 867, "untested_files": 1852, "paired_files": 1184 }, "untested": [ { "source": "src/Foo/Bar.cs", "decl_count": 8, // # of type declarations in the file "suggested_test_path": // mirror of source under a discovered test project "tests/Foo.Tests/Bar/BarTests.cs" } ], "source_to_tests": { "src/Foo/Baz.cs": [ "tests/Foo.Tests/BazTests.cs", "tests/Foo.IntegrationTests/Scenarios/BazScenarios.cs" ] } } ``` ### How it works 1. **File discovery** — recursive walk pruning `bin/`, `obj/`, `node_modules/`, `.git/`, `.vs/`, `packages/`, and any dotted subdir. Skips generated files (`.g.cs`, `.Designer.cs`, `.AssemblyInfo.cs`). 2. **Test vs source classification** — walks up to the nearest `.csproj` and marks it a test project if the project name ends in `.Tests`, `.Test`, `.UnitTests`, `.IntegrationTests`, `.E2E`, `.EndToEnd`, `.Spec`, `.Specs`, or the content references `Microsoft.NET.Test.Sdk`, `MSTest.Sdk`, `Microsoft.Testing.Platform`, `xunit`, `NUnit`, `TUnit`, or `<IsTestProject>true</IsTestProject>`. 3. **Source index (parallel)** — parse each source file with `CSharpSyntaxTree.ParseText` (syntax only, no compilation); record every `BaseTypeDeclarationSyntax` / `DelegateDeclarationSyntax` as `(ShortName, EnclosingNamespace, FilePath)`. 4. **Test scan (parallel)** — parse each test file, collect `using` directives + enclosing namespace, walk every `IdentifierToken`, look it up in the short-name index, and **disambiguate strictly**: an identifier is attributed only if the declaration's namespace matches one of the test file's `using` directives, the enclosing namespace, or a prefix of them. This avoids noise where common names like `Settings` or `Context` match every project. 5. **Pairing & suggestion** — invert into `source → [tests]`. Build a production-to-test project map from `<ProjectReference>` entries; for each untested source, mirror its in-project relative path under the referencing test project to suggest a path. 6. **JSON emit** — ordered by declaration count desc, then alphabetical. ## Polyglot engine (tree-sitter) ### Prerequisites - Python 3.10+. - `pip install tree-sitter-language-pack` (single self-contained wheel that bundles parsers for 300+ languages and the high-level `process()` API). No native build, no per-language grammar install. ### Usage ```powershell # From the skill folder python scripts/find_untested_sources.py <repo-root> # Restrict to a language (repeatable) python scripts/find_untested_sources.py <repo-root> --lang python --lang typescript # Truncate the report (top 20 by declared API surface) python scripts/find_untested_sources.py <repo-root> --limit-untested 20 > pairing.json # Iterate, highest-API-surface first $report = Get-Content pairing.json | ConvertFrom-Json $report.untested_sources | Select-Object -First 10 path, declaration_count, suggested_test_path ``` Pass `--include-tested` to additionally emit `tested_sources` (omitted by default to keep the payload small for LLM consumption). Diagnostics go to stderr; JSON goes to stdout. ### Output schema ```jsonc { "repo_root": "<absolute path>", "summary": { "source_files": 3138, "test_files": 761, "tested_source_files": 1419, "untested_source_files": 1719, "orphan_test_files": 15, "languages": ["csharp"] }, "untested_sources": [ { "path": "src/Foo/Bar.cs", "language": "csharp", "declaration_count": 8, "declarations": ["Bar", "BarOptions", "IBar", "..."], "suggested_test_path": "src/Foo/BarTests.cs" } ], "orphan_tests": [ { "path": "tests/SomeIntegrationTest.cs", "language": "csharp" } ] } ``` ### How it works 1. **File discovery** — recursive walk pruning common build/vendor dirs (`bin`, `obj`, `node_modules`, `target`, `dist`, `build`, `vendor`, `__pycache__`, `.venv`, `.git`, …) and generated files (`.d.ts`, `.g.cs`, `.Designer.cs`, `_pb2.py`, `*.min.js`, `AssemblyInfo.cs`, …). 2. **Language detection** — `detect_language_from_path` maps the extension to a supported language; unknown extensions are skipped. 3. **Test-vs-source classification** — per-language path heuristics: | Language | Test rule | |---|---| | Python | path contains `tests/`/`test/`; or filename starts with `test_` or ends `_test.py`; or `conftest.py`. | | JS/TS/TSX | path contains `__tests__`, `tests`, `test`, `spec`, `e2e`; or filename contains `.test.`/`.spec.`. | | Go | filename ends `_test.go`. | | Java | path contains `test`/`tests`; or filename ends `Test.java`/`Tests.java`. | | Rust | path contains `tests/`/`benches/`. | | C# | path contains `tests/`; or project segment ends `.Tests`/`.Test`/`.UnitTests`/`.IntegrationTests`; or filename ends `Tests`/`Test`. | | Ruby | path contains `spec/`/`test/`; or filename ends `_spec.rb`/`_test.rb`. | | Kotlin | path contains `test/`/`tests/`/`spec/`; or filename ends `Test.kt`/`Tests.kt`/`Spec.kt`. | | Swift | path contains `test/`/`tests/`/`uitests/`/`integrationtests/` (case-insensitive); or filename ends `Test.swift`/`Tests.swift`. | | PowerShell | path contains `test/`/`tests/`/`pester/`; or filename ends `.Tests.ps1`/`.Test.ps1`. | | C++ | path contains `test/`/`tests/`/`testing/`; or filename starts `test_` or ends `_test.cpp`/`_tests.cpp`. | 4. **Per-file extraction** — `process(text, ProcessConfig(structure, imports, symbols))` returns declared items, raw import statements, and a flat declared -name list. 5. **Pairing** — for each test file, union **import resolution** (per language, e.g. Python `from pkg.mod import x` → `pkg/mod.py`; Java `import a.b.C;` → `a/b/C.java`; C# `using` is namespace-not-file, so a no-op) with **identifier overlap** (word-like tokens, length ≥ 4, matched against declared names). 6. **JSON emit** — `untested_sources` ordered by declaration count descending. ## Limitations (be honest with the agent) Both engines are static, parse-only heuristics that trade a little accuracy for orders-of-magnitude lower cost than coverage. Known gaps: - **Reflection / DI-resolved types** referenced only via a string name or container resolution won't be detected — the type's short name never appears in the test source. - **Extension methods** invoked as instance methods (C#): the declaring static class is not named, so its file is not credited. - **`var`, target-typed `new()`, pattern matching** lose the type token; the file-level union usually still catches it through other references. - **Short identifier names** (polyglot, < 4 chars) are dropped to avoid noisy pairings on names like `id`, `db`, `Tag`. - **Monorepo path aliases** (TS path mapping, Java module-info) are not resolved; a suffix-match fallback may pick the wrong source if two files share a trailing path segment. For these cases, run actual coverage (`coverage-analysis`) on the unpaired candidates the agent has already triaged. Always label the final result as a static pairing heuristic, not evidence of line or branch coverage. Include that caveat even when every requested source file has an obvious matching or missing test. ## Outputs the agent should consume - `untested[*].source` / `untested_sources[*].path` — pick the next source file to test (highest declaration count first). - `*.suggested_test_path` — drop-in target for the new test file; the Roslyn engine honors the test project that already `<ProjectReference>`s the source's project, so `dotnet sln add` is not needed. The polyglot engine may suggest a co-located test when no test root is discoverable. When a source sibling is already paired, its test directory is the established convention and must be reused for the missing sibling rather than falling back to source co-location. - `source_to_tests` (Roslyn) / `--include-tested` `tested_sources` (polyglot) — verify a newly written test file lands in the list for the intended source. - `orphan_tests` (polyglot) — tests that don't reference any same-language source file; useful for triaging stale or integration-only tests.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.