Claude Skill

searching-codebases

Binding-resolved Python symbol queries via pyright — every true caller (--refs), the real definition (--def), or an inferred signature (--hover) of a .py symbol, excluding the same-named false positives text search cannot tell apart. Use when a task needs ALL callers or users of

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download oaustegard-claude-skills-plugins_code-intelligence_skills_searching-codebases-e39c726.zip · 36 KB

Install

skills CLI npx skills add https://github.com/oaustegard/claude-skills/tree/main/plugins/code-intelligence/skills/searching-codebases
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oaustegard-claude-skills@llmmart
Git git clone https://github.com/oaustegard/claude-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oaustegard/claude-skills collection as a plugin from our marketplace. Git is the plain clone.

README

searching-codebases

Find code in any codebase by regex pattern or natural language concept. Auto-routes between n-gram indexed regex search (2-20x faster than ripgrep) and TF-IDF semantic search. Expands results to full functions via tree-sitting AST data. For Python sources, a binding-resolved reference/definition tier (pyright, via python-lsp) finds real callers and definitions without same-name false positives.

Features

  • Dual search modes — regex (pattern/identifier) and semantic (natural language concepts), auto-routed per query
  • N-gram indexed regex — sparse inverted index narrows candidate files by 90-99% before ripgrep verification
  • TF-IDF semantic search — cosine similarity ranking over code chunks (functions, classes)
  • Binding-resolved Python tier — --refs/--def/--hover via pyright: true find-all-callers and go-to-definition that exclude same-named, unrelated symbols and follow imports. Engaged lazily; degrades to the regex text path when pyright/node is unavailable
  • AST context expansion — --expand returns complete function/class bodies instead of line fragments
  • Flexible sources — accepts GitHub URLs, local directories, uploaded files/archives, or project knowledge
  • Mixed queries — multiple queries with different modes in a single invocation; indexes built once per mode

Dependencies

  • ripgrep — required for regex verification
  • tree-sitting — auto-installs the bare tree-sitter package when needed: for --expand context and for the symbol→position resolution that seeds the binding-resolved tier (grammars ship bundled). Regex and semantic search work without it
  • scikit-learn — required for semantic mode (auto-installs)
  • python-lsp — provides the binding-resolved tier (--refs/--def/--hover); self-bootstraps pyright on first use and needs system node (v18+). Without it those flags degrade to the regex text path

Skill manifest

Searching Codebases

Find code in any codebase by pattern or concept. One entry point, two search strategies, automatic routing.

When NOT to use this skill

Python callers and definitions only. Every other code question is cheaper elsewhere, and the measurement below is why this skill is edge-case-only.

Situation Use
What symbols does this file contain? tree-sitting
Where is X defined, so I can read it? tree-sitting
Any structure question, any language tree-sitting
A literal string or regex plain ripgrep
Which files are most about a concept? bm25
First look at an unfamiliar repo exploring-codebases

Non-Python code has no pyright binding resolution to offer, so there is no version of this skill that applies to it.

Prerequisites

uv tool install ripgrep

tree-sitting installs automatically when needed — for --expand context expansion and for the binding-resolved --refs/--def/--hover tier, which uses it to resolve symbol positions. Only the bare tree-sitter package is fetched; the language grammars ship bundled.

Primary Command

SKILL_DIR=/mnt/skills/user/searching-codebases

python3 $SKILL_DIR/scripts/search.py SOURCE "query1" ["query2" ...] [OPTIONS]

SOURCE is any of:

  • Local directory path
  • GitHub URL (downloads tarball automatically)
  • uploads (uses /mnt/user-data/uploads/)
  • project (uses /mnt/project/)
  • Path to a .zip or .tar.gz archive

Search Modes

Regex mode (patterns, identifiers, literal text):

python3 $SKILL_DIR/scripts/search.py ./repo "def handle_error"
python3 $SKILL_DIR/scripts/search.py ./repo "class.*Exception" --regex
python3 $SKILL_DIR/scripts/search.py ./repo "TODO|FIXME|HACK"

Semantic mode (concepts, natural language):

python3 $SKILL_DIR/scripts/search.py ./repo "retry logic with backoff" --semantic
python3 $SKILL_DIR/scripts/search.py ./repo "authentication flow"
python3 $SKILL_DIR/scripts/search.py ./repo "error handling strategy"

Auto-detection: short queries and code-like tokens → regex. Multi-word natural language → semantic. Override with --regex or --semantic.

Binding-resolved mode (Python only — pyright via the python-lsp skill):

python3 $SKILL_DIR/scripts/search.py ./repo --refs SYMBOL    # find all real uses
python3 $SKILL_DIR/scripts/search.py ./repo --def SYMBOL     # go-to-definition
python3 $SKILL_DIR/scripts/search.py ./repo --hover SYMBOL   # inferred type/signature

Regex mode matches text, so a cross-reference for a function false-positives on shadowed and same-named-but-unrelated symbols. --refs is binding-resolved: pyright excludes the unrelated same-named symbol and follows imports. Use it when you need a true "find all callers/users" for a .py symbol, not a text grep.

The tier is engaged lazily — pyright's index cost is paid only when you ask for --refs/--def/--hover, never on ordinary searches. It is Python-only; for non-.py sources, or when pyright/node is unavailable, it prints a one-line degradation note and falls back to the regex text path. Each takes a single bare symbol name and is mutually exclusive with the other two and with text queries.

Options

  • --regex / --semantic: Force search mode
  • --refs SYMBOL / --def SYMBOL / --hover SYMBOL: Binding-resolved Python queries via pyright (see Binding-resolved mode above)
  • --expand: Return full function bodies via tree-sitting AST context
  • --benchmark: Compare indexed regex vs brute-force ripgrep
  • --branch NAME: Git branch for GitHub URLs (default: main)
  • --skip DIRS: Comma-separated directories to skip
  • --json: Machine-readable output
  • -v: Show index stats and query routing decisions

How It Works

Regex search builds a sparse n-gram inverted index over all files. Queries are decomposed into literal fragments, looked up in the index to identify candidate files (typically 90-99% reduction), then verified with ripgrep. Frequency-weighted n-grams make rare character sequences more selective.

Semantic search builds a TF-IDF index over code chunks (functions, classes, structural entries). Queries are ranked by cosine similarity.

Context expansion (--expand) uses tree-sitting's AST cache to identify function/class boundaries, returning complete structural units rather than line fragments. On first use, tree-sitting scans the repo (~700ms for 250 files); subsequent expansions are sub-millisecond.

Small codebases (< 20 files) skip indexing entirely — direct ripgrep is faster when there's nothing to narrow.

Mixed Queries

Multiple queries can use different modes in a single invocation. Each query is auto-routed independently, and indexes are built once per mode:

python3 $SKILL_DIR/scripts/search.py ./repo \
  "class.*Error" \
  "error recovery strategy" \
  "def retry"

Dependencies

  • tree-sitting: Provides AST context expansion for --expand and the symbol→position resolution that seeds the binding-resolved tier (--refs/--def/--hover). Auto-installs the bare tree-sitter package when either is used (grammars are bundled). Regex and semantic search work without it.
  • ripgrep: Required for regex verification. Install via uv tool install ripgrep.
  • scikit-learn: Required for semantic mode. Installs automatically.
  • python-lsp: Provides the binding-resolved tier (--refs/--def/--hover). Self-bootstraps pyright on first use and requires system node (v18+). Not required — without it those flags degrade to the regex text path.

When to Use — narrow, by design

The ONE recommended use: binding-resolved Python symbol queries.

  • "find all callers of X" / "where is X really defined" for a .py symbol, when same-named-but-unrelated symbols would pollute a text grep. Empirical basis: rg get on psf/requests returned 232 hits, 224 of them false; --refs get excluded all 224 (2026-06-15).

When NOT to Use — which is most of the time

Everything else. Measured head-to-head on real issue-localization tasks (7 scikit-learn issues with merged fix-PRs, gold = PR diff files, 2026-07-04, replicating the file-discovery metric of arXiv:2602.11988):

  • Literal tokens / identifiers: naive rg -l tied or beat the indexed tier on recall@10 in every instance, at 0.4s vs 25s.
  • Concept / natural-language search: the TF-IDF semantic tier never beat identifier grep — not even on identifier-poor issues, which are themselves rare (~0.3% of merged-PR traffic in the sample).
  • First encounter / "what is this repo": use exploring-codebases.
  • Repos under ~20 files: read them.

The self-test before invoking: would plain rg return the same answer? If yes, use rg. The indexed-regex and semantic tiers are retained for completeness and for corpora where they may yet earn their cost (very large repos, non-code document collections), but they carry the burden of proof.

Files

  • scripts/search.py — Entry point, query routing, output formatting
  • scripts/resolve.py — Input source resolution (GitHub, uploads, archives)
  • scripts/context.py — tree-sitting-based AST context expansion
  • scripts/ngram_index.py — Sparse n-gram inverted index, regex decomposition
  • scripts/sparse_ngrams.py — Core n-gram algorithms, frequency weights
  • scripts/code_rag.py — TF-IDF semantic search over code chunks
  • scripts/lsp_refs.py — Binding-resolved Python tier: symbol→position resolution (tree-sitting), pyright queries (python-lsp), soft fallback
Files (claude-skills)
  • scripts
    • code_rag.py 20.4 KB
      Loading…
    • context.py 6.3 KB
      Loading…
    • lsp_refs.py 18.4 KB
      Loading…
    • ngram_index.py 20.7 KB
      Loading…
    • resolve.py 4.5 KB
      Loading…
    • search.py 12.8 KB
      Loading…
    • sparse_ngrams.py 6.1 KB
      Loading…
  • tests
    • fixture
      • pkg
        • models.py 188 B
          Loading…
        • other.py 62 B
          Loading…
        • service.py 125 B
          Loading…
        • __init__.py 0 B
          Loading…
    • test_lsp_refs.py 5.5 KB
      Loading…
  • CHANGELOG.md 3.6 KB
    Loading…
  • README.md 1.9 KB
    # searching-codebases
    
    Find code in any codebase by regex pattern or natural language concept. Auto-routes between n-gram indexed regex search (2-20x faster than ripgrep) and TF-IDF semantic search. Expands results to full functions via tree-sitting AST data. For Python sources, a binding-resolved reference/definition tier (pyright, via python-lsp) finds real callers and definitions without same-name false positives.
    
    ## Features
    
    - **Dual search modes** — regex (pattern/identifier) and semantic (natural language concepts), auto-routed per query
    - **N-gram indexed regex** — sparse inverted index narrows candidate files by 90-99% before ripgrep verification
    - **TF-IDF semantic search** — cosine similarity ranking over code chunks (functions, classes)
    - **Binding-resolved Python tier** — `--refs`/`--def`/`--hover` via pyright: true find-all-callers and go-to-definition that exclude same-named, unrelated symbols and follow imports. Engaged lazily; degrades to the regex text path when pyright/node is unavailable
    - **AST context expansion** — `--expand` returns complete function/class bodies instead of line fragments
    - **Flexible sources** — accepts GitHub URLs, local directories, uploaded files/archives, or project knowledge
    - **Mixed queries** — multiple queries with different modes in a single invocation; indexes built once per mode
    
    ## Dependencies
    
    - **ripgrep** — required for regex verification
    - **tree-sitting** — auto-installs the bare `tree-sitter` package when needed: for `--expand` context and for the symbol→position resolution that seeds the binding-resolved tier (grammars ship bundled). Regex and semantic search work without it
    - **scikit-learn** — required for semantic mode (auto-installs)
    - **python-lsp** — provides the binding-resolved tier (`--refs`/`--def`/`--hover`); self-bootstraps pyright on first use and needs system `node` (v18+). Without it those flags degrade to the regex text path
    
  • SKILL.md 8.1 KB
    ---
    name: searching-codebases
    description: >-
      Binding-resolved Python symbol queries via pyright — every true caller
      (--refs), the real definition (--def), or an inferred signature (--hover)
      of a .py symbol, excluding the same-named false positives text search
      cannot tell apart. Use when a task needs ALL callers or users of a named
      Python symbol and grep would over-match. Python only, and only for the
      caller/definition question: measured 2026-07-04 on real issue-localization
      tasks, plain ripgrep tied or beat the semantic and indexed-regex tiers at
      4-60x less wall-clock, so those tiers survive below without being a
      default.
    metadata:
      version: 2.5.0
    ---
    
    # Searching Codebases
    
    Find code in any codebase by pattern or concept. One entry point, two
    search strategies, automatic routing.
    
    ## When NOT to use this skill
    
    Python callers and definitions only. Every other code question is cheaper
    elsewhere, and the measurement below is why this skill is edge-case-only.
    
    | Situation | Use |
    |---|---|
    | What symbols does this file contain? | tree-sitting |
    | Where is X defined, so I can read it? | tree-sitting |
    | Any structure question, any language | tree-sitting |
    | A literal string or regex | plain ripgrep |
    | Which files are most about a concept? | bm25 |
    | First look at an unfamiliar repo | exploring-codebases |
    
    Non-Python code has no pyright binding resolution to offer, so there is no
    version of this skill that applies to it.
    
    ## Prerequisites
    
    ```bash
    uv tool install ripgrep
    ```
    
    tree-sitting installs automatically when needed — for `--expand` context
    expansion and for the binding-resolved `--refs`/`--def`/`--hover` tier, which
    uses it to resolve symbol positions. Only the bare `tree-sitter` package is
    fetched; the language grammars ship bundled.
    
    ## Primary Command
    
    ```bash
    SKILL_DIR=/mnt/skills/user/searching-codebases
    
    python3 $SKILL_DIR/scripts/search.py SOURCE "query1" ["query2" ...] [OPTIONS]
    ```
    
    SOURCE is any of:
    - Local directory path
    - GitHub URL (downloads tarball automatically)
    - `uploads` (uses `/mnt/user-data/uploads/`)
    - `project` (uses `/mnt/project/`)
    - Path to a `.zip` or `.tar.gz` archive
    
    ## Search Modes
    
    **Regex mode** (patterns, identifiers, literal text):
    ```bash
    python3 $SKILL_DIR/scripts/search.py ./repo "def handle_error"
    python3 $SKILL_DIR/scripts/search.py ./repo "class.*Exception" --regex
    python3 $SKILL_DIR/scripts/search.py ./repo "TODO|FIXME|HACK"
    ```
    
    **Semantic mode** (concepts, natural language):
    ```bash
    python3 $SKILL_DIR/scripts/search.py ./repo "retry logic with backoff" --semantic
    python3 $SKILL_DIR/scripts/search.py ./repo "authentication flow"
    python3 $SKILL_DIR/scripts/search.py ./repo "error handling strategy"
    ```
    
    Auto-detection: short queries and code-like tokens → regex. Multi-word
    natural language → semantic. Override with `--regex` or `--semantic`.
    
    **Binding-resolved mode** (Python only — pyright via the `python-lsp` skill):
    ```bash
    python3 $SKILL_DIR/scripts/search.py ./repo --refs SYMBOL    # find all real uses
    python3 $SKILL_DIR/scripts/search.py ./repo --def SYMBOL     # go-to-definition
    python3 $SKILL_DIR/scripts/search.py ./repo --hover SYMBOL   # inferred type/signature
    ```
    
    Regex mode matches *text*, so a cross-reference for a function false-positives
    on shadowed and same-named-but-unrelated symbols. `--refs` is **binding-resolved**:
    pyright excludes the unrelated same-named symbol and follows imports. Use it when
    you need a true "find all callers/users" for a `.py` symbol, not a text grep.
    
    The tier is engaged **lazily** — pyright's index cost is paid only when you ask
    for `--refs`/`--def`/`--hover`, never on ordinary searches. It is **Python-only**;
    for non-`.py` sources, or when pyright/node is unavailable, it prints a one-line
    degradation note and falls back to the regex text path. Each takes a single bare
    symbol name and is mutually exclusive with the other two and with text queries.
    
    ## Options
    
    - `--regex` / `--semantic`: Force search mode
    - `--refs SYMBOL` / `--def SYMBOL` / `--hover SYMBOL`: Binding-resolved Python
      queries via pyright (see Binding-resolved mode above)
    - `--expand`: Return full function bodies via tree-sitting AST context
    - `--benchmark`: Compare indexed regex vs brute-force ripgrep
    - `--branch NAME`: Git branch for GitHub URLs (default: main)
    - `--skip DIRS`: Comma-separated directories to skip
    - `--json`: Machine-readable output
    - `-v`: Show index stats and query routing decisions
    
    ## How It Works
    
    **Regex search** builds a sparse n-gram inverted index over all files.
    Queries are decomposed into literal fragments, looked up in the index
    to identify candidate files (typically 90-99% reduction), then verified
    with ripgrep. Frequency-weighted n-grams make rare character sequences
    more selective.
    
    **Semantic search** builds a TF-IDF index over code chunks (functions,
    classes, structural entries). Queries are ranked by cosine similarity.
    
    **Context expansion** (`--expand`) uses tree-sitting's AST cache to
    identify function/class boundaries, returning complete structural units
    rather than line fragments. On first use, tree-sitting scans the repo
    (~700ms for 250 files); subsequent expansions are sub-millisecond.
    
    **Small codebases** (< 20 files) skip indexing entirely — direct ripgrep is
    faster when there's nothing to narrow.
    
    ## Mixed Queries
    
    Multiple queries can use different modes in a single invocation. Each query
    is auto-routed independently, and indexes are built once per mode:
    
    ```bash
    python3 $SKILL_DIR/scripts/search.py ./repo \
      "class.*Error" \
      "error recovery strategy" \
      "def retry"
    ```
    
    ## Dependencies
    
    - **tree-sitting**: Provides AST context expansion for `--expand` *and* the
      symbol→position resolution that seeds the binding-resolved tier
      (`--refs`/`--def`/`--hover`). Auto-installs the bare `tree-sitter` package
      when either is used (grammars are bundled). Regex and semantic search work
      without it.
    - **ripgrep**: Required for regex verification. Install via `uv tool install ripgrep`.
    - **scikit-learn**: Required for semantic mode. Installs automatically.
    - **python-lsp**: Provides the binding-resolved tier (`--refs`/`--def`/`--hover`).
      Self-bootstraps pyright on first use and requires system `node` (v18+). Not
      required — without it those flags degrade to the regex text path.
    
    ## When to Use — narrow, by design
    
    The ONE recommended use: **binding-resolved Python symbol queries**.
    
    - "find all callers of `X`" / "where is `X` really defined" for a `.py`
      symbol, when same-named-but-unrelated symbols would pollute a text grep.
      Empirical basis: `rg get` on psf/requests returned 232 hits, 224 of them
      false; `--refs get` excluded all 224 (2026-06-15).
    
    ## When NOT to Use — which is most of the time
    
    Everything else. Measured head-to-head on real issue-localization tasks
    (7 scikit-learn issues with merged fix-PRs, gold = PR diff files,
    2026-07-04, replicating the file-discovery metric of arXiv:2602.11988):
    
    - **Literal tokens / identifiers**: naive `rg -l` tied or beat the indexed
      tier on recall@10 in every instance, at 0.4s vs 25s.
    - **Concept / natural-language search**: the TF-IDF semantic tier never
      beat identifier grep — not even on identifier-poor issues, which are
      themselves rare (~0.3% of merged-PR traffic in the sample).
    - **First encounter / "what is this repo"**: use exploring-codebases.
    - **Repos under ~20 files**: read them.
    
    The self-test before invoking: would plain `rg` return the same answer?
    If yes, use rg. The indexed-regex and semantic tiers are retained for
    completeness and for corpora where they may yet earn their cost (very
    large repos, non-code document collections), but they carry the burden
    of proof.
    
    ## Files
    
    - `scripts/search.py` — Entry point, query routing, output formatting
    - `scripts/resolve.py` — Input source resolution (GitHub, uploads, archives)
    - `scripts/context.py` — tree-sitting-based AST context expansion
    - `scripts/ngram_index.py` — Sparse n-gram inverted index, regex decomposition
    - `scripts/sparse_ngrams.py` — Core n-gram algorithms, frequency weights
    - `scripts/code_rag.py` — TF-IDF semantic search over code chunks
    - `scripts/lsp_refs.py` — Binding-resolved Python tier: symbol→position
      resolution (tree-sitting), pyright queries (python-lsp), soft fallback
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.