Claude Skill

exploring-data

Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. A

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Download oaustegard-claude-skills-plugins_data-and-visualization_skills_exploring-data-2c9224e.zip · 22 KB

Install

skills CLI npx skills add https://github.com/oaustegard/claude-skills/tree/main/plugins/data-and-visualization/skills/exploring-data
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oaustegard-claude-skills@llmmart
Git git clone https://github.com/oaustegard/claude-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oaustegard/claude-skills collection as a plugin from our marketplace. Git is the plain clone.

README

exploring-data

Exploratory data analysis using ydata-profiling. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Generates interactive HTML or JSON reports with statistics, visualizations, correlations, and quality alerts.

Skill manifest

Exploring Data

0. Route by size FIRST

ls -la <filepath>   # or: wc -l for row estimate
  • < 200MB and < ~5M rows → ydata-profiling path (section A). Exact stats, interactive HTML.
  • Larger → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size.
  • Task-specific ops (any size): duplicates, join feasibility, drift → section C.

A. Standard path (ydata-profiling)

1. Check if installed (instant)

bash /mnt/skills/user/exploring-data/scripts/check_install.sh

Returns: installed or not_installed

2. Install if needed (one-time, ~19s)

if [ "$(bash /mnt/skills/user/exploring-data/scripts/check_install.sh)" = "not_installed" ]; then
    bash /mnt/skills/user/exploring-data/scripts/install_ydata.sh
fi

3. Run analysis (always generates JSON + HTML by default)

bash /mnt/skills/user/exploring-data/scripts/analyze.sh <filepath> [minimal|full] [html|json]

Defaults: minimal + html (also generates JSON)

Output:

  • eda_report.html - Interactive report for user
  • eda_report.json - Machine-readable for Claude analysis

4. If Claude needs to analyze (user asks "what do you think?" etc.)

python /mnt/skills/user/exploring-data/scripts/summarize_insights.py /mnt/user-data/outputs/eda_report.json

Claude should read the stdout markdown summary, NOT the full JSON report.

5. Present findings visually (don't just hand over the ydata HTML)

The ydata report is exhaustive but dense; a link to it is a weak deliverable. Turn the JSON into a compact dashboard of the findings that matter:

python3 /mnt/skills/user/exploring-data/scripts/visualize_findings.py \
    /mnt/user-data/outputs/eda_report.json
# → /mnt/user-data/outputs/eda_findings.html

Emits a single self-contained HTML file (Chart.js from cdnjs, dark-mode aware): missingness by column (tiered good/bad), the most skewed or zero-inflated numeric distributions as small-multiple histograms, and the largest categorical breakdowns. --top N caps charts per category (default 6). Also reads profile_large.py --json output, so the large-file path gets the same treatment.

Present BOTH files: eda_findings.html for the headline read, eda_report.html for the full drill-down. In a chat surface that renders inline visuals, prefer rendering the two or three findings that actually answer the user's question as inline charts over linking a file — a link the user has to open is the weakest form of "showing" data.

Modes

Minimal (default, 5-10s): overview, variable analysis, correlations, missing values, alerts Full (10-20s): minimal + scatter matrices, sample data, character analysis

Full-mode triggers: "comprehensive analysis", "detailed EDA", "full profiling", "deep analysis". Otherwise minimal.

Time series

If the data has a datetime index/column and the user cares about temporal behavior (gaps, trends, seasonality, autocorrelation), pass tsmode=True to ProfileReport — run the venv python directly instead of analyze.sh:

ProfileReport(df, tsmode=True, sortby="<datetime_col>", title=...)

This adds gap detection, stationarity and seasonality checks that the default report omits.

Small-file drift

Comparing two versions of a dataset that BOTH fit in memory: use ydata's native compare — ProfileReport(df_a).compare(ProfileReport(df_b)).to_file(...). For files too big to load, or comparing against a months-old file you no longer have, use the sketch snapshot/drift ops in section C.

B. Large-file path (DuckDB, fixed memory)

1. Install deps (idempotent, ~10s first time)

bash /mnt/skills/user/exploring-data/scripts/install_large.sh

2. Profile

python3 /mnt/skills/user/exploring-data/scripts/profile_large.py <file> [--json out.json]

Streams the file through DuckDB: per-column null%, approximate distinct counts (HLL), min/max/mean, approximate quantiles (t-digest) for numerics, top-5 values for strings, plus quality flags (mostly-null, constant, id-like columns). Markdown lands on stdout — read it directly, no summarize step needed. Handles csv/tsv/parquet/json/ndjson. 1M rows profiles in seconds; memory is flat regardless of file size.

For ad-hoc follow-up queries on the same large file, use DuckDB SQL directly (duckdb.connect().execute("SELECT ... FROM read_csv_auto('...')")) rather than loading pandas.

C. Sketch ops (any file size, fixed memory)

All via scripts/sketch_ops.py (deps from install_large.sh). These answer questions profilers don't:

Near-duplicate rows

python3 sketch_ops.py dups <file> [--threshold 0.9] [--cols a,b,c] [--unweighted]

Exact duplicates counted by hash; near-duplicates via MinHash LSH over row tokens. --threshold is a weighted Jaccard cutoff: a token occurring c times in a row counts c times, so new york new york and york new score 0.5 rather than 1.0. Pass --unweighted for set semantics, where repeats are discarded. Use --cols to restrict to the columns that define identity.

Key overlap / join feasibility

python3 sketch_ops.py overlap <fileA> <fileB> --key <col> [--key-b <col>]

Theta sketches per key column → estimated intersection, Jaccard, and "% of A's keys in B" both ways — answers "will this join hold?" without loading either file.

Drift vs stored baseline

python3 sketch_ops.py snapshot <file> --out baseline.sketch.json   # ~20KB
python3 sketch_ops.py drift <newfile> --baseline baseline.sketch.json

Snapshot serializes HLL (all columns) + KLL quantile sketches (numeric columns) to a small JSON. Drift reports schema changes, >10% shifts in distinct counts, and IQR-relative quantile movement. The snapshot is a few KB — store it (repo, memory) and diff next month's delivery against it without keeping the original file.

Note: snapshot/dups stream rows through Python (~1M rows in a few seconds); profile_large is pure DuckDB and faster. For a quick look at a big file, profile first, sketch ops only when the question calls for them.

Files (claude-skills)
  • references
    • USAGE.md 1.8 KB
      Loading…
  • scripts
    • analyze.sh 2.2 KB
      Loading…
    • check_install.sh 184 B
      Loading…
    • install_large.sh 335 B
      Loading…
    • install_ydata.sh 717 B
      Loading…
    • profile_large.py 4.2 KB
      Loading…
    • sketch_ops.py 8.1 KB
      Loading…
    • summarize_insights.py 12.3 KB
      Loading…
    • visualize_findings.py 12.1 KB
      Loading…
  • tests
    • test_sketch_ops.py 3 KB
      Loading…
  • CHANGELOG.md 3.5 KB
    Loading…
  • README.md 300 B
    # exploring-data
    
    Exploratory data analysis using ydata-profiling. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Generates interactive HTML or JSON reports with statistics, visualizations, correlations, and quality alerts.
    
  • SKILL.md 6.5 KB
    ---
    name: exploring-data
    description: Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
    metadata:
      version: 0.2.1
    ---
    
    # Exploring Data
    
    ## 0. Route by size FIRST
    
    ```bash
    ls -la <filepath>   # or: wc -l for row estimate
    ```
    
    - **< 200MB and < ~5M rows** → ydata-profiling path (section A). Exact stats, interactive HTML.
    - **Larger** → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size.
    - **Task-specific ops** (any size): duplicates, join feasibility, drift → section C.
    
    ## A. Standard path (ydata-profiling)
    
    ### 1. Check if installed (instant)
    ```bash
    bash /mnt/skills/user/exploring-data/scripts/check_install.sh
    ```
    Returns: `installed` or `not_installed`
    
    ### 2. Install if needed (one-time, ~19s)
    ```bash
    if [ "$(bash /mnt/skills/user/exploring-data/scripts/check_install.sh)" = "not_installed" ]; then
        bash /mnt/skills/user/exploring-data/scripts/install_ydata.sh
    fi
    ```
    
    ### 3. Run analysis (always generates JSON + HTML by default)
    ```bash
    bash /mnt/skills/user/exploring-data/scripts/analyze.sh <filepath> [minimal|full] [html|json]
    ```
    
    **Defaults:** minimal + html (also generates JSON)
    
    **Output:**
    - `eda_report.html` - Interactive report for user
    - `eda_report.json` - Machine-readable for Claude analysis
    
    ### 4. If Claude needs to analyze (user asks "what do you think?" etc.)
    ```bash
    python /mnt/skills/user/exploring-data/scripts/summarize_insights.py /mnt/user-data/outputs/eda_report.json
    ```
    
    Claude should read the stdout markdown summary, NOT the full JSON report.
    
    ### 5. Present findings visually (don't just hand over the ydata HTML)
    
    The ydata report is exhaustive but dense; a link to it is a weak deliverable.
    Turn the JSON into a compact dashboard of the findings that matter:
    ```bash
    python3 /mnt/skills/user/exploring-data/scripts/visualize_findings.py \
        /mnt/user-data/outputs/eda_report.json
    # → /mnt/user-data/outputs/eda_findings.html
    ```
    Emits a single self-contained HTML file (Chart.js from cdnjs, dark-mode aware):
    missingness by column (tiered good/bad), the most skewed or zero-inflated
    numeric distributions as small-multiple histograms, and the largest categorical
    breakdowns. `--top N` caps charts per category (default 6). Also reads
    `profile_large.py --json` output, so the large-file path gets the same treatment.
    
    Present BOTH files: `eda_findings.html` for the headline read, `eda_report.html`
    for the full drill-down. In a chat surface that renders inline visuals, prefer
    rendering the two or three findings that actually answer the user's question as
    inline charts over linking a file — a link the user has to open is the weakest
    form of "showing" data.
    
    ### Modes
    
    **Minimal (default, 5-10s):** overview, variable analysis, correlations, missing values, alerts
    **Full (10-20s):** minimal + scatter matrices, sample data, character analysis
    
    Full-mode triggers: "comprehensive analysis", "detailed EDA", "full profiling", "deep analysis". Otherwise minimal.
    
    ### Time series
    If the data has a datetime index/column and the user cares about temporal behavior
    (gaps, trends, seasonality, autocorrelation), pass `tsmode=True` to ProfileReport —
    run the venv python directly instead of analyze.sh:
    ```python
    ProfileReport(df, tsmode=True, sortby="<datetime_col>", title=...)
    ```
    This adds gap detection, stationarity and seasonality checks that the default
    report omits.
    
    ### Small-file drift
    Comparing two versions of a dataset that BOTH fit in memory: use ydata's native
    compare — `ProfileReport(df_a).compare(ProfileReport(df_b)).to_file(...)`.
    For files too big to load, or comparing against a months-old file you no longer
    have, use the sketch snapshot/drift ops in section C.
    
    ## B. Large-file path (DuckDB, fixed memory)
    
    ### 1. Install deps (idempotent, ~10s first time)
    ```bash
    bash /mnt/skills/user/exploring-data/scripts/install_large.sh
    ```
    
    ### 2. Profile
    ```bash
    python3 /mnt/skills/user/exploring-data/scripts/profile_large.py <file> [--json out.json]
    ```
    
    Streams the file through DuckDB: per-column null%, approximate distinct counts
    (HLL), min/max/mean, approximate quantiles (t-digest) for numerics, top-5
    values for strings, plus quality flags (mostly-null, constant, id-like
    columns). Markdown lands on stdout — read it directly, no summarize step
    needed. Handles csv/tsv/parquet/json/ndjson. 1M rows profiles in seconds;
    memory is flat regardless of file size.
    
    For ad-hoc follow-up queries on the same large file, use DuckDB SQL directly
    (`duckdb.connect().execute("SELECT ... FROM read_csv_auto('...')")`) rather
    than loading pandas.
    
    ## C. Sketch ops (any file size, fixed memory)
    
    All via `scripts/sketch_ops.py` (deps from install_large.sh). These answer
    questions profilers don't:
    
    ### Near-duplicate rows
    ```bash
    python3 sketch_ops.py dups <file> [--threshold 0.9] [--cols a,b,c] [--unweighted]
    ```
    Exact duplicates counted by hash; near-duplicates via MinHash LSH over row
    tokens. `--threshold` is a **weighted** Jaccard cutoff: a token occurring c
    times in a row counts c times, so `new york new york` and `york new` score 0.5
    rather than 1.0. Pass `--unweighted` for set semantics, where repeats are
    discarded. Use `--cols` to restrict to the columns that define identity.
    
    ### Key overlap / join feasibility
    ```bash
    python3 sketch_ops.py overlap <fileA> <fileB> --key <col> [--key-b <col>]
    ```
    Theta sketches per key column → estimated intersection, Jaccard, and "% of A's
    keys in B" both ways — answers "will this join hold?" without loading either
    file.
    
    ### Drift vs stored baseline
    ```bash
    python3 sketch_ops.py snapshot <file> --out baseline.sketch.json   # ~20KB
    python3 sketch_ops.py drift <newfile> --baseline baseline.sketch.json
    ```
    Snapshot serializes HLL (all columns) + KLL quantile sketches (numeric
    columns) to a small JSON. Drift reports schema changes, >10% shifts in
    distinct counts, and IQR-relative quantile movement. The snapshot is a few KB
    — store it (repo, memory) and diff next month's delivery against it without
    keeping the original file.
    
    Note: snapshot/dups stream rows through Python (~1M rows in a few seconds);
    profile_large is pure DuckDB and faster. For a quick look at a big file,
    profile first, sketch ops only when the question calls for them.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.