# Execution surfaces

How to run the engines on each surface, and when to escalate between them.

## Persistent kernel, JavaScript (Bun)

One-time setup per machine — the bundled script installs `@duckdb/node-api` into a user-level
cache outside any repo and prints the absolute import path (its only stdout line):

```bash
bash scripts/ensure-js-deps.sh          # run from the skill directory
```

In the kernel — top-level `require` may not exist, dynamic import always works:

```js
const { DuckDBInstance } = await import("<printed path>");
const db = await DuckDBInstance.create(":memory:");
const conn = await db.connect();
const reader = await conn.runAndReadAll("SELECT category, SUM(v) AS total FROM 'data.csv' GROUP BY 1");
reader.getRowObjects();                  // array of plain row objects
```

- The connection and any tables created live across cells — connect once per session, reuse.
- COUNT/SUM over integer columns return BigInt; convert (`Number(x)` or `String(x)`) before
  `JSON.stringify`, which throws on BigInt.
- Bun builtins cover ingest gaps with zero installs: `Bun.JSONL.parse`, `Bun.JSON5.parse`,
  `Bun.XML.parse`, `Bun.TOML.parse`, `Bun.Archive` for tarballs.
- nodejs-polars is NOT part of this skill's toolkit: its API lags the Python release by
  major versions (option objects that work in Python throw napi type errors). Polars work
  belongs to the Python kernel (below).

## Persistent kernel, Python (the default Python surface)

duckdb, numpy, and matplotlib are typically resident — import and use them directly.
Polars and pyarrow rarely ship with a kernel, so inject them once per session (run from the
skill directory; the script installs on first use, then just prints the path):

```python
import subprocess, sys
site = subprocess.run(["bash", "scripts/ensure-py-deps.sh", sys.executable],
                      capture_output=True, text=True, check=True).stdout.strip()
sys.path.insert(0, site)
import polars as pl
import pyarrow
```

- The install goes to a user cache keyed to the kernel's interpreter version; the
  interpreter itself is never mutated (it is frequently an externally-managed system
  Python, and mutating it breaks other tools).
- After injection the whole Python stack is resident: `duckdb.sql(...).pl()` hands off via
  Arrow, `duckdb.register(name, df)` goes the other way, and Polars lazy pipelines run
  in-kernel across cells.
- `duckdb.sql("SELECT ... FROM 'data.csv'")` queries files in place; without the injection,
  keep results in DuckDB or fetch plain Python values (`.fetchall()`).
- matplotlib figures render natively in kernels that display rich output; also save a PNG so
  the artifact survives the session.

## uv lane (fallback and isolation)

```bash
uv run --with duckdb --with polars --with pyarrow --with numpy python -c "<code>"
```

- Reach for it when there is no kernel, or when a heavy, crash-prone one-shot should not
  run inside (and possibly take down) the kernel.
- Include exactly the packages the code imports, plus pyarrow whenever `.pl()` is used.
- Each invocation pays process spawn plus imports (roughly 0.3s warm) and re-reads its inputs —
  fine for one-shots, wasteful for exploration loops.
- Past a few lines, a temp file beats `-c` quoting: write the script, `uv run script.py`.

## No kernel at all

Same engines, one process per batch of questions:

```bash
bun -e '<the JavaScript kernel pattern above>'         # DuckDB via @duckdb/node-api
uv run --with duckdb python -c "<sql via duckdb.sql>"  # DuckDB via Python
uv run scripts/quick-query.py data.csv "SELECT ..."    # zero-code CLI fallback
```

## Escalation rules

Start on the resident kernel. Move a step down when a concrete need appears:

- polars/pyarrow missing from the kernel — inject via `ensure-py-deps.sh` (above), not a
  uv one-shot.
- Crash-prone or memory-hungry one-shot that should not take the kernel down — uv lane.
- No kernel on this harness — one-shot recipes above.
- Data lives remotely or exceeds local RAM — read `placement.md` and move the query, not
  the data.

# Placement: where should this computation live?

Decide before touching the data. Wrong placement wastes minutes (re-scanning a file queried
ten times) or kills the machine (loading a dataset larger than RAM and swapping).

## The probe (run first, once)

Three facts, one cell or script:

```python
import os, shutil, subprocess, sys
size = os.path.getsize("data.csv")            # bytes on disk
disk_free = shutil.disk_usage(".").free       # spill headroom
if sys.platform == "darwin":
    ram = int(subprocess.run(["sysctl", "-n", "hw.memsize"], capture_output=True, text=True).stdout)
else:
    ram = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")
# row estimate without loading (DuckDB streams the scan):
# duckdb.sql("SELECT count(*) FROM 'data.csv'")
```

CSV typically expands 2-5x in memory (string columns dominate); Parquet expands less
predictably — compressed columns can inflate 10x. Estimate the working set from the
decompressed size of the columns the question actually touches, not the file size.

## In memory — load once, iterate

When the working set stays within roughly 25% of free RAM AND the session will run repeated
queries: load once (`CREATE TABLE t AS SELECT ...` in DuckDB, or a collected DataFrame),
then iterate. One scan up front converts every later query from a file re-scan into
milliseconds. Prune at load time — select only the needed columns, filter obvious dross —
so the resident table is the working set, not the raw file.

## In place / streaming — single pass, or bigger than RAM

- Single-pass questions: query the file directly (`FROM 'data.csv'`). Loading first is pure
  waste.
- Bigger than RAM, SQL-shaped: cap DuckDB and let it spill —

  ```sql
  SET memory_limit = '4GB';
  SET temp_directory = '/tmp/duckdb_spill';
  ```

  Aggregations, sorts, and window functions run out-of-core: slower, but bounded.
- Bigger than RAM, DataFrame-shaped: Polars streaming (`collect(engine="streaming")` on a
  lazy plan) in the resident kernel — or a uv one-shot on kernel-less harnesses.
- Manual chunked loops (read N rows, process, repeat) are the last resort — the engines'
  own out-of-core paths are faster and simpler than hand-rolled chunking.

## Remote, in place — move the query to the data

- Files behind http(s)/S3: DuckDB's httpfs extension reads Parquet and CSV remotely with
  projection and predicate pushdown —

  ```sql
  INSTALL httpfs; LOAD httpfs;   -- one-time per environment
  SELECT region, SUM(amount) FROM 'https://example.com/sales.parquet'
  WHERE sale_date >= '2026-01-01' GROUP BY region;
  ```

  Only matching row groups and referenced columns cross the network, not the file.
- Data on another machine you can execute on (a remote worker with more RAM, a box closer
  to the data): run the query there and return the aggregate. A group-by result is
  kilobytes; the source is gigabytes.
- Decision rule: result much smaller than data — move the query. Repeated local iteration
  on one slice — move a pruned copy of that slice once, then work locally in memory.

## Hardware notes

- Both engines parallelize across all cores by default; leave that alone except on shared
  machines (`SET threads = N` in DuckDB, `POLARS_MAX_THREADS` for Polars).
- Sustained swapping is the failure mode to avoid on memory-tight machines: when the probe
  says the working set is close to free RAM, choose streaming, not hope.

# Polars lane (resident Python kernel)

When the work is DataFrame-shaped, Polars is the right engine — and it runs in the resident
Python kernel by default. Kernels rarely ship polars/pyarrow preinstalled, so inject them
once per session; the install lands in a user cache keyed to the kernel's interpreter, and
the interpreter itself is never mutated (run with the skill directory as cwd, or spell out
the script's absolute path):

```python
import subprocess, sys
site = subprocess.run(["bash", "scripts/ensure-py-deps.sh", sys.executable],
                      capture_output=True, text=True, check=True).stdout.strip()
sys.path.insert(0, site)
import polars as pl
```

After this, Polars lives across cells like every other resident engine: lazy frames,
intermediate results, and the DuckDB handoff all persist with no per-call process cost.

## When Polars wins over DuckDB SQL

- Expression-chain transforms: many derived columns, per-column conditional logic, string
  pipelines — `with_columns` chains read and optimize better than nested SQL SELECTs.
- Reshapes: `unpivot`/`pivot` beat SQL gymnastics.
- Larger-than-RAM pipelines: the streaming engine executes lazy plans in chunks.
- Window-heavy feature engineering with `over()`.

SQL-shaped work (joins, aggregation, ad-hoc questions) stays in DuckDB; mixed pipelines hand
off zero-copy (below) instead of forcing one engine to do everything.

## Current API (1.x) — older spellings fail or warn

Training data is full of the pre-1.0 API. Current names:

| Use | Not |
| --- | --- |
| `pl.scan_csv` / `pl.scan_parquet` + `.collect()` | eager `read_*` on big files |
| `.group_by(...)` | `.groupby(...)` |
| `pl.len()` | `pl.count()` |
| `.collect(engine="streaming")` | `.collect(streaming=True)` |
| `.unpivot(...)` | `.melt(...)` |

Lazy first: `scan_*` builds a plan, pushes filters and projections down to the file read, and
executes once at `.collect()`. Eager `read_*` is for small files mutated interactively.

```python
out = (pl.scan_csv("data.csv")
       .filter(pl.col("value") > 100)
       .group_by("category")
       .agg(pl.col("value").sum().alias("total"), pl.len().alias("n"))
       .sort("total", descending=True)
       .collect())
```

## Zero-copy handoff with DuckDB

Both engines speak Arrow, so mixed pipelines pay no serialization cost — all in-kernel:

```python
import duckdb
df = duckdb.sql("SELECT * FROM 'orders.csv' o JOIN 'items.csv' i USING (id)").pl()
shaped = df.with_columns((pl.col("qty") * pl.col("price")).alias("rev"))
duckdb.register("shaped", shaped)
out = duckdb.sql("SELECT category, SUM(rev) AS total FROM shaped GROUP BY 1").pl()
```

- `.pl()` requires pyarrow — the injection above provides it; without it, it raises
  `ModuleNotFoundError`.
- Never `.df()`: it requires pandas (raising without it), and pandas is banned and absent.

## Streaming past RAM

```python
out = (pl.scan_parquet("huge.parquet")
       .filter(pl.col("status") == "active")
       .group_by("region").agg(pl.len())
       .collect(engine="streaming"))
```

Streaming executes lazy plans only — keep the plan lazy end-to-end, with no intermediate
`.collect()` breaking it into eager pieces.

## Kernel-less fallback (uv one-shot)

On a harness with no persistent kernel, the same code runs as one-shots — batch several
questions per process, since each invocation pays spawn plus imports:

```bash
uv run --with duckdb --with polars --with pyarrow python -c "
import duckdb
import polars as pl
df = duckdb.sql(\"SELECT * FROM 'data.csv'\").pl()
print(df.group_by('category').agg(pl.len()).sort('category'))
"
```

# uv Setup — Per-Platform

This skill's uv lane (kernel-less harnesses, isolated one-shots) runs through `uv run --with ...`, and `scripts/ensure-py-deps.sh` uses uv as its installer. If `uv --version` fails, set uv up with the automated scripts or the manual commands below, then verify.

## Automated (recommended)

| Platform | Command |
|---|---|
| macOS / Linux / WSL / Git Bash | `bash scripts/setup-uv.sh` |
| Windows (PowerShell) | `powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1` |

Both scripts: detect OS + architecture → install uv to the latest release when missing → upgrade it when present (`uv self update`) → make it resolvable for the current shell → verify with `uv --version`. They are idempotent — safe to re-run any time.

## Manual install per platform

### macOS

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh        # official installer → ~/.local/bin/uv
# or, with Homebrew:
brew install uv
```

### Linux (x86_64 / aarch64)

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh        # official installer → ~/.local/bin/uv
```

The installer detects glibc vs musl and downloads the right static binary. On minimal containers, ensure `curl` (or `wget`) exists; `wget -qO- https://astral.sh/uv/install.sh | sh` is the fallback.

### Windows (native)

```powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# or, with winget:
winget install --id=astral-sh.uv -e
```

### Windows Subsystem for Linux / Git Bash

Use the Linux/macOS installer inside the Unix shell, not the PowerShell installer:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

### CI

```yaml
# GitHub Actions
- uses: astral-sh/setup-uv@v5
# or plain shell anywhere:
- run: curl -LsSf https://astral.sh/uv/install.sh | sh
```

## PATH notes

- The official installers put the binary in `~/.local/bin` (Unix) or `%USERPROFILE%\.local\bin` (Windows). New shells get it automatically on most setups; an already-open shell needs `export PATH="$HOME/.local/bin:$PATH"` (Unix) or `$env:Path = "$env:USERPROFILE\.local\bin;$env:Path"` (PowerShell) once.
- Homebrew and winget install into their own prefixes that are already on PATH.

## Upgrade to latest

```bash
uv self update
```

`uv self update` only works for official-installer binaries; Homebrew/winget installs upgrade through their package manager (`brew upgrade uv` / `winget upgrade astral-sh.uv`). The setup scripts handle this automatically.

## Verify

```bash
uv --version
```

## Offline / air-gapped

Download the matching archive from the uv GitHub releases page, extract it, and put the `uv` binary anywhere on PATH. `uv run --with <pkg>` still needs network for first-time package resolution unless a mirror is configured via `UV_INDEX_URL`.

# Visualization

A chart exists to answer a question at a glance. Render it with matplotlib (resident in most
Python kernels; `uv run --with matplotlib` otherwise), then look at it before delivering —
a chart nobody inspected is not evidence.

## When to chart

Chart when the user asked for one, and default to charting when the answer is a shape prose
cannot carry: a trend over time, a distribution, a comparison across many categories, a
relationship between variables. Skip the chart when a number or a five-row table answers the
question — decoration dilutes the answer.

## Chart type follows the question

| Question shape | Chart |
| --- | --- |
| How did X change over time? | line, datetime x-axis |
| Which categories are biggest? | horizontal bar, sorted by value |
| How is X distributed? | histogram (tune bin count) or box plot per group |
| Is X related to Y? | scatter; add a trend line only when it aids the eye |
| Composition of a whole? | stacked or 100% bar — pie only for four or fewer slices |
| Many series over time? | small multiples over one spaghetti chart |

## Quality bar — every chart

- Title states the finding ("Seoul overtook Busan in March"), not the dataset name.
- Axis labels carry units. Tick density stays readable: `fig.autofmt_xdate()` for dates,
  rotate or abbreviate long category names.
- Size for the medium: inline chat reads well around `figsize=(10, 6)` at default dpi;
  documents want `dpi=150` or more at export.
- `tight_layout()` (or `constrained_layout=True`) before saving — clipped labels are the
  most common chart defect.
- Few series: label lines directly, or keep the legend inside empty plot space. Many
  series: gray the context, color only the series that answers the question.
- The default color cycle is fine; avoid rainbow palettes and 3D. Sort categorical bars by
  value, never alphabetically.

## CJK and other non-Latin text

Matplotlib's default font renders CJK as empty boxes (tofu). Set a fallback before plotting
whenever any label or title contains CJK:

```python
import platform
import matplotlib
cjk = {"Darwin": "AppleGothic", "Windows": "Malgun Gothic"}.get(platform.system(), "Noto Sans CJK KR")
matplotlib.rcParams["font.family"] = [cjk, "DejaVu Sans"]
matplotlib.rcParams["axes.unicode_minus"] = False   # keeps the minus sign rendering
```

## Output contract

1. Save a PNG next to the work: `plt.savefig(path, dpi=150, bbox_inches="tight")`.
2. Also render inline when the surface displays rich output (kernels usually do).
3. Report the file path together with the answer.

## Visual QA — mandatory

Open the produced image — kernel display, or the harness's image-reading surface — and
check four things: labels readable and unclipped, no tofu or mojibake, nothing overlapping,
and the chart actually shows the finding the title claims. A failed check means fix and
re-render, not ship with a caveat. This one pass catches nearly every chart defect;
skipping it is how tofu titles reach users.

#!/usr/bin/env bash
# Install @duckdb/node-api into a user-level cache (outside any repo) and print the
# absolute import path as the ONLY stdout line. Idempotent: re-runs reuse the install.
set -euo pipefail

log() { printf '[ensure-js-deps] %s\n' "$*" >&2; }

CACHE_DIR="${OMO_DATA_SCIENTIST_CACHE:-$HOME/.cache/omo-data-scientist}"
IMPORT_PATH="$CACHE_DIR/node_modules/@duckdb/node-api/lib/index.js"

if ! command -v bun >/dev/null 2>&1; then
  log "bun is required (https://bun.sh); install it, or use the uv lane instead."
  exit 1
fi

if [ ! -f "$IMPORT_PATH" ]; then
  log "installing @duckdb/node-api into $CACHE_DIR"
  mkdir -p "$CACHE_DIR"
  [ -f "$CACHE_DIR/package.json" ] || printf '{"name":"omo-data-scientist-cache","private":true}\n' > "$CACHE_DIR/package.json"
  (cd "$CACHE_DIR" && bun add @duckdb/node-api 1>&2)
fi

if [ ! -f "$IMPORT_PATH" ]; then
  log "install finished but $IMPORT_PATH is missing; inspect $CACHE_DIR"
  exit 1
fi

printf '%s\n' "$IMPORT_PATH"

#!/usr/bin/env bash
# Install polars + pyarrow for a given Python interpreter into a user-level cache
# (never mutating the interpreter itself) and print the site directory as the ONLY
# stdout line. Idempotent: re-runs reuse the install. arg1 = python executable
# (default: python3); pass the kernel's sys.executable for kernel use.
set -euo pipefail

log() { printf '[ensure-py-deps] %s\n' "$*" >&2; }

PYTHON_BIN="${1:-python3}"

if ! command -v uv >/dev/null 2>&1; then
  log "uv is required (see references/uv-setup.md); install it first."
  exit 1
fi
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1 && [ ! -x "$PYTHON_BIN" ]; then
  log "python executable not found: $PYTHON_BIN"
  exit 1
fi

TAG="$("$PYTHON_BIN" -c 'import sys; print(f"cp{sys.version_info[0]}{sys.version_info[1]}")')" \
  || { log "not a working python interpreter: $PYTHON_BIN"; exit 1; }
CACHE_DIR="${OMO_DATA_SCIENTIST_CACHE:-$HOME/.cache/omo-data-scientist}"
SITE_DIR="$CACHE_DIR/py-$TAG"

if [ ! -d "$SITE_DIR/polars" ] || [ ! -d "$SITE_DIR/pyarrow" ]; then
  log "installing polars + pyarrow for $TAG into $SITE_DIR"
  mkdir -p "$SITE_DIR"
  uv pip install --python "$PYTHON_BIN" --target "$SITE_DIR" polars pyarrow 1>&2
fi

if [ ! -d "$SITE_DIR/polars" ]; then
  log "install finished but $SITE_DIR/polars is missing; inspect $SITE_DIR"
  exit 1
fi

printf '%s\n' "$SITE_DIR"

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "duckdb",
#     "polars",
#     "numpy",
#     "pyarrow",
#     "typer",
#     "rich",
# ]
# ///

"""Quick data query runner. SQL arg -> DuckDB; --filter (Polars SQL) -> Polars; --describe -> schema + stats."""

from __future__ import annotations

from pathlib import Path

import typer
from rich import print as rprint
from rich.table import Table


def _run_duckdb(file_path: Path, sql: str) -> None:
    import duckdb

    table_ref = f"'{file_path}'"
    stem = file_path.stem
    query = sql
    for alias in ("data", "df", stem):
        query = query.replace(f"FROM {alias} ", f"FROM {table_ref} ")
        query = query.replace(f"FROM {alias}\n", f"FROM {table_ref}\n")
        query = query.replace(f"from {alias} ", f"from {table_ref} ")
        query = query.replace(f"from {alias}\n", f"from {table_ref}\n")
        if query.endswith(f"FROM {alias}") or query.endswith(f"from {alias}"):
            query = query[: -len(alias)] + table_ref

    result = duckdb.sql(query)
    df = result.pl()
    _print_polars(df)


def _run_polars_filter(file_path: Path, expr: str) -> None:
    import polars as pl

    df = _read_file(file_path)
    filtered = df.filter(pl.sql_expr(expr))
    _print_polars(filtered)


def _run_describe(file_path: Path) -> None:
    df = _read_file(file_path)

    rprint(f"\n[bold]Schema:[/bold] {file_path.name} ({len(df)} rows × {len(df.columns)} cols)")
    for name, dtype in zip(df.columns, df.dtypes):
        rprint(f"  {name}: [cyan]{dtype}[/cyan]")

    rprint("\n[bold]Statistics:[/bold]")
    _print_polars(df.describe())


def _read_file(file_path: Path):  # noqa: ANN202
    import polars as pl

    suffix = file_path.suffix.lower()
    if suffix == ".csv":
        return pl.read_csv(file_path)
    if suffix == ".parquet":
        return pl.read_parquet(file_path)
    if suffix == ".json":
        return pl.read_json(file_path)
    if suffix in (".jsonl", ".ndjson"):
        return pl.read_ndjson(file_path)
    rprint(f"[red]Unsupported format:[/red] {suffix} (Excel: export to CSV or Parquet first)")
    raise SystemExit(1)


def _print_polars(df) -> None:  # noqa: ANN001
    table = Table(show_lines=False)
    for col_name in df.columns:
        table.add_column(col_name)
    for row in df.iter_rows():
        table.add_row(*(str(v) for v in row))
    rprint(table)
    rprint(f"[dim]{df.shape[0]} rows × {df.shape[1]} cols[/dim]")


def main(
    file: Path = typer.Argument(help="Data file (csv, parquet, json, ndjson)"),
    sql: str = typer.Argument(None, help="SQL query (uses DuckDB). Use 'data' as table name."),
    filter_expr: str = typer.Option(None, "--filter", "-f", help="Polars SQL filter, e.g. 'amount > 100'"),
    describe: bool = typer.Option(False, "--describe", "-d", help="Print schema + stats"),
) -> None:
    """Query data files with DuckDB (SQL) or Polars (SQL filter expressions)."""
    if not file.exists():
        rprint(f"[red]File not found:[/red] {file}")
        raise SystemExit(1)

    if describe:
        _run_describe(file)
    elif filter_expr:
        _run_polars_filter(file, filter_expr)
    elif sql:
        _run_duckdb(file, sql)
    else:
        _run_describe(file)


if __name__ == "__main__":
    typer.run(main)

#!/usr/bin/env bash
set -euo pipefail

log() { printf '[setup-uv] %s\n' "$*"; }

os="$(uname -s 2>/dev/null || echo unknown)"
arch="$(uname -m 2>/dev/null || echo unknown)"
log "detected os=${os} arch=${arch}"

case "$os" in
  Darwin|Linux) ;;
  MINGW*|MSYS*|CYGWIN*)
    log "Git Bash / MSYS environment detected - installing the Windows build is not supported here."
    log "Use the Linux installer inside WSL, or run scripts/setup-uv.ps1 in PowerShell instead."
    exit 1
    ;;
  *)
    log "unsupported OS: ${os} - see references/uv-setup.md for manual steps."
    exit 1
    ;;
esac

if command -v uv >/dev/null 2>&1; then
  current="$(uv --version 2>/dev/null | head -n1)"
  log "uv already installed (${current}) - upgrading"
  if uv self update >/dev/null 2>&1; then
    log "uv self update succeeded"
  else
    log "uv self update unavailable (package-manager install) - trying Homebrew"
    if command -v brew >/dev/null 2>&1; then
      brew upgrade uv >/dev/null 2>&1 || brew install uv
    else
      log "no Homebrew; keeping current uv (already functional)"
    fi
  fi
else
  log "uv not found - installing latest via the official installer"
  if command -v curl >/dev/null 2>&1; then
    curl -LsSf https://astral.sh/uv/install.sh | sh
  elif command -v wget >/dev/null 2>&1; then
    wget -qO- https://astral.sh/uv/install.sh | sh
  elif command -v brew >/dev/null 2>&1; then
    log "no curl/wget - installing via Homebrew"
    brew install uv
  else
    log "neither curl, wget, nor brew available - install one of them and re-run."
    exit 1
  fi
fi

if ! command -v uv >/dev/null 2>&1; then
  export PATH="$HOME/.local/bin:$PATH"
fi

if ! command -v uv >/dev/null 2>&1; then
  log "FAIL: uv still not on PATH after install - add \$HOME/.local/bin to PATH and retry."
  exit 1
fi

log "OK: $(uv --version)"