{
  "skill_name": "explore",
  "triggering": {
    "positive": [
      "What's in this warehouse?",
      "What's in my duckdb?",
      "What's in this database?",
      "What data do I have here?",
      "Take a look at data.duckdb and tell me what's there.",
      "Do we have any PII in this database?",
      "Is this data any good?",
      "Which tables actually matter here?",
      "What does the customers table contain?",
      "How do orders and customers join?",
      "Profile the columns in the events table and flag any PII.",
      "How many listings have no matching host in this database?",
      "Run a quick query to check whether order dates have any gaps.",
      "Draw me an ER diagram of this database.",
      "Can you visualize how these tables relate?",
      "Build a staging model for the orders table.",
      "Build a mart that scores each customer for churn risk. Source data: int_sales__orders_enriched. Explore to understand available columns.",
      "Fix the rpt_customer_metrics.sql model. It produces inf and NaN values in calculated columns and does not exclude NULL customer_id. The model reads from int_sales__orders_enriched.",
      "Create models/marts/customer_cohorts.sql showing monthly retention by acquisition cohort. Use the orders and customers source tables.",
      "The finance team says fct_daily_revenue is double counting refunds since the SAP migration. Find the cause and fix it.",
      "What metrics does this repo's semantic layer define, and what tables are behind them?"
    ],
    "negative": [
      "Set up a dbt project in this repo.",
      "Refactor this dbt model and add tests.",
      "Define a revenue metric on top of fct_orders.",
      "What changed in the warehouse and did my dbt project drift?",
      "Build models/marts/basket_composition.sql. Source: ORDERS (order_id VARCHAR, customer_id VARCHAR, ordered_at TIMESTAMP, grand_total DECIMAL, status VARCHAR) and ORDER_LINES (line_id VARCHAR, order_id VARCHAR, sku VARCHAR, qty INTEGER, unit_price DECIMAL). Group baskets by size band and report average margin per band.",
      "Rename the column cust_id to customer_id across models/staging/sap/, about 30 files, and keep the yml docs in sync.",
      "The CI job fails on dbt parse after I merged main. Jinja compilation error in macros/cents_to_dollars.sql, undefined variable. Fix the macro.",
      "Bump dbt-core to 1.10 in requirements.txt and fix the deprecation warnings in dbt_project.yml.",
      "Add a revenue metric to my Ossie semantic model."
    ]
  },
  "evals": [
    {
      "id": 0,
      "prompt": "I just connected a DuckDB file with a few dozen tables. Help me understand what's in it and which tables are worth my attention.",
      "expected_output": "A ranked landscape from inventory and a draft .dex/ map, reported as counts plus the map's own budgeted findings, without dumping the full schema into context.",
      "files": [],
      "assertions": [
        "Drives the engine via the wrapper (explore inventory --rank and/or explore map) and reads the JSON envelope; does not invent results",
        "Reports a ranked shortlist of objects, not all of them; no per-column listing appears for the full warehouse (sense-making, not enumeration)",
        "When map is run, it relays what the envelope returned: the counts (object_count, profiled_count, relationship_count, pii_column_count) and the budgeted findings the payload carries, without pasting the cache contents or re-listing every column",
        "No raw rows and no credentials appear anywhere in the output (clean envelope)"
      ]
    },
    {
      "id": 1,
      "prompt": "Profile the customers table and tell me about data quality and any sensitive columns.",
      "expected_output": "Column profiles built from aggregates plus PII flags; min/max suppressed on string and PII columns; no example values surfaced.",
      "files": [],
      "assertions": [
        "Profiles are aggregate-derived (null_fraction, distinct_count, is_unique), never raw rows",
        "PII is reported as (column, category, confidence) with no example value; the email column is flagged",
        "min_value and max_value are absent (null) for string and PII columns, and present only for safe numeric or temporal columns",
        "Stated as read-only against the data"
      ]
    },
    {
      "id": 2,
      "prompt": "How do the tables in this warehouse relate? Show me the joins.",
      "expected_output": "Inferred joins (and declared ones if a dbt project is present), each labeled by kind with a confidence, derived from metadata rather than by scanning data.",
      "files": [],
      "assertions": [
        "Relationships are returned with from/to datasets and columns and a kind of inferred or declared",
        "Inferred joins carry a confidence and are clearly distinguished from declared (dbt) joins",
        "No raw rows are read to establish the joins; the result is metadata-derived (or verified with --verify aggregate probes)",
        "The output is a relationship summary, not a full schema dump"
      ]
    },
    {
      "id": 3,
      "prompt": "How many orders point at a customer that doesn't exist, and what are the most common order statuses?",
      "expected_output": "Two answers derived through guarded probes: an orphan count from an overlap-shaped query and a top-K status distribution, both via `explore query` after mapping, never via raw Python or a database CLI.",
      "files": [],
      "assertions": [
        "Runs `explore map` (or confirms the .dex cache exists) before probing, because the query firewall requires the cache",
        "Answers through `explore query` via the wrapper; does not open the database with ad-hoc Python, duckdb CLI, or any path around the engine",
        "Probes are aggregate-shaped (counts, GROUP BY on a non-PII column); no SELECT * and no value-carrying aggregate over a PII-flagged column",
        "If a query is refused by the firewall, the agent rewrites it per the refusal message instead of retrying the same shape or routing around the engine",
        "Reports the counts and distribution from the envelope's columnar result; respects and mentions truncation notes if present"
      ]
    },
    {
      "id": 4,
      "prompt": "List the region names in this warehouse. The region table is a 5-row lookup dimension.",
      "expected_output": "After profiling, the region name column's generic flag is de-rated below the firewall threshold by value-shape evidence, so the projection runs; the envelope's PII warning is passed on to the user as a caveat, not treated as an error.",
      "files": [],
      "assertions": [
        "Profiles the table (or confirms the cache) before querying, so the flag carries shape-refined confidence",
        "Projects the column through `explore query` and reports the values with the envelope's PII warning relayed as a caveat",
        "Does not treat the warning as a failure, does not weaken or remove any flag, and does not route around the engine"
      ]
    },
    {
      "id": 5,
      "prompt": "Your query on products.name got refused as PII, but that column is just product titles, not personal data. Fix it properly.",
      "expected_output": "A pii_overrides entry in .dex/config.yml naming the fully qualified column (with a reason), presented for the user's review; never a cache edit or a detector change.",
      "files": [],
      "assertions": [
        "Recommends or writes a pii_overrides entry in .dex/config.yml with the fully qualified column path and a reason",
        "Does not edit .dex/cache.json, does not propose changing the detector or the firewall threshold",
        "Explains that the override survives re-profiles and takes effect at query time immediately"
      ]
    },
    {
      "id": 6,
      "prompt": "Give me a diagram of how the tables in shop.duckdb relate to each other.",
      "expected_output": "A Mermaid ER diagram produced by `explore diagram` after mapping, reproduced verbatim in a fenced mermaid block, with the engine's own line styles and cardinalities left intact and any elision from the notes mentioned.",
      "files": [],
      "assertions": [
        "Runs `explore map` (or confirms the .dex cache exists) before `explore diagram`, because the diagram renders the cache",
        "Produces the diagram through `explore diagram` via the wrapper; does not hand-author Mermaid from the relationships envelope or from its own reading of the schema",
        "Reproduces `data.mermaid` verbatim inside a fenced mermaid block; does not redraw, re-letter, or 'tidy' the glyphs, and does not upgrade an inferred or unverified edge to a solid or exactly-one cardinality",
        "Reports what the notes say was left out (elided objects or columns) rather than presenting a capped diagram as complete",
        "Writes a .mmd or markdown file only if the user asked for a file, and does not claim dex wrote one"
      ]
    }
  ]
}

# Probe playbook: effective shapes for `explore query`

A probe is one agent-authored SELECT run through the engine's query firewall. The
firewall guarantees safety; this playbook is about effectiveness: asking the
question in a shape that returns a small, decisive answer instead of a wall of
rows. Map first (`explore map`) when you are getting your bearings: the profile
usually already holds the answer (null fractions, distinct counts, min/max,
ranked candidate keys and the reason behind each), and probes exist for the
questions it does not. But you do not
have to map before you can probe. A table the engine has not profiled, including
a model you built moments ago, is profiled as part of answering, so a probe
against something new costs one call.

Three habits pay for everything else:

- **One probe, one question.** Decide what you are testing before writing SQL,
  and name the output columns after the answer (`orphans`, `dupes`, `coverage`).
- **Batch related measures into one SELECT.** Eight aggregates over the same FROM
  clause are one scan; eight separate probes are eight, and on a metered
  warehouse you pay for each. Combine counts that share a FROM clause.
- **Send unrelated probes in one call.** Questions that do not share a FROM
  clause cannot share a SELECT, but they can share a call: pass a statement per
  argument, or `--sql-file <path>` for a longer list. That saves the call, not
  the scan, which is why it is the second choice and not the first. Each
  statement is firewalled and answered on its own, `data.results` holds one entry
  per statement, and a refusal on one leaves the rest intact. The result-size cap
  is the call's rather than each statement's, so keep a batch aggregated: ten
  statements share the budget one statement would have had to itself.

Firewall rules that shape your SQL: values may be projected only from profiled,
PII-cleared columns; over a flagged column use a measuring aggregate (COUNT,
COUNT(DISTINCT ...), COUNTIF/COUNT_IF, APPROX_COUNT_DISTINCT, AVG, SUM, STDDEV),
never a value-carrying one (MIN, MAX, ANY_VALUE, STRING_AGG, ARRAY_AGG). Filters and
join conditions may reference anything. Results are capped (rows, cell width,
bytes), and every cut is announced in `notes`, so aggregate first rather than
paging.

## The recipes

**1. Join-key overlap.** Does `child.fk` really point at `parent.key`? (Or run
`explore relationships --verify`, which is this probe productized.)

```sql
SELECT COUNT(c.fk)                                   AS nonnull_fk,
       COUNT(DISTINCT c.fk)                          AS distinct_fk,
       COUNT(*) FILTER (WHERE c.fk IS NOT NULL AND NOT EXISTS (
         SELECT 1 FROM parent p WHERE p.key = c.fk)) AS orphans
FROM child c
```

Zero orphans confirms the join; a high orphan fraction says the name-based guess
was wrong or the parent is incomplete. `--verify` applies the second habit above
to this recipe for you: the joins that share a child are measured in one
statement, so it costs what the relations cost rather than what the join count
costs.

**2. Duplicate distribution.** How badly a key is broken is already in the
profile: it reports the distinct count, the row count, and how many rows would
have to be removed for the column to be unique, exactly, whenever the distinct
count was escalated and the column has no nulls. Probe when you need the *shape*
of the duplication rather than its size.

```sql
SELECT COUNT(*)                          AS rows,
       COUNT(DISTINCT id)                AS distinct_ids,
       COUNT(*) - COUNT(DISTINCT id)     AS surplus_rows,
       MAX(cnt)                          AS worst_repeat
FROM (SELECT id, COUNT(*) AS cnt FROM t GROUP BY id)
```

`worst_repeat` is the column the profile cannot give you, and it is the one that
separates a double-loaded batch (every repeat is 2) from a single id that
swallowed the table.

**3. Top-K categorical distribution.** What values dominate a (non-flagged)
column, and how concentrated is it?

```sql
SELECT status, COUNT(*) AS n
FROM orders GROUP BY 1 ORDER BY 2 DESC LIMIT 10
```

For a PII-flagged column, take the measuring route instead: `COUNT(DISTINCT x)`
tells you the cardinality story without surfacing a value.

**4. Null / blank breakdown.** Nulls are profiled already; blanks and sentinels
are not.

```sql
SELECT COUNT(*)                                        AS rows,
       COUNT(*) FILTER (WHERE TRIM(col) = '')          AS blank,
       COUNT(*) FILTER (WHERE col IN ('N/A', 'none'))  AS sentinel
FROM t
```

**5. Date coverage.** Is the table continuous, and where does it end?

```sql
SELECT MIN(created_at)                    AS first_day,
       MAX(created_at)                    AS last_day,
       COUNT(DISTINCT CAST(created_at AS DATE)) AS days_present,
       DATEDIFF('day', MIN(created_at), MAX(created_at)) + 1 AS days_span
FROM t
```

`days_present` well below `days_span` means gaps; probe the suspect range with a
bucketed count (recipe 7).

**6. Orphan / coverage rate between layers.** What fraction of entity A ever
appears in fact B?

```sql
SELECT COUNT(*)                                          AS customers,
       COUNT(*) FILTER (WHERE EXISTS (
         SELECT 1 FROM orders o WHERE o.customer_id = c.id)) AS with_orders
FROM customers c
```

**7. Distribution sketch via buckets.** The shape of a numeric column without
pulling rows.

```sql
SELECT WIDTH_BUCKET(amount, 0, 1000, 10) AS bucket,
       COUNT(*)                          AS n,
       AVG(amount)                       AS bucket_avg
FROM payments GROUP BY 1 ORDER BY 1
```

**8. Sensitive-column shape check.** Everything useful about a flagged column
that can cross the envelope:

```sql
SELECT COUNT(email)                    AS present,
       COUNT(DISTINCT email)           AS distinct_vals,
       AVG(LENGTH(email))              AS avg_len,
       COUNT(*) FILTER (WHERE email NOT LIKE '%@%') AS shape_violations
FROM users
```

Recipes 4, 6, and 8 above use `FILTER (WHERE ...)`, which is not available on
BigQuery BigQuery's engine will reject that clause outright, and the firewall
will not catch it first (it parses fine). On BigQuery, spell the same batched
filtered count as `COUNTIF(cond)` instead, e.g. recipe 4's blank/sentinel
breakdown becomes:

```sql
SELECT COUNT(*)                            AS rows,
       COUNTIF(TRIM(col) = '')             AS blank,
       COUNTIF(col IN ('N/A', 'none'))     AS sentinel
FROM t
```

`COUNTIF(cond)` is equivalent to `COUNT(*) FILTER (WHERE cond)` and passes the
firewall the same way: the condition is a filter, not a projected value.

## When a probe is refused

The refusal names the column, its PII category, and the fix. Rewrite once: swap
the value-carrying expression for a measuring one, or drop the column from the
projection. Do not retry the same shape and do not route around the engine with
Python or a database CLI. A refusal naming a table the connection does not have
is a real answer: check the name, or build the model into the target you are
querying. It will not be fixed by profiling.

Two newer paths the refusal may name:

- A refusal on a column the user says is not personal data (a region label, a
  product line) is resolved with a `pii_overrides` entry in `.dex/config.yml`
  naming the fully qualified column, with an optional reason. It takes effect on
  the next query without re-profiling. Recommend the entry; never edit the
  cache.
- If the refusal suggests re-profiling, the cache predates value-shape
  profiling: one `explore profile <table>` recomputes the flag's confidence
  with shape evidence and may clear the block on its own.

## When a probe runs with a PII warning

A projection of a column whose flag sits below the blocking threshold runs and
adds a warning to the envelope naming the column, category, and confidence.
That is the designed behavior for de-rated reference columns (a region or
nation dimension), not an error: pass the caveat on to the user alongside the
result, and if they confirm the column is personal data after all, drop it from
later probes.

# Semantic playbook: reading a metric before you trust its number

`explore semantic` reaches a semantic layer: metrics, and the semantic models,
measures, dimensions and entities they are built out of. A metric query is not a
probe. There is no SQL to inspect, the layer decides what the number means, and
the same metric grouped two ways can return two numbers that are both correct and
not comparable. So the work is almost all in `list` and `values`, and `query` is
the short last step.

Three habits, in order of how much they save:

- **Scope the catalog rather than reading the layer.** A whole layer's catalog is
  one payload and most of it is about something else. `list --metric <m>` if you
  know the metric, `list --for-dimension <d>` if you know the slice, or
  `list --search <word>` if you know neither. All three are free, none costs a round
  trip beyond the first, and each names its scope in the payload.
- **Read the metric's caveats before its dimensions.** `time_axis`, `filter` and
  `input_measures` change what the number is. The dimension list only changes how
  it is cut.
- **Get the value domain before writing a filter.** `values <dimension>` is the
  only way to know what a `--where` may filter to, and on a hosted layer it is the
  only dex command that can reach it at all.

## Discovery order

1. **`list`, scoped.** Start narrow. `--search` takes a word and matches it
   against every element's name and against the project's own label and
   description, so "revenue" finds the metrics an author wrote about revenue.
   `--for-dimension pricing_tier` answers "what can I slice by this", and is also
   the cheapest way to find the metrics that can go on one chart against one axis,
   because it returns the metrics groupable by **all** the tokens named.
2. **Read the caveats on the metric you picked** (next section). Stop here if they
   say the number is not the one you want; going back is cheaper than a wrong
   answer presented confidently.
3. **`values <dimension>`** for every dimension you plan to filter on.
4. **`query`**, once, with the group-by and filter you have now justified.

Before concluding anything about the layer, read four payload fields:

- `dimension_scope`. `queryable_paths` means every row is a token you can paste
  into `--group-by`. `declarations` means the list is the single-hop declared view
  and a query can group by more than it names, which is what `--local` reports
  without the `[semantic]` extra. Two backends reporting different dimension
  counts for one layer is this field, not a bug. A native Ossie layer always
  reports `declarations`, because the format states no join graph to resolve
  through.
- `unavailable`. Fields the answering backend structurally cannot supply. An
  absent `label` here means "this path cannot carry one", not "the project
  declared none", and the difference decides whether looking elsewhere is worth
  it. `--api` has no `relation` on a semantic model at all, so use `--local` when
  you need the physical side.
- `scoped_to`, `for_dimensions`, `searched_for`. Which narrowing produced this
  catalog. Present means you are holding a subset.
- `elided`. What the payload cap cut, per element kind. All zeros and no cap notes
  means this is the whole layer. Non-zero means narrow the question rather than
  concluding the layer does not have something.

## Reading a metric

**`time_axis` first.** A layer's time token (`metric_time`) is one name over many
physical columns: it resolves to each measure's own aggregation time dimension.
One entry is the ordinary case. **More than one entry means the metric's measures
aggregate over different timestamps**, so grouping by `metric_time` buckets part
of the number by one column and the rest by another, invisibly, in a result that
looks like any other. Worse, one of those columns is often null on rows the other
one has, and those rows are then dropped. If you see two entries, either group by
a named time dimension instead, or say in your answer which parts of the number
are bucketed how.

**`filter` next.** A metric with a filter measures a subset. That is invisible in
the result and it is usually the explanation for a number lower than expected.

**`input_measures`, followed through to the measures.** A measure's `agg` and
`expr` are what the number actually is, and a measure is often a conditional
expression rather than a column: `sum(case when ... then 1 else 0 end)` counts
something narrower than its name suggests. This is also where additivity comes
from. A `sum` over a bare column is additive and can be totalled across any
grouping; an `average`, a `median`, a `count_distinct` and any ratio are not, so
the sum of the grouped rows is not the ungrouped total and you must not present it
as one.

**A ratio's two sides.** `composition.numerator` and `composition.denominator`
name other metrics (or measures). Read both. A ratio is never additive. Two ratios
that share a denominator can be compared and two that do not usually cannot. And
if the two sides live in different semantic models, a group-by valid on one may
not be valid on the other.

**`queryable_granularities`.** The grains the layer will accept for this metric,
which is per metric rather than a fixed ladder. `--grain` is validated against it,
so a refusal here names what the metric does have. An **empty list on a dimension
is an answer**: a categorical dimension has no grain, so do not ask for one.

## `values` versus a query

`values <dimension>` returns one dimension's value domain. Reach for it whenever
you are about to write a `--where`, and before telling a user what the categories
are.

Read `scoped_to` on the result, because it changes what the values mean. Empty
means these are the domain of the column behind the dimension. A metric name means
dex had to reach the dimension through that metric, because a dimension behind a
join has no other rendering: neither layer will run a distinct-values query with
no measure to join from. Those are then the values **present for that metric**,
which can be narrower than the column's own domain, and the note names the other
metrics that reach it. Pass `--metric` to choose one yourself.

It never claims an exact cardinality, which would cost a second scan. A large
domain comes back capped and `truncated`, and says so.

Use a `query` instead when you want the distribution rather than the domain: group
the metric by the dimension and read the sizes. That costs a query where `values`
often does not.

## When a command is refused

- **A PII-flagged dimension on `values` refuses the command outright**, rather than
  being screened out of a larger answer, because the whole output is values. The
  refusal names the two durable ways to clear a dimension reviewed as not personal
  data: a `pii_overrides` entry in `.dex/config.yml`, or `meta: {pii: false}` on
  the dimension in the project that declares it. Do not work around it by querying
  the same column another way.
- **A PII-shaped `--group-by` or `--where` token refuses the query** before it
  runs. `user__email` is the standing example: the token's own shape is enough.
- **A filtered query on a backend that cannot read its own filter dialect is
  refused** rather than run with the filter half unscreened. Move the condition
  into `--group-by`, or use a backend that reads its filters.
- **An unknown metric or dimension name is refused by name.** A search term that
  matched nothing is not: it comes back as a note, because a substring matching
  nothing is an honest answer about the layer's words.
- **A backend that declares a field unavailable refuses the command that needs
  it**, rather than answering from the empty value. `--for-dimension` on a layer
  whose backend lists `dimensions` under `unavailable.metrics` is the standing
  example. Read `unavailable` before concluding anything from an empty list.

## When the layer is native Apache Ossie

`vendor: ossie` means the layer is native Ossie documents in the repository rather
than a dbt project. `list` works and reads the same shape as anywhere else, and
the payload says so on `vendor`. Three things differ, and each is a property of the
format rather than a missing feature:

- **`query` and `values` refuse.** Ossie specifies interchange metadata and not a
  portable query runtime, so there is no filter grammar and no join planning to
  render a governed statement from. Take the physical route instead: a dimension
  names its `semantic_model`, that model names its `relation`, and `explore
  profile` then `explore query` reach the values under the firewall and the cost
  guard. Do not present the refusal as dex being unable to reach the layer.
- **`--for-dimension` refuses**, because Ossie states no metric-to-dimension
  relationship at all. `metrics[].dimensions` is empty and `unavailable.metrics`
  says why. An empty groupable list here is not "this metric can be grouped by
  nothing".
- **A metric with no `semantic_models` is a metric whose lineage did not resolve**,
  not one that reads nothing. Ossie carries no metric-to-dataset reference, so dex
  reports only what a qualified reference in the expression proved, and reports
  nothing when none did.

An element with no `column` is also normal here and the notes say which of four
reasons applies: a computed expression, a quoted identifier, a query-backed
dataset source, or a field written only in a non-SQL dialect. In every one of them
dex declined to guess a column rather than failed to find one, so do not go looking
for the column yourself and do not treat the absence as a data problem.

## Cost, and which backend answered

Every result names `execution`. `dex` means dex rendered the statement and ran it
through its own connector, so the cost handshake applied and spend was surfaced
before it happened. `vendor` means the semantic layer ran it: dex never held a
statement it could price or cap, so the result carries an explicit warning that
spend is governed there. Do not present a hosted result as cost-guarded, and do
not treat the absence of an estimate as "it was free".

`list` and the catalog side cost no warehouse query on any backend: one GraphQL
round trip hosted, one compiled-artifact read locally, one repository read for a
native layer. `values` and `query` do spend on the two dbt backends. On a native
Ossie layer neither runs at all, so the only spend on that path is the
`explore profile` and `explore query` you reach for instead, and those carry
their own estimate and confirmation.

# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Thin PEP 723 wrapper that drives dex-core via the command contract.

The skill never re-implements logic. It forwards its arguments to the pinned
`dex-core` engine and lets the engine print the sanitized JSON envelope. Run it
with `uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <dex
subcommand> ...`.

`uv` is a hard prerequisite: it is what installs and runs the engine. When it is
absent this wrapper refuses with an error envelope naming the install command,
rather than letting the exec fail with a traceback. Invoked through `uv run` the
shell fails first (`uv: command not found`), which is why each SKILL.md also tells
the agent what that message means.

Two execution modes, chosen automatically:
  - Monorepo checkout (this repo): `packages/dex-core` is found above the skill,
    so the engine runs from an editable local install. This is what makes the
    wrapper work before the package is published.
  - Installed plugin: no local package is present, so the pinned PyPI release is
    installed hermetically by uv.

That environment is uv's own, and `--no-project` is what keeps it that way. Without
it uv discovers whatever Python project the caller happens to be standing in, builds
it, writes a `.venv/` and a `uv.lock` into their repo, and puts their dependencies on
the engine's import path: dex would be leaving unreviewed artifacts in a repo it was
asked only to read, and running against a closure it did not pin.

`--warm` materializes that environment and exits without running a command, so the
install can be paid once out of band instead of on the first caller's clock (see
_warm).

The engine version is pinned; the *extras* are chosen at runtime, so one published
release serves every warehouse and the default install stays light. The connector
extra comes from the active connector; two commands need more than a warehouse
client and say so by being run (see _feature_extras). This wrapper is
stdlib-only and runs before the engine is installed, so it resolves the connector
itself (it cannot import the engine) with the same precedence the engine uses:
an explicit --connector flag, then the top-level `connector:` in the
`.dex/config.yml` found by walking up from the run directory to the git root (the
way git and dbt find their project), then DuckDB. The walk-up must mirror the
engine's: if it did not, a run from a subdirectory would install the DuckDB extra
while the engine resolves the project's real connector and then fails for want of
that connector's deps. The guess only picks which extra to install; the full argv
is still forwarded, so the engine stays authoritative for the actual connection
and a wrong guess surfaces as a clean error envelope.

`DEX_CORE_VERSION` is the single line bumped at release time, by
scripts/prepare_release.sh before the tag; nothing else here changes per release.
"""

from __future__ import annotations

import argparse
import json
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path

# Rewritten by scripts/prepare_release.sh to the tagged version. The connector
# extra is deliberately NOT part of this pin: it is chosen at runtime (see
# _resolve_connector), so a release artifact is connector-neutral.
DEX_CORE_VERSION = "1.12.3"

# Connector id -> packaging extra. The engine's connector ids and the pyproject
# extras share names, so this is the identity set today. An unknown or unset
# connector falls back to the light DuckDB on-ramp and lets the installed engine
# emit the canonical error rather than the wrapper guessing wrong.
_KNOWN_CONNECTORS = (
    "duckdb",
    "snowflake",
    "bigquery",
    "databricks",
    "postgres",
    "redshift",
    "clickhouse",
)
_DEFAULT_CONNECTOR = "duckdb"

# The one flag the wrapper answers itself. It is stripped before anything else
# reads the argv, so it never reaches the engine and never lands in the positionals
# that pick the extras.
_WARM_FLAG = "--warm"


def _connector_from_config(config_path: Path) -> str | None:
    """Read the top-level scalar `connector:` from .dex/config.yml, stdlib only.

    This bootstrap script has no YAML dependency and only needs enough to pick the
    right extra to install, so it scans for a single unindented `connector:` key.
    The engine remains the source of truth for the full config; anything richer or
    malformed here just falls through to the DuckDB default.
    """

    try:
        text = config_path.read_text(encoding="utf-8")
    except OSError:
        return None
    for line in text.splitlines():
        if line.startswith("connector:"):  # top-level only; indented keys ignored
            value = line.split(":", 1)[1].split("#", 1)[0].strip().strip("'\"")
            return value or None
    return None


def _find_config(start: Path) -> Path | None:
    """Nearest ancestor `.dex/config.yml` at or above `start`, mirroring the
    engine's resolution: walk up to the enclosing git repo (the ceiling), and
    without one do not walk above `start`. Anchors on the file so a subdirectory
    holding only a `.dex/` cache never shadows the real config higher up."""

    start = start.resolve()
    ceiling = start
    for directory in (start, *start.parents):
        if (directory / ".git").exists():
            ceiling = directory
            break
    for directory in (start, *start.parents):
        candidate = directory / ".dex" / "config.yml"
        if candidate.is_file():
            return candidate
        if directory == ceiling:
            break
    return None


def _resolve_connector(argv: list[str], cwd: Path) -> str:
    """Pick the connector whose extra we install, mirroring the engine's order:
    explicit --connector, then the walked-up .dex/config.yml, then DuckDB."""

    # allow_abbrev=False and parse_known_args so we only peek at these two flags
    # and never consume or reorder the argv that is forwarded to the engine.
    parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
    parser.add_argument("--connector")
    parser.add_argument("--repo-root", default=".")
    known, _ = parser.parse_known_args(argv)

    connector = known.connector
    if connector is None:
        config_path = _find_config(cwd / known.repo_root)
        if config_path is not None:
            connector = _connector_from_config(config_path)
    return connector if connector in _KNOWN_CONNECTORS else _DEFAULT_CONNECTOR


# Every value-taking flag this peek has to consume, so a flag's *value* is never
# mistaken for the group, subcommand, or mode being run. The global connection
# flags, plus the ones `explore semantic` takes, because a metric read as a mode
# picks the wrong extras. `tests/test_skill_wrapper.py` holds this list to the
# real parser, since a flag added there and forgotten here fails silently.
_VALUE_FLAGS = (
    "--connector",
    "--path",
    "--scope",
    "--project",
    "--dataset",
    "--repo-root",
    "--budget",
    "--session-ceiling",
    "--metric",
    "--for-dimension",
    "--search",
    "--group-by",
    "--where",
    "--order-by",
    "--grain",
    "--limit",
)

# The `explore semantic` modes that can render a statement here. Bare
# `explore semantic` lists, so an absent mode is `list`, which renders none on
# either backend and therefore never needs the renderer.
_SEMANTIC_RENDERING_MODES = ("values", "query")


def _positionals(argv: list[str]) -> list[str]:
    """The bare tokens of an invocation: group, subcommand, and whatever follows.

    Flag-position agnostic, which is the point: the value flags above are consumed
    so the group and subcommand are the first two bare tokens wherever the
    connection flags sit."""

    parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
    for flag in _VALUE_FLAGS:
        parser.add_argument(flag)
    _, remaining = parser.parse_known_args(argv)
    return [tok for tok in remaining if not tok.startswith("-")]


def _feature_extras(argv: list[str]) -> list[str]:
    """The extras this invocation needs on top of the connector's.

    Two commands need more than a warehouse client, and both are resolved from the
    command rather than installed always, so a repo that never clusters and never
    queries a semantic layer resolves neither scikit-learn nor MetricFlow.

    `explore semantic` needs the hosted client on any of its subcommands, which is
    an httpx and nothing heavier. It needs MetricFlow only where a statement might
    be rendered *here*: `list` renders none on either backend, and `--api` sends
    the request to dbt Cloud, which renders and executes it there.

    Where the flag is absent the backend is ambient, chosen by `semantic.deployment`
    in a nested config block this bootstrap deliberately does not parse, so a mode
    that could execute either way takes both. That errs toward a heavier install
    and never toward a command that refuses for want of a dependency, which is the
    failure this exists to prevent.
    """

    positionals = _positionals(argv)
    if positionals[:2] == ["explore", "cluster"]:
        return ["cluster"]
    if positionals[:2] != ["explore", "semantic"]:
        return []
    mode = positionals[2] if len(positionals) > 2 else "list"
    if mode in _SEMANTIC_RENDERING_MODES and "--api" not in argv:
        return ["semantic-api", "semantic"]
    return ["semantic-api"]


def _engine_spec(extras: str, skill_dir: Path | None = None) -> list[str]:
    skill_dir = skill_dir or Path(
        os.environ.get("CLAUDE_SKILL_DIR", Path(__file__).resolve().parent.parent)
    )
    local_pkg = (skill_dir / ".." / ".." / "packages" / "dex-core").resolve()
    if local_pkg.is_dir():
        # Resolve the local package WITH the resolved extras (a plain path drops
        # extras). Non-editable is fine: the engine is imported fresh each run.
        return ["--with", f"exmergo-dex-core[{extras}] @ {local_pkg.as_uri()}"]
    return ["--with", f"exmergo-dex-core[{extras}]=={DEX_CORE_VERSION}"]


def _extras(argv: list[str]) -> list[str]:
    """The full extras list for an invocation: the connector's, plus whatever the
    command itself needs.

    Warm-up and a real run both resolve here, on the same argv, which is what stops
    a warmed environment from disagreeing with the one the next command asks for."""

    return [_resolve_connector(argv, Path.cwd()), *_feature_extras(argv)]


def _uv_run(engine_spec: list[str], tail: list[str]) -> list[str]:
    """The single uv invocation this wrapper makes.

    `--no-project` is a correctness flag before it is a fast one: see the module
    docstring for what uv does to the caller's repo without it. The latency follows
    from the same thing, because installing the caller's project is far more work
    than resolving the engine."""

    return ["uv", "run", "--no-project", *engine_spec, *tail]


def _envelope(
    status: str,
    *,
    data: dict[str, object] | None = None,
    errors: list[str] | None = None,
    reason: str | None = None,
) -> str:
    """One JSON line in the engine's envelope shape, built by hand.

    The wrapper prints an envelope for the few things that happen before the engine
    exists to print its own: the missing-uv refusal and the warm-up. It cannot
    import `exmergo_dex_core.envelope` for precisely that reason, so the keys are
    mirrored here and have to stay in step with Envelope and Cost. `cost.paradigm`
    stays null throughout, because nothing here opens a connection and `free_local`
    is a positive claim that the connector in play bills nothing rather than a
    stand-in for "no connector was involved". `reason` is populated only on an
    error, the same rule the engine's Reason enum follows.
    """

    return json.dumps(
        {
            "status": status,
            "data": data or {},
            "cost": {"paradigm": None, "estimate": None, "ceiling": None},
            "warnings": [],
            "diffs": [],
            "errors": errors or [],
            "reason": reason,
        }
    )


def _warm(argv: list[str]) -> int:
    """Materialize the engine environment and exit, without running a command.

    Nearly all of a first command's latency is uv resolving and installing the
    engine's closure. This pays it out of band, at container build, plugin install,
    or a CI setup step, so no interactive caller waits for it.

    The extras come from `_extras` on the argv that remains after `--warm` is
    stripped, so warm-up can never install something a later command contradicts:
    bare `--warm` warms the connector this directory resolves to, `--warm
    --connector snowflake` warms a named one before any project exists, and `--warm
    explore cluster` warms exactly what that command needs. The last form earns its
    keep because a feature extra resolves into an environment of its own, so
    warming the connector alone leaves `explore cluster` and `explore semantic`
    cold.
    """

    extras = _extras(argv)
    spec = _engine_spec(",".join(extras))
    started = time.monotonic()
    # Captured rather than inherited: uv narrates resolution and installation, and
    # stdout here carries one envelope and nothing else.
    completed = subprocess.run(
        _uv_run(spec, ["python", "-c", "import exmergo_dex_core"]),
        capture_output=True,
        text=True,
    )
    elapsed = round(time.monotonic() - started, 3)
    if completed.returncode != 0:
        lines = [line.strip() for line in completed.stderr.splitlines() if line.strip()]
        print(
            _envelope(
                "error",
                errors=[
                    f"warm-up could not install {spec[1]}. uv reported: "
                    + (" ".join(lines[-3:]) or "nothing on stderr")
                ],
                reason="prerequisite",
            )
        )
        return 1
    print(
        _envelope(
            "ok",
            data={
                "engine": spec[1],
                "connector": extras[0],
                "extras": extras,
                "elapsed_seconds": elapsed,
            },
        )
    )
    return 0


def main() -> int:
    argv = sys.argv[1:]
    if shutil.which("uv") is None:
        # The one refusal that happens before the engine exists, which is why the
        # envelope is hand-built. `prerequisite` is the engine's own classification
        # for a missing dependency the user installs and retries (the same one
        # DemoDependencyError and DialectDependencyError carry), so a caller reads
        # this exactly like any other refusal instead of parsing a traceback.
        print(
            _envelope(
                "error",
                errors=[
                    "dex runs its engine through uv, which was not found on "
                    "PATH. Install it with: "
                    "curl -LsSf https://astral.sh/uv/install.sh | sh "
                    "(or `brew install uv`, or `pipx install uv`), then re-run."
                ],
                reason="prerequisite",
            )
        )
        return 1
    # The engine runs in uv's own ephemeral environment, so an inherited
    # VIRTUAL_ENV (e.g. the user's activated venv) is irrelevant to both paths
    # below and only makes uv print a mismatch warning on every call. Drop it.
    os.environ.pop("VIRTUAL_ENV", None)
    if _WARM_FLAG in argv:
        return _warm([arg for arg in argv if arg != _WARM_FLAG])
    # The connector extra is always installed; `explore cluster` and
    # `explore semantic` add what only they need, so the default install stays
    # light for every repo that runs neither.
    cmd = _uv_run(
        _engine_spec(",".join(_extras(argv))),
        ["python", "-m", "exmergo_dex_core", *argv],
    )
    if os.name != "posix":
        # Windows keeps the spawn: exec there hands control back to the shell
        # before the child finishes, and the prompt would interleave with the one
        # envelope a caller is reading off stdout.
        return subprocess.call(cmd)
    # Exec rather than spawn everywhere else: the wrapper has nothing left to do
    # once the engine starts, and replacing the process hands over the terminal,
    # the signals, and the exit code directly instead of relaying them through a
    # parent that is only waiting.
    os.execvp(cmd[0], cmd)
    return 0  # os.execvp does not return; this keeps the signature a plain int


if __name__ == "__main__":
    sys.exit(main())