Claude Cursor GitHub Copilot Skill

maintain

Use this to keep a dbt project and its semantic layer correct as the warehouse and the business change, including a semantic layer that is native Apache Ossie documents rather than dbt. It detects drift on four axes and proposes the fix: schema drift (source columns and tables ad

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download exmergo-dex-skills_maintain-9823c5c.zip · 15 KB
exmergo/dex 25 8 forks Apache-2.0 Updated 1d ago
Part of exmergo/dex — 3 skills

Install

skills CLI npx skills add https://github.com/exmergo/dex/tree/main/skills/maintain
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install exmergo-dex@llmmart
Git git clone https://github.com/exmergo/dex.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole exmergo/dex collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Maintain

Keep the repository correct as the world underneath it moves, on both of its axes: the dbt project and the semantic layer. Maintenance is the recurring half of the loop: warehouses drift, loads half-fail, models go stale, keys stop being unique, and business definitions change. This skill compares a known-good baseline against current reality, classifies what drifted, and proposes the reconciling edit. It is manual and on-demand here; continuous drift detection and automated PRs are the commercial product.

The model: baseline, detect, reconcile

Drift is measured against a baseline (the .dex/snapshot.json fingerprint of the warehouse map and the repository's per-layer definitions). Detection is read-only; only reconcile proposes edits.

The two project layers are fingerprinted independently. The transform layer comes from the dbt project; the semantic layer comes from whichever vendor semantic.vendor names, which may be dbt's own or a native format such as Apache Ossie. A repository with a semantic layer and no dbt project at all still gets a baseline and still runs every free axis: transform_layer comes back null, and the warning that names why no project was fingerprinted is reserved for the case where neither layer answered, since that is the one you could otherwise mistake for a clean read.

Snapshot discipline matters. A snapshot is only as trustworthy as the moment it froze. Take one right after a known-good build (maintain snapshot), and commit .dex/snapshot.json like a lockfile so the whole team diffs against the same reference. Snapshot a state that is already drifted and check will mask the very drift you care about. When you accept a change as the new normal (re-run explore map first, then maintain snapshot); check warns when the baseline looks stale.

On a warehouse past the rank cutoff, use explore map --full before snapshotting. Past 50 objects explore map profiles the top 25 by rank and enters the rest as metadata alone, and the baseline can only compare columns for objects it has columns for. Snapshotting a partial map is still valid, and the envelope reports column_detail_count against dataset_count plus a warning naming what it could not cover, so the gap is visible rather than silently mistaken for a clean bill.

How to drive it

uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <subcommand> [flags]

dex runs its engine through uv, which is a prerequisite and is not installed by Claude Code. If the shell reports uv: command not found, stop and tell the user to install it (curl -LsSf https://astral.sh/uv/install.sh | sh, or brew install uv, or pipx install uv), then re-run. Never fall back to diffing the warehouse against the project by hand instead: the drift axes and the baseline comparison live in the engine, so any other path is guesswork.

The first command in a fresh environment installs the engine, so it can take tens of seconds where later ones take well under a second. --warm pays that install up front and exits without running anything:

uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" --warm

Offer it once at setup. It is not something to run before an ordinary command.

  • maintain snapshot captures or refreshes the baseline. Run it after a clean explore or transform session so later runs have a known-good reference. It pins the current .dex/cache.json (so the grain baseline is the exact-distinct verdicts explore map already computed) plus per-layer fingerprints of the dbt project and of the semantic layer. A native semantic layer contributes its definitions per dataset and per metric, each with a content hash, the relation behind it, the column each field resolves to, its declared keys in the arity they were written, and its relationships with every ordered column pair; whether that side was captured is itself recorded, so a baseline written before it reports the relationship axis as unchecked rather than clean. Without a cache it captures a metadata-only baseline and says so. It also warns when the cache it pinned is thin (objects without column detail) or older than the profile freshness window, because either makes an "accept current state" only partly true.

  • maintain snapshot --project-only is for a project-only refactor, such as moved model files or a dbt project rename. It refreshes transform and semantic fingerprints without opening the warehouse, carrying the previous warehouse evidence and original capture time forward instead. It requires an existing snapshot and refuses connection-target flags.

  • maintain check is the everyday entry point: it sweeps every axis and returns a report ranked by blast radius. Read-only.

  • maintain schema [<objects>] detects structural drift: source columns and tables added, dropped, retyped, or renamed; nullability changes; declared sources the warehouse no longer honors.

  • maintain volume [<objects>] detects freshness drift: row counts that collapsed, spiked, or went to zero. This is the "is the data still flowing correctly?" axis, distinct from "did the shape change?".

  • maintain grain [<objects>] detects grain drift: a key that now has duplicates, a changed row-per-entity cardinality, or an increased join fanout. It also re-verifies the grains the repository declares, which measurement on its own can miss: a dbt model-level unique_combination_of_columns, and a semantic layer's own key declarations. A multi-column declaration is measured as one complete composite and never one column at a time. Uses aggregates, never raw rows.

    A native semantic layer's keys reach this axis and nothing else reaches it for them, since such a layer is never the transformation project. They go through the identical billed handshake on a metered warehouse: nothing here is cheaper or less gated because the declaration came from a document rather than from dbt.

    Two findings come out of the uniqueness checks and the difference is the baseline. key_lost_uniqueness is a key that was proven unique and is not any more: something changed in the data. declared_grain_not_unique is a declared combination that does not hold, and nothing changed at all: the project asserts a grain the data never had, so the fix is to the declaration (widen it, dedup upstream, or drop the claim) rather than to the data.

  • maintain semantic [<objects>] detects definition drift: definitions that changed, were added, or were removed against the baseline; a source relation that is gone; a dimension, entity, measure, or declared key naming a column that is gone; a relationship whose endpoint or column pairs no longer resolve, which is high because a join nothing can resolve is a broken layer rather than a stale one; and categorical dimensions whose set of values widened or narrowed underneath their metrics.

    Read unavailable on the layer before hunting for an element kind. A native Ossie layer has no measures and no entities at all, so their absence is the format rather than drift. Its cardinality half also never fires, because that check needs a semantic model naming a transformation model and Ossie names none: on such a layer this command is free and offers no scan.

  • maintain verify [<selector>] answers a different question from every command above it: not "what changed since the baseline" but "is this project right now", and it needs no baseline at all, so it works on a project that was never correct and on one somebody else just built. Two classes of finding. Build status: nodes that failed, nodes skipped because a parent failed (naming the one that actually failed), nodes that warned rather than failed, and models the project declares that built no relation. A warning ranks low deliberately: a project that runs relationship tests at severity: warn over documented gaps has warnings by design, so this is a list to compare against last run's rather than a defect on its own. What it must not be is missing, which is what leaves a caller counting statuses in a run's raw node list to find out which tests warned. Row population: row_loss where a model holds materially fewer rows than its driving parent (the relation in its FROM clause, followed through the CTE chain, as distinct from anything it joins) and nothing in its SQL accounts for the shortfall, and row_fanout where it holds materially more, each naming the join and its key and stating both counts.

    Row population is conservative on purpose. A model with a WHERE, GROUP BY, DISTINCT, QUALIFY, LIMIT, a semi or anti join, or a set operation was written to hold a different number of rows and is never reported for loss; an incremental model is skipped outright. So a quiet answer here is weaker evidence than a finding, and the warnings say which models could not be lined up at all.

    A project that does not compile is reported first and suppresses everything else, since a manifest a broken project could not have produced is not evidence. Read data.suppressed before reading an empty data.findings as a clean bill of health.

    The same sweep runs from the other side of the loop, as transform build --verify, scoped to the nodes one build touched. Use that when the question is whether a change you just made is right; use this one when the question is the whole project, or when the build was somebody else's.

  • maintain reconcile [<class>] proposes the dbt edits that bring the project back in sync, as reviewable diffs. Optionally scope it to one class (schema, volume, grain, or semantic). It composes every layer's declarations first, so a grain the semantic layer already declares is not proposed as though nothing declared it. Where there is no editable dbt project it has nothing to author: every proposal is advisory and no plan is stored. Authoring into a native semantic layer is semantic ossie in the transform skill, never this command.

The usual flow: check to triage, a focused detector to understand one axis in depth, then reconcile to get the proposed fix. With no baseline, or on a project whose numbers were never right, start at verify instead: it is the one command here that does not need a snapshot, and it answers "is this right" rather than "what moved".

Per-axis cost: what is free and what scans

Detection is read-only, but read-only is not the same as free on a metered connector (BigQuery, Snowflake, Databricks, Postgres, Redshift, ClickHouse). The axes split:

  • Schema, volume, and the reference/definition half of semantic are free everywhere: they read metadata and the snapshot, and run immediately.
  • Grain and the dimension-cardinality half of semantic scan the warehouse, so on a metered connector they run the two-step handshake. Asked for directly, maintain grain returns needs_confirmation with an estimate in cost.estimate (and a per-table breakdown). Surface it to the user in human units, get an explicit budget, and re-issue the same command with --confirm --budget <magnitude> in the paradigm's unit (bytes on BigQuery, warehouse-seconds on Snowflake and Databricks, compute-seconds on Redshift, database-seconds on Postgres and ClickHouse). Never invent a budget the user did not agree to, and never retry with a raised budget on an over-ceiling refusal without asking. An over-ceiling refusal carries a calibration line from .dex/spend.jsonl (what this connector's recent commands billed as a fraction of estimate, or a sentence saying there is too little history to say): relay it, and note that the ceiling binds on the estimate, so a budget set at that fraction of the estimate is refused again.
  • verify is free except for the counts a warehouse does not keep. Its build-status findings read artifacts on disk, and its row counts come from object metadata. A view has no stored row count anywhere, and a view is dbt's default materialization, so on a metered connector those counts are batched into one aggregate-only statement, priced, and returned in data.offer beside findings that are already final. On DuckDB there is no gate, so every count is measured rather than estimated and the findings come back exact.
  • check, semantic and verify answer first and offer second. Their free axes complete on every call, so the envelope is ok and the findings in it are final. The price of the scanning axes sits in data.offer, with axes naming what it would add; data.axes_run names what already ran. Confirming is a choice, not a required next step: quote the estimate, say which axes are still dark, and let the user decide. A triage pass that stops at the free axes is a complete piece of work, not an abandoned one.
  • Read warnings on these responses, always. They carry the reasons the baseline may no longer describe the warehouse (a cache newer than the snapshot, a baseline pinned from a stale cache), which bound every finding above them. A stale baseline is often the most important line in the response and it is never in findings.

On DuckDB everything is free and local, so nothing prompts.

A needs_confirmation envelope carrying suggested_session_ceiling is the project's one-time ask for a cumulative daily cap, separate from the per-command --budget. Surface it, get the user's answer, and add --session-ceiling <value> or --no-session-ceiling to the same re-issue; it is written to .dex/config.yml once and never asked again. Never answer it for them.

Reconcile proposals are mechanical or advisory

Reconcile tags every proposal by kind, because the fix differs sharply by axis:

  • mechanical: schema drift reconciles in one of two shapes. On a dex-scaffolded staging model it re-scaffolds the model from the drifted source; on a project format that places a declaration but authors no staging model, it edits the drifted columns into that declaration and says so. High-confidence, but still a reviewable diff: read it for hand-written logic the scaffold cannot know about.
  • advisory: grain, volume, and semantic drift are decisions, not auto-fixes (dex cannot dedup your warehouse or decide whether a new 'refunded' status belongs in a metric). The proposal is the decision surfaced, at most backed by a test edit that makes the break visible in builds. It declines that test where the test would be wrong: if your model declares a composite grain covering the column, no column-level unique is proposed on it, and the warning names the combination so you can tell "re-baseline, this is still the grain" from "something relied on that column alone".

A type change is advisory on every format. Nothing dex writes declares a type, and the type it holds is the connector's own spelling rather than a canonical one (Snowflake reports NUMBER(38,0) and NUMBER(10,2) both as FIXED), so the proposal names both spellings and the edit is yours. One consequence to know: on ClickHouse nullability is part of the type, so a column that starts accepting nulls is reported as a retype and gets advice where other connectors get an edit.

When reconcile produces edits it stores them as a plan and prints a plan_id. Apply them with transform apply <plan-id> (the one apply door): a human edit made since detection surfaces as a conflict, never a silent overwrite.

Guardrails (enforced in the engine, not here)

  • Read-only against data. Schema, volume, and semantic references are computed from metadata and the snapshot; grain and dimension-cardinality use aggregates only. Raw rows and dimension values never cross the envelope.
  • Propose, don't impose. Reconciliation is always a reviewable diff, applied through transform apply. Human edits to the project and to the semantic layer are authoritative; on conflict the engine surfaces the divergence and asks rather than overwriting.
  • The repository is the source of truth, on both axes; the .dex/ snapshot is a non-canonical fingerprint used only to detect change.
  • The cost guard behind the scanning axes, in full, in the engine repository: references/cost-controls.md.
Files (dex)
  • evals
    • evals.json 4.2 KB
      {
        "skill_name": "maintain",
        "triggering": {
          "positive": [
            "What changed in the warehouse since I last looked?",
            "Did anything drift in my dbt project?",
            "The orders source gained a column and dropped another; is my project still in sync?",
            "My stg_customers primary key has duplicates now, what broke?",
            "The business changed the revenue metric definition; does my semantic model still match?",
            "Reconcile my models with the current source schema.",
            "Which of my models are stale after the upstream change?",
            "The row count on orders dropped overnight; did a load half-fail?",
            "The unique test on order_id started failing yesterday and nobody changed the model. Figure out what happened.",
            "Our revenue dashboard dropped 40% overnight but the code has not changed. What is going on upstream?",
            "Did my Ossie semantic layer drift from the warehouse?",
            "One of my semantic layer's relationships stopped resolving. What broke?"
          ],
          "negative": [
            "What's in this warehouse and which tables matter?",
            "Profile the events table and flag PII.",
            "Set up a dbt project in this repo.",
            "Build a staging model for raw orders.",
            "Define a revenue metric on top of fct_orders.",
            "I just rewrote stg_orders and now the not_null test fails. Fix my SQL.",
            "Add a revenue metric to my Ossie semantic model."
          ]
        },
        "evals": [
          {
            "id": 0,
            "prompt": "The upstream orders table gained a column and dropped another and changed a type. Tell me what drifted and how to fix it.",
            "expected_output": "A schema-drift report from diffing the current warehouse against the .dex/ snapshot, followed by proposed reconciliation edits as reviewable diffs; nothing applied.",
            "files": [],
            "assertions": [
              "Drives the maintain skill (check or schema, then reconcile) and reads the JSON envelope; does not invent results",
              "Structural drift is computed from metadata and the snapshot, not by scanning source rows",
              "The report distinguishes added, dropped, and type-changed columns and the models they affect",
              "Proposed reconciliation edits are reviewable diffs, never silently applied",
              "No raw rows and no credentials appear in the output (clean envelope)"
            ]
          },
          {
            "id": 1,
            "prompt": "My stg_customers model assumes customer_id is unique, but I think that broke. Check the grain.",
            "expected_output": "A grain-drift finding that the declared unique key now has duplicates (or a changed cardinality), derived from aggregates, with a proposed fix as a diff.",
            "files": [],
            "assertions": [
              "Grain drift is established from SQL aggregates (uniqueness, cardinality), never raw rows",
              "The finding names the key that lost uniqueness or the changed row-per-entity cardinality",
              "A key that was proven unique and lapsed is reported differently from a declared grain that never held",
              "On a billed connector, surfaces the dry-run cost estimate and re-issues with --confirm and a user-agreed budget rather than inventing one; on DuckDB it runs directly",
              "Any proposed fix is a reviewable diff; nothing is applied silently",
              "No column-level unique test is proposed against a column covered by a declared composite grain"
            ]
          },
          {
            "id": 2,
            "prompt": "Finance changed how revenue is defined and added a new order_status value. Does my semantic model still match, and reconcile it if not.",
            "expected_output": "A semantic-drift finding (definition change and new categorical value) against the baseline, with a proposed diff to bring the dbt semantic model back in sync; conflicts with hand-written intent surfaced and asked about.",
            "files": [],
            "assertions": [
              "Definition drift is compared against the .dex/ snapshot baseline",
              "New categorical dimension values and changed metric/measure definitions are surfaced",
              "Human dbt edits are authoritative by construction; conflicts surface as a diff and ask, never a silent overwrite",
              "The proposed reconciliation is valid MetricFlow YAML in the existing semantic model"
            ]
          }
        ]
      }
      
  • scripts
    • run.py 16.1 KB
      # /// script
      # requires-python = ">=3.11"
      # dependencies = []
      # ///
      """Thin PEP 723 wrapper that drives dex-core via the command contract.
      
      The skill never re-implements logic. It forwards its arguments to the pinned
      `dex-core` engine and lets the engine print the sanitized JSON envelope. Run it
      with `uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <dex
      subcommand> ...`.
      
      `uv` is a hard prerequisite: it is what installs and runs the engine. When it is
      absent this wrapper refuses with an error envelope naming the install command,
      rather than letting the exec fail with a traceback. Invoked through `uv run` the
      shell fails first (`uv: command not found`), which is why each SKILL.md also tells
      the agent what that message means.
      
      Two execution modes, chosen automatically:
        - Monorepo checkout (this repo): `packages/dex-core` is found above the skill,
          so the engine runs from an editable local install. This is what makes the
          wrapper work before the package is published.
        - Installed plugin: no local package is present, so the pinned PyPI release is
          installed hermetically by uv.
      
      That environment is uv's own, and `--no-project` is what keeps it that way. Without
      it uv discovers whatever Python project the caller happens to be standing in, builds
      it, writes a `.venv/` and a `uv.lock` into their repo, and puts their dependencies on
      the engine's import path: dex would be leaving unreviewed artifacts in a repo it was
      asked only to read, and running against a closure it did not pin.
      
      `--warm` materializes that environment and exits without running a command, so the
      install can be paid once out of band instead of on the first caller's clock (see
      _warm).
      
      The engine version is pinned; the *extras* are chosen at runtime, so one published
      release serves every warehouse and the default install stays light. The connector
      extra comes from the active connector; two commands need more than a warehouse
      client and say so by being run (see _feature_extras). This wrapper is
      stdlib-only and runs before the engine is installed, so it resolves the connector
      itself (it cannot import the engine) with the same precedence the engine uses:
      an explicit --connector flag, then the top-level `connector:` in the
      `.dex/config.yml` found by walking up from the run directory to the git root (the
      way git and dbt find their project), then DuckDB. The walk-up must mirror the
      engine's: if it did not, a run from a subdirectory would install the DuckDB extra
      while the engine resolves the project's real connector and then fails for want of
      that connector's deps. The guess only picks which extra to install; the full argv
      is still forwarded, so the engine stays authoritative for the actual connection
      and a wrong guess surfaces as a clean error envelope.
      
      `DEX_CORE_VERSION` is the single line bumped at release time, by
      scripts/prepare_release.sh before the tag; nothing else here changes per release.
      """
      
      from __future__ import annotations
      
      import argparse
      import json
      import os
      import shutil
      import subprocess
      import sys
      import time
      from pathlib import Path
      
      # Rewritten by scripts/prepare_release.sh to the tagged version. The connector
      # extra is deliberately NOT part of this pin: it is chosen at runtime (see
      # _resolve_connector), so a release artifact is connector-neutral.
      DEX_CORE_VERSION = "1.12.3"
      
      # Connector id -> packaging extra. The engine's connector ids and the pyproject
      # extras share names, so this is the identity set today. An unknown or unset
      # connector falls back to the light DuckDB on-ramp and lets the installed engine
      # emit the canonical error rather than the wrapper guessing wrong.
      _KNOWN_CONNECTORS = (
          "duckdb",
          "snowflake",
          "bigquery",
          "databricks",
          "postgres",
          "redshift",
          "clickhouse",
      )
      _DEFAULT_CONNECTOR = "duckdb"
      
      # The one flag the wrapper answers itself. It is stripped before anything else
      # reads the argv, so it never reaches the engine and never lands in the positionals
      # that pick the extras.
      _WARM_FLAG = "--warm"
      
      
      def _connector_from_config(config_path: Path) -> str | None:
          """Read the top-level scalar `connector:` from .dex/config.yml, stdlib only.
      
          This bootstrap script has no YAML dependency and only needs enough to pick the
          right extra to install, so it scans for a single unindented `connector:` key.
          The engine remains the source of truth for the full config; anything richer or
          malformed here just falls through to the DuckDB default.
          """
      
          try:
              text = config_path.read_text(encoding="utf-8")
          except OSError:
              return None
          for line in text.splitlines():
              if line.startswith("connector:"):  # top-level only; indented keys ignored
                  value = line.split(":", 1)[1].split("#", 1)[0].strip().strip("'\"")
                  return value or None
          return None
      
      
      def _find_config(start: Path) -> Path | None:
          """Nearest ancestor `.dex/config.yml` at or above `start`, mirroring the
          engine's resolution: walk up to the enclosing git repo (the ceiling), and
          without one do not walk above `start`. Anchors on the file so a subdirectory
          holding only a `.dex/` cache never shadows the real config higher up."""
      
          start = start.resolve()
          ceiling = start
          for directory in (start, *start.parents):
              if (directory / ".git").exists():
                  ceiling = directory
                  break
          for directory in (start, *start.parents):
              candidate = directory / ".dex" / "config.yml"
              if candidate.is_file():
                  return candidate
              if directory == ceiling:
                  break
          return None
      
      
      def _resolve_connector(argv: list[str], cwd: Path) -> str:
          """Pick the connector whose extra we install, mirroring the engine's order:
          explicit --connector, then the walked-up .dex/config.yml, then DuckDB."""
      
          # allow_abbrev=False and parse_known_args so we only peek at these two flags
          # and never consume or reorder the argv that is forwarded to the engine.
          parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
          parser.add_argument("--connector")
          parser.add_argument("--repo-root", default=".")
          known, _ = parser.parse_known_args(argv)
      
          connector = known.connector
          if connector is None:
              config_path = _find_config(cwd / known.repo_root)
              if config_path is not None:
                  connector = _connector_from_config(config_path)
          return connector if connector in _KNOWN_CONNECTORS else _DEFAULT_CONNECTOR
      
      
      # Every value-taking flag this peek has to consume, so a flag's *value* is never
      # mistaken for the group, subcommand, or mode being run. The global connection
      # flags, plus the ones `explore semantic` takes, because a metric read as a mode
      # picks the wrong extras. `tests/test_skill_wrapper.py` holds this list to the
      # real parser, since a flag added there and forgotten here fails silently.
      _VALUE_FLAGS = (
          "--connector",
          "--path",
          "--scope",
          "--project",
          "--dataset",
          "--repo-root",
          "--budget",
          "--session-ceiling",
          "--metric",
          "--for-dimension",
          "--search",
          "--group-by",
          "--where",
          "--order-by",
          "--grain",
          "--limit",
      )
      
      # The `explore semantic` modes that can render a statement here. Bare
      # `explore semantic` lists, so an absent mode is `list`, which renders none on
      # either backend and therefore never needs the renderer.
      _SEMANTIC_RENDERING_MODES = ("values", "query")
      
      
      def _positionals(argv: list[str]) -> list[str]:
          """The bare tokens of an invocation: group, subcommand, and whatever follows.
      
          Flag-position agnostic, which is the point: the value flags above are consumed
          so the group and subcommand are the first two bare tokens wherever the
          connection flags sit."""
      
          parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
          for flag in _VALUE_FLAGS:
              parser.add_argument(flag)
          _, remaining = parser.parse_known_args(argv)
          return [tok for tok in remaining if not tok.startswith("-")]
      
      
      def _feature_extras(argv: list[str]) -> list[str]:
          """The extras this invocation needs on top of the connector's.
      
          Two commands need more than a warehouse client, and both are resolved from the
          command rather than installed always, so a repo that never clusters and never
          queries a semantic layer resolves neither scikit-learn nor MetricFlow.
      
          `explore semantic` needs the hosted client on any of its subcommands, which is
          an httpx and nothing heavier. It needs MetricFlow only where a statement might
          be rendered *here*: `list` renders none on either backend, and `--api` sends
          the request to dbt Cloud, which renders and executes it there.
      
          Where the flag is absent the backend is ambient, chosen by `semantic.deployment`
          in a nested config block this bootstrap deliberately does not parse, so a mode
          that could execute either way takes both. That errs toward a heavier install
          and never toward a command that refuses for want of a dependency, which is the
          failure this exists to prevent.
          """
      
          positionals = _positionals(argv)
          if positionals[:2] == ["explore", "cluster"]:
              return ["cluster"]
          if positionals[:2] != ["explore", "semantic"]:
              return []
          mode = positionals[2] if len(positionals) > 2 else "list"
          if mode in _SEMANTIC_RENDERING_MODES and "--api" not in argv:
              return ["semantic-api", "semantic"]
          return ["semantic-api"]
      
      
      def _engine_spec(extras: str, skill_dir: Path | None = None) -> list[str]:
          skill_dir = skill_dir or Path(
              os.environ.get("CLAUDE_SKILL_DIR", Path(__file__).resolve().parent.parent)
          )
          local_pkg = (skill_dir / ".." / ".." / "packages" / "dex-core").resolve()
          if local_pkg.is_dir():
              # Resolve the local package WITH the resolved extras (a plain path drops
              # extras). Non-editable is fine: the engine is imported fresh each run.
              return ["--with", f"exmergo-dex-core[{extras}] @ {local_pkg.as_uri()}"]
          return ["--with", f"exmergo-dex-core[{extras}]=={DEX_CORE_VERSION}"]
      
      
      def _extras(argv: list[str]) -> list[str]:
          """The full extras list for an invocation: the connector's, plus whatever the
          command itself needs.
      
          Warm-up and a real run both resolve here, on the same argv, which is what stops
          a warmed environment from disagreeing with the one the next command asks for."""
      
          return [_resolve_connector(argv, Path.cwd()), *_feature_extras(argv)]
      
      
      def _uv_run(engine_spec: list[str], tail: list[str]) -> list[str]:
          """The single uv invocation this wrapper makes.
      
          `--no-project` is a correctness flag before it is a fast one: see the module
          docstring for what uv does to the caller's repo without it. The latency follows
          from the same thing, because installing the caller's project is far more work
          than resolving the engine."""
      
          return ["uv", "run", "--no-project", *engine_spec, *tail]
      
      
      def _envelope(
          status: str,
          *,
          data: dict[str, object] | None = None,
          errors: list[str] | None = None,
          reason: str | None = None,
      ) -> str:
          """One JSON line in the engine's envelope shape, built by hand.
      
          The wrapper prints an envelope for the few things that happen before the engine
          exists to print its own: the missing-uv refusal and the warm-up. It cannot
          import `exmergo_dex_core.envelope` for precisely that reason, so the keys are
          mirrored here and have to stay in step with Envelope and Cost. `cost.paradigm`
          stays null throughout, because nothing here opens a connection and `free_local`
          is a positive claim that the connector in play bills nothing rather than a
          stand-in for "no connector was involved". `reason` is populated only on an
          error, the same rule the engine's Reason enum follows.
          """
      
          return json.dumps(
              {
                  "status": status,
                  "data": data or {},
                  "cost": {"paradigm": None, "estimate": None, "ceiling": None},
                  "warnings": [],
                  "diffs": [],
                  "errors": errors or [],
                  "reason": reason,
              }
          )
      
      
      def _warm(argv: list[str]) -> int:
          """Materialize the engine environment and exit, without running a command.
      
          Nearly all of a first command's latency is uv resolving and installing the
          engine's closure. This pays it out of band, at container build, plugin install,
          or a CI setup step, so no interactive caller waits for it.
      
          The extras come from `_extras` on the argv that remains after `--warm` is
          stripped, so warm-up can never install something a later command contradicts:
          bare `--warm` warms the connector this directory resolves to, `--warm
          --connector snowflake` warms a named one before any project exists, and `--warm
          explore cluster` warms exactly what that command needs. The last form earns its
          keep because a feature extra resolves into an environment of its own, so
          warming the connector alone leaves `explore cluster` and `explore semantic`
          cold.
          """
      
          extras = _extras(argv)
          spec = _engine_spec(",".join(extras))
          started = time.monotonic()
          # Captured rather than inherited: uv narrates resolution and installation, and
          # stdout here carries one envelope and nothing else.
          completed = subprocess.run(
              _uv_run(spec, ["python", "-c", "import exmergo_dex_core"]),
              capture_output=True,
              text=True,
          )
          elapsed = round(time.monotonic() - started, 3)
          if completed.returncode != 0:
              lines = [line.strip() for line in completed.stderr.splitlines() if line.strip()]
              print(
                  _envelope(
                      "error",
                      errors=[
                          f"warm-up could not install {spec[1]}. uv reported: "
                          + (" ".join(lines[-3:]) or "nothing on stderr")
                      ],
                      reason="prerequisite",
                  )
              )
              return 1
          print(
              _envelope(
                  "ok",
                  data={
                      "engine": spec[1],
                      "connector": extras[0],
                      "extras": extras,
                      "elapsed_seconds": elapsed,
                  },
              )
          )
          return 0
      
      
      def main() -> int:
          argv = sys.argv[1:]
          if shutil.which("uv") is None:
              # The one refusal that happens before the engine exists, which is why the
              # envelope is hand-built. `prerequisite` is the engine's own classification
              # for a missing dependency the user installs and retries (the same one
              # DemoDependencyError and DialectDependencyError carry), so a caller reads
              # this exactly like any other refusal instead of parsing a traceback.
              print(
                  _envelope(
                      "error",
                      errors=[
                          "dex runs its engine through uv, which was not found on "
                          "PATH. Install it with: "
                          "curl -LsSf https://astral.sh/uv/install.sh | sh "
                          "(or `brew install uv`, or `pipx install uv`), then re-run."
                      ],
                      reason="prerequisite",
                  )
              )
              return 1
          # The engine runs in uv's own ephemeral environment, so an inherited
          # VIRTUAL_ENV (e.g. the user's activated venv) is irrelevant to both paths
          # below and only makes uv print a mismatch warning on every call. Drop it.
          os.environ.pop("VIRTUAL_ENV", None)
          if _WARM_FLAG in argv:
              return _warm([arg for arg in argv if arg != _WARM_FLAG])
          # The connector extra is always installed; `explore cluster` and
          # `explore semantic` add what only they need, so the default install stays
          # light for every repo that runs neither.
          cmd = _uv_run(
              _engine_spec(",".join(_extras(argv))),
              ["python", "-m", "exmergo_dex_core", *argv],
          )
          if os.name != "posix":
              # Windows keeps the spawn: exec there hands control back to the shell
              # before the child finishes, and the prompt would interleave with the one
              # envelope a caller is reading off stdout.
              return subprocess.call(cmd)
          # Exec rather than spawn everywhere else: the wrapper has nothing left to do
          # once the engine starts, and replacing the process hands over the terminal,
          # the signals, and the exit code directly instead of relaying them through a
          # parent that is only waiting.
          os.execvp(cmd[0], cmd)
          return 0  # os.execvp does not return; this keeps the signature a plain int
      
      
      if __name__ == "__main__":
          sys.exit(main())
      
  • SKILL.md 17.7 KB
    ---
    name: maintain
    description: 'Use this to keep a dbt project and its semantic layer correct as the warehouse and the business change, including a semantic layer that is native Apache Ossie documents rather than dbt. It detects drift on four axes and proposes the fix: schema drift (source columns and tables added, dropped, retyped, or renamed), volume drift (a row count that collapsed, a table that emptied, a load that half-failed), grain drift (a key that lost uniqueness, a changed row-per-entity cardinality, an increased join fanout), and semantic drift (a metric, measure, dimension, or entity definition that no longer matches, new categorical values, dangling semantic references). Reach for this when something that used to work has started failing or producing different numbers and the cause is more likely upstream than in the code you just wrote: a test that began failing with no code change, a dashboard whose numbers moved, a model that is suddenly empty or duplicated. Trigger it for requests like "what changed in the warehouse", "did anything drift", "is my dbt project still in sync", "my primary key has duplicates now", "the row count dropped", "did the load run", "the data stopped flowing", "the revenue metric definition changed", "reconcile my models with the source schema", "which models are stale", "did my Ossie semantic layer drift", or "is this relationship still valid". It reads the .dex/ snapshot and proposes reviewable diffs; it never overwrites hand-written work. To author new models or metrics from scratch, use transform. To learn an unfamiliar warehouse for the first time, use explore.'
    ---
    
    # Maintain
    
    Keep the repository correct as the world underneath it moves, on both of its
    axes: the dbt project and the semantic layer. Maintenance is the recurring half
    of the loop: warehouses drift, loads half-fail, models go stale, keys stop being
    unique, and business definitions change. This skill compares a known-good
    baseline against current reality, classifies what drifted, and proposes the
    reconciling edit. It is manual and on-demand here; continuous drift detection and
    automated PRs are the commercial product.
    
    ## The model: baseline, detect, reconcile
    
    Drift is measured against a **baseline** (the `.dex/snapshot.json` fingerprint of
    the warehouse map and the repository's per-layer definitions). Detection is
    read-only; only reconcile proposes edits.
    
    **The two project layers are fingerprinted independently.** The transform layer
    comes from the dbt project; the semantic layer comes from whichever vendor
    `semantic.vendor` names, which may be dbt's own or a native format such as
    Apache Ossie. A repository with a semantic layer and no dbt project at all still
    gets a baseline and still runs every free axis: `transform_layer` comes back
    null, and the warning that names why no project was fingerprinted is reserved for
    the case where neither layer answered, since that is the one you could otherwise
    mistake for a clean read.
    
    **Snapshot discipline matters.** A snapshot is only as trustworthy as the moment
    it froze. Take one right after a known-good build (`maintain snapshot`), and
    **commit `.dex/snapshot.json` like a lockfile** so the whole team diffs against
    the same reference. Snapshot a state that is already drifted and `check` will
    mask the very drift you care about. When you accept a change as the new normal
    (re-run `explore map` first, then `maintain snapshot`); `check` warns when the
    baseline looks stale.
    
    **On a warehouse past the rank cutoff, use `explore map --full` before
    snapshotting.** Past 50 objects `explore map` profiles the top 25 by rank and
    enters the rest as metadata alone, and the baseline can only compare columns for
    objects it has columns for. Snapshotting a partial map is still valid, and the
    envelope reports `column_detail_count` against `dataset_count` plus a warning
    naming what it could not cover, so the gap is visible rather than silently
    mistaken for a clean bill.
    
    ## How to drive it
    
    ```bash
    uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" <subcommand> [flags]
    ```
    
    dex runs its engine through `uv`, which is a prerequisite and is not installed by
    Claude Code. If the shell reports `uv: command not found`, stop and tell the user
    to install it (`curl -LsSf https://astral.sh/uv/install.sh | sh`, or
    `brew install uv`, or `pipx install uv`), then re-run. Never fall back to diffing
    the warehouse against the project by hand instead: the drift axes and the baseline
    comparison live in the engine, so any other path is guesswork.
    
    The first command in a fresh environment installs the engine, so it can take tens
    of seconds where later ones take well under a second. `--warm` pays that install up
    front and exits without running anything:
    
    ```bash
    uv run --no-project --script "${CLAUDE_SKILL_DIR}/scripts/run.py" --warm
    ```
    
    Offer it once at setup. It is not something to run before an ordinary command.
    
    - `maintain snapshot` captures or refreshes the baseline. Run it after a clean
      explore or transform session so later runs have a known-good reference. It pins
      the current `.dex/cache.json` (so the grain baseline is the exact-distinct
      verdicts `explore map` already computed) plus per-layer fingerprints of the dbt
      project and of the semantic layer. A native semantic layer contributes its
      definitions per dataset and per metric, each with a content hash, the relation
      behind it, the column each field resolves to, its declared keys in the arity
      they were written, and its relationships with every ordered column pair;
      whether that side was captured is itself recorded, so a baseline written before
      it reports the relationship axis as unchecked rather than clean.
      Without a cache it captures a metadata-only baseline and says so. It
      also warns when the cache it pinned is thin (objects without column detail) or
      older than the profile freshness window, because either makes an "accept
      current state" only partly true.
    - `maintain snapshot --project-only` is for a project-only refactor, such as
      moved model files or a dbt project rename. It refreshes transform and semantic
      fingerprints without opening the warehouse, carrying the previous warehouse
      evidence and original capture time forward instead. It requires an existing
      snapshot and refuses connection-target flags.
    - `maintain check` is the everyday entry point: it sweeps every axis and returns
      a report ranked by blast radius. Read-only.
    - `maintain schema [<objects>]` detects **structural drift**: source columns and
      tables added, dropped, retyped, or renamed; nullability changes; declared
      sources the warehouse no longer honors.
    - `maintain volume [<objects>]` detects **freshness drift**: row counts that
      collapsed, spiked, or went to zero. This is the "is the data still flowing
      correctly?" axis, distinct from "did the shape change?".
    - `maintain grain [<objects>]` detects **grain drift**: a key that now has
      duplicates, a changed row-per-entity cardinality, or an increased join fanout.
      It also re-verifies the grains the repository *declares*, which measurement on
      its own can miss: a dbt model-level `unique_combination_of_columns`, and a
      semantic layer's own key declarations. A multi-column declaration is measured
      as one complete composite and never one column at a time. Uses aggregates,
      never raw rows.
    
      A native semantic layer's keys reach this axis and nothing else reaches it for
      them, since such a layer is never the transformation project. They go through
      the identical billed handshake on a metered warehouse: nothing here is cheaper
      or less gated because the declaration came from a document rather than from
      dbt.
    
      Two findings come out of the uniqueness checks and the difference is the
      baseline. `key_lost_uniqueness` is a key that was proven unique and is not any
      more: something changed in the data. `declared_grain_not_unique` is a declared
      combination that does not hold, and nothing changed at all: the project asserts
      a grain the data never had, so the fix is to the declaration (widen it, dedup
      upstream, or drop the claim) rather than to the data.
    - `maintain semantic [<objects>]` detects **definition drift**: definitions that
      changed, were added, or were removed against the baseline; a source relation
      that is gone; a dimension, entity, measure, or declared key naming a column
      that is gone; a relationship whose endpoint or column pairs no longer resolve,
      which is `high` because a join nothing can resolve is a broken layer rather
      than a stale one; and categorical dimensions whose set of values widened or
      narrowed underneath their metrics.
    
      Read `unavailable` on the layer before hunting for an element kind. A native
      Ossie layer has no measures and no entities at all, so their absence is the
      format rather than drift. Its cardinality half also never fires, because that
      check needs a semantic model naming a transformation model and Ossie names
      none: on such a layer this command is free and offers no scan.
    - `maintain verify [<selector>]` answers a different question from every command
      above it: not "what changed since the baseline" but **"is this project right
      now"**, and it needs no baseline at all, so it works on a project that was
      never correct and on one somebody else just built. Two classes of finding.
      Build status: nodes that failed, nodes skipped because a parent failed (naming
      the one that actually failed), nodes that warned rather than failed, and models
      the project declares that built no relation. A warning ranks low deliberately:
      a project that runs relationship tests at `severity: warn` over documented gaps
      has warnings by design, so this is a list to compare against last run's rather
      than a defect on its own. What it must not be is missing, which is what leaves
      a caller counting statuses in a run's raw node list to find out which tests
      warned. Row population: `row_loss` where a model holds materially fewer rows
      than its **driving parent** (the relation in its FROM clause, followed through
      the CTE chain, as distinct from anything it joins) and nothing in its SQL
      accounts for the shortfall, and `row_fanout` where it holds materially more,
      each naming the join and its key and stating both counts.
    
      Row population is conservative on purpose. A model with a `WHERE`, `GROUP BY`,
      `DISTINCT`, `QUALIFY`, `LIMIT`, a semi or anti join, or a set operation was
      written to hold a different number of rows and is never reported for loss; an
      incremental model is skipped outright. So a quiet answer here is weaker
      evidence than a finding, and the `warnings` say which models could not be
      lined up at all.
    
      A project that does not compile is reported first and suppresses everything
      else, since a manifest a broken project could not have produced is not
      evidence. Read `data.suppressed` before reading an empty `data.findings` as a
      clean bill of health.
    
      The same sweep runs from the other side of the loop, as
      `transform build --verify`, scoped to the nodes one build touched. Use that
      when the question is whether a change you just made is right; use this one
      when the question is the whole project, or when the build was somebody
      else's.
    - `maintain reconcile [<class>]` proposes the dbt edits that bring the project
      back in sync, as reviewable diffs. Optionally scope it to one class (`schema`,
      `volume`, `grain`, or `semantic`). It composes every layer's declarations
      first, so a grain the semantic layer already declares is not proposed as though
      nothing declared it. Where there is no editable dbt project it has nothing to
      author: every proposal is advisory and no plan is stored. Authoring into a
      native semantic layer is `semantic ossie` in the transform skill, never this
      command.
    
    The usual flow: `check` to triage, a focused detector to understand one axis in
    depth, then `reconcile` to get the proposed fix. With no baseline, or on a
    project whose numbers were never right, start at `verify` instead: it is the one
    command here that does not need a snapshot, and it answers "is this right"
    rather than "what moved".
    
    ## Per-axis cost: what is free and what scans
    
    Detection is read-only, but read-only is not the same as free on a metered
    connector (BigQuery, Snowflake, Databricks, Postgres, Redshift, ClickHouse).
    The axes split:
    
    - **Schema, volume, and the reference/definition half of semantic are free**
      everywhere: they read metadata and the snapshot, and run immediately.
    - **Grain and the dimension-cardinality half of semantic scan the warehouse**, so
      on a metered connector they run the two-step handshake. Asked for directly,
      `maintain grain` returns `needs_confirmation` with an estimate in
      `cost.estimate` (and a per-table breakdown). Surface it to the user in human
      units, get an explicit budget, and re-issue the same command with
      `--confirm --budget <magnitude>` in the paradigm's unit (bytes on BigQuery,
      warehouse-seconds on Snowflake and Databricks, compute-seconds on Redshift,
      database-seconds on Postgres and
      ClickHouse). Never invent a budget the user did not agree
      to, and never retry with a raised budget on an over-ceiling refusal without
      asking. An over-ceiling
      refusal carries a calibration line from `.dex/spend.jsonl` (what this
      connector's recent commands billed as a fraction of estimate, or a sentence
      saying there is too little history to say): relay it, and note that the
      ceiling binds on the estimate, so a budget set at that fraction of the
      estimate is refused again.
    - **`verify` is free except for the counts a warehouse does not keep.** Its
      build-status findings read artifacts on disk, and its row counts come from
      object metadata. A view has no stored row count anywhere, and a view is dbt's
      default materialization, so on a metered connector those counts are batched
      into one aggregate-only statement, priced, and returned in `data.offer` beside
      findings that are already final. On DuckDB there is no gate, so every count is
      measured rather than estimated and the findings come back `exact`.
    - **`check`, `semantic` and `verify` answer first and offer second.** Their free axes
      complete on every call, so the envelope is `ok` and the findings in it are
      final. The price of the scanning axes sits in `data.offer`, with `axes` naming
      what it would add; `data.axes_run` names what already ran. Confirming is a
      choice, not a required next step: quote the estimate, say which axes are still
      dark, and let the user decide. A triage pass that stops at the free axes is a
      complete piece of work, not an abandoned one.
    - **Read `warnings` on these responses, always.** They carry the reasons the
      baseline may no longer describe the warehouse (a cache newer than the
      snapshot, a baseline pinned from a stale cache), which bound every finding
      above them. A stale baseline is often the most important line in the response
      and it is never in `findings`.
    
    On DuckDB everything is free and local, so nothing prompts.
    
    A `needs_confirmation` envelope carrying `suggested_session_ceiling` is the
    project's one-time ask for a *cumulative* daily cap, separate from the
    per-command `--budget`. Surface it, get the user's answer, and add
    `--session-ceiling <value>` or `--no-session-ceiling` to the same re-issue; it is
    written to `.dex/config.yml` once and never asked again. Never answer it for
    them.
    
    ## Reconcile proposals are mechanical or advisory
    
    Reconcile tags every proposal by `kind`, because the fix differs sharply by axis:
    
    - **`mechanical`**: schema drift reconciles in one of two shapes. On a
      dex-scaffolded staging model it re-scaffolds the model from the drifted source;
      on a project format that places a declaration but authors no staging model, it
      edits the drifted columns into that declaration and says so. High-confidence, but
      still a reviewable diff: read it for hand-written logic the scaffold cannot know
      about.
    - **`advisory`**: grain, volume, and semantic drift are decisions, not auto-fixes
      (dex cannot dedup your warehouse or decide whether a new `'refunded'` status
      belongs in a metric). The proposal is the decision surfaced, at most backed by a
      test edit that makes the break visible in builds. It declines that test where
      the test would be wrong: if your model declares a composite grain covering the
      column, no column-level `unique` is proposed on it, and the warning names the
      combination so you can tell "re-baseline, this is still the grain" from
      "something relied on that column alone".
    
    **A type change is advisory on every format.** Nothing dex writes declares a type,
    and the type it holds is the connector's own spelling rather than a canonical one
    (Snowflake reports `NUMBER(38,0)` and `NUMBER(10,2)` both as `FIXED`), so the
    proposal names both spellings and the edit is yours. One consequence to know: on
    ClickHouse nullability is part of the type, so a column that starts accepting nulls
    is reported as a retype and gets advice where other connectors get an edit.
    
    When reconcile produces edits it stores them as a plan and prints a `plan_id`.
    Apply them with `transform apply <plan-id>` (the one apply door): a human edit made
    since detection surfaces as a conflict, never a silent overwrite.
    
    ## Guardrails (enforced in the engine, not here)
    
    - Read-only against data. Schema, volume, and semantic references are computed from
      metadata and the snapshot; grain and dimension-cardinality use aggregates only.
      Raw rows and dimension values never cross the envelope.
    - Propose, don't impose. Reconciliation is always a reviewable diff, applied
      through `transform apply`. Human edits to the project and to the semantic layer
      are authoritative; on conflict the engine surfaces the divergence and asks
      rather than overwriting.
    - The repository is the source of truth, on both axes; the `.dex/` snapshot is a
      non-canonical fingerprint used only to detect change.
    - The cost guard behind the scanning axes, in full, in the engine repository:
      `references/cost-controls.md`.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related