# RCH Configuration Reference

## Contents

- [Precedence and File Locations](#precedence-and-file-locations)
- [Main Config (`~/.config/rch/config.toml`)](#main-config-configrchconfigtoml)
- [Workers Config (`~/.config/rch/workers.toml`)](#workers-config-configrchworkerstoml)
- [Environment Variables](#environment-variables)
- [Hook Configuration (Claude Code)](#hook-configuration-claude-code)
- [Validation and Diagnostics](#validation-and-diagnostics)
- [Runtime Data Paths](#runtime-data-paths)

> **Authority:** reading config is autonomous (`rch config get|show|validate|
> doctor|diff`, `rch hook status`). Writing it is not — editing
> `~/.config/rch/config.toml` or `workers.toml`, `rch config set|edit|init`, and
> `rch hook install|uninstall` (which write `~/.claude/settings.json`) are host
> mutations requiring explicit caller authorization first (see `FAIL_OPEN.md`
> §"Autonomous Remediation Envelope").

## Precedence and File Locations

RCH resolves settings in this order (highest to lowest):

1. CLI flags (`--json`, `--verbose`, etc.)
2. Environment variables (`RCH_*`)
3. Profile defaults (`RCH_PROFILE`)
4. `.env` / `.rch.env`
5. Project config (`.rch/config.toml`)
6. User config (`~/.config/rch/config.toml`)
7. Built-in defaults

Primary files:

- User config: `~/.config/rch/config.toml`
- Worker config: `~/.config/rch/workers.toml`
- Project override: `.rch/config.toml`
- Optional transfer excludes: `.rchignore`

---

## Main Config (`~/.config/rch/config.toml`)

```toml
[general]
enabled = true
force_local = false
force_remote = false
log_level = "info"                    # trace, debug, info, warn, error, off
socket_path = "~/.cache/rch/rch.sock" # default resolves from runtime/cache path

[compilation]
confidence_threshold = 0.85
min_local_time_ms = 2000
remote_speedup_threshold = 1.2
build_slots = 4
test_slots = 8
check_slots = 2
build_timeout_sec = 300
test_timeout_sec = 1800
bun_timeout_sec = 600
external_timeout_enabled = true

[transfer]
compression_level = 3
remote_base = "/tmp/rch"
adaptive_compression = true
verify_artifacts = false
exclude_patterns = [
  "target/",
  ".git/objects/",
  "node_modules/",
]

[selection]
strategy = "fair_fastest"

[output]
visibility = "summary"                # none, summary, verbose
first_run_complete = true

[self_healing]
hook_starts_daemon = true
daemon_installs_hooks = true
auto_start_timeout_secs = 3
```

Socket path default behavior:

- First choice: `$XDG_RUNTIME_DIR/rch.sock`
- Fallback: `~/.cache/rch/rch.sock`
- Last resort: `/tmp/rch.sock`

---

## Workers Config (`~/.config/rch/workers.toml`)

```toml
[[workers]]
id = "worker-name"
host = "203.0.113.20"
user = "ubuntu"
identity_file = "~/.ssh/id_ed25519"
total_slots = 16
priority = 100
tags = ["rust", "bun", "fast"]
```

Slot guidance:

- Start with ~`2x` physical CPU cores for mixed workloads.
- Reduce slots if workers hit CPU steal, swap pressure, or I/O saturation.
- Increase `priority` for faster/more reliable workers.

---

## Environment Variables

Common overrides:

| Variable | Purpose |
|----------|---------|
| `RCH_PROFILE` | Base profile (`dev`, `prod`, `test`) |
| `RCH_LOG_LEVEL` | Logging level override |
| `RCH_DAEMON_SOCKET` | Daemon socket override (CLI layer) |
| `RCH_SOCKET_PATH` | Socket override (config layer) |
| `RCH_DAEMON_TIMEOUT_MS` | Daemon IPC timeout |
| `RCH_SSH_KEY` | Default SSH key path |
| `RCH_TRANSFER_ZSTD_LEVEL` | Transfer compression level |
| `RCH_ENV_ALLOWLIST` | Forwarded env vars for remote execution |
| `RCH_VISIBILITY` / `RCH_VERBOSE` / `RCH_QUIET` | Hook/CLI visibility controls |
| `RCH_OUTPUT_FORMAT` / `TOON_DEFAULT_FORMAT` | Machine output format |
| `RCH_JSON` / `RCH_HOOK_MODE` | Force machine/hook output mode |
| `NO_COLOR` / `FORCE_COLOR` | ANSI color behavior |

---

## Hook Configuration (Claude Code)

Location: `~/.claude/settings.json`

```json
{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "/absolute/path/to/rch"
          }
        ]
      }
    ]
  }
}
```

Recommended management commands:

```bash
rch hook status       # autonomous
rch hook install      # (authorize first) — writes ~/.claude/settings.json
rch hook uninstall    # (authorize first) — edits ~/.claude/settings.json
```

---

## Validation and Diagnostics

```bash
rch config show --sources
rch config validate
rch config lint
rch config doctor
rch check
```

---

## Runtime Data Paths

| Path | Purpose |
|------|---------|
| `~/.local/share/rch/telemetry/telemetry.db` | Telemetry persistence |
| `~/.local/share/rch/fleet_history/` | Fleet deployment history |
| `~/.cache/rch/` | Cache + default socket parent |
| `/tmp/rch/` | Remote transfer workspace base (default) |

# RCH Error Code Catalog

## Contents

- [Live Catalog](#live-catalog)
- [Categories](#categories)
- [High-Frequency Codes (with the right reaction)](#high-frequency-codes-with-the-right-reaction)
- [Cross-References](#cross-references)
- [Schema Discovery](#schema-discovery)

RCH ships a stable error catalog of 94 codes in the `RCH-Exxx` namespace. Every user-visible failure that is *expected and explainable* carries one of these codes. They appear in:

- `[RCH] remote <worker> failed [RCH-Exxx] <summary>` (build env failures)
- `[RCH] local (dependency preflight RCH-Exxx: <remediation>)` (closure planner)
- `rch doctor --json` (`.checks[].code`)
- `rch --json` responses on errors (`.error.code`)
- Daemon log lines

Treat the code as the **stable handle**. Don't grep for the human-readable summary, which can be reworded between releases.

> **Authority:** only read-only diagnosis is autonomous. Every "First action"
> below that starts/restarts/reconfigures the daemon, adds/provisions/drains/
> enables a worker or its toolchain, edits config, or mutates a remote host
> requires explicit caller authorization first — see `FAIL_OPEN.md` §"Autonomous
> Remediation Envelope". Mutating first actions are marked **(authorize first)**.

---

## Live Catalog

The authoritative catalog is shipped with the binary. Always prefer this over what's quoted below:

```bash
rch schema export -o /tmp/rch-schemas
jq -r '.errors[] | "\(.code) | \(.message)"' /tmp/rch-schemas/error-codes.json | sort
```

Per-code remediation steps:

```bash
jq '.errors[] | select(.code=="RCH-E210") | {code, message, remediation}' /tmp/rch-schemas/error-codes.json
```

---

## Categories

| Range | Category | Lives in |
|---|---|---|
| 001–099 | Configuration | TOML, env vars, profile resolution, path topology, closure plan validation |
| 100–199 | Network | SSH, DNS, TCP — see the Network rows below and `RECOVERY_PLAYBOOKS.md` Playbook C |
| 200–299 | Worker | Selection, health, slots, disk pressure |
| 300–399 | Build | Compilation, toolchain, process triage, cancellation |
| 400–499 | Transfer | rsync, checksums, disk space, perms |
| 500–599 | Internal | Daemon, IPC, hook execution, metrics |

---

## High-Frequency Codes (with the right reaction)

These are the codes agents actually see in practice. The rest are in the schema export.

### Configuration

| Code | Meaning | First action |
|---|---|---|
| RCH-E001 | Config file not found | `rch config init` creates `~/.config/rch/config.toml` — **(authorize first)**. |
| RCH-E003 | Invalid TOML syntax | `rch config validate` to get the line (autonomous). |
| RCH-E007 | No workers configured | `rch workers discover --add --yes && rch workers setup --all` adds/provisions workers — **(authorize first)**. |
| RCH-E008 | Worker config invalid | `rch config doctor` shows which `[[workers]]` block is bad (autonomous). |
| RCH-E009 | SSH key path invalid/inaccessible | Check `identity_file` exists (autonomous); `chmod 600 <key>` mutates a local file — **(authorize first)**. |
| RCH-E013 | Cargo manifest parse failure during path-dep resolution | `cargo metadata --no-deps --format-version 1 > /dev/null` to see the parser error. |
| RCH-E014 | Path dependency declared but target dir missing | The `path = "..."` in a `Cargo.toml` points nowhere. Resolve before retry. |
| RCH-E015 | Cyclic path dependency | Break the cycle in the workspace. |
| RCH-E016 | Path dep violates canonical-root topology | Sibling repo lives outside `[path_topology] canonical_root`. Either move it under the canonical root, or set `[path_topology] canonical_root` to a parent that contains both repos. See the path-dep section of `TROUBLESHOOTING.md`. |
| RCH-E017 | `cargo metadata` invocation failed | Run `cargo metadata --format-version 1` and read the error directly. |
| RCH-E019 | Closure plan computation failed | Re-run with `RCH_LOG_LEVEL=debug rch diagnose --dry-run "<command>"`. |
| RCH-E020 | Closure entered fail-open due to unverifiable data | RCH refuses to ship unsafe closure. Fix the workspace topology, or set `[deps] policy = "permissive"` (a config edit that accepts the risk) — **(authorize first)**. |

### Network

| Code | Meaning | First action |
|---|---|---|
| RCH-E100 | SSH connection failed | `ssh -v ubuntu@<host>` reproduces. Check host reachability. |
| RCH-E101 | SSH auth failed | Wrong key or wrong user. `ssh-add -l` to confirm agent has the right key; `rch config get` for `identity_file`. |
| RCH-E103 | Host key verification failed | Worker rebuilt? Compare with `ssh-keygen -F <host>` (autonomous); removing the old `known_hosts` entry mutates local state — **(authorize first)**, and only if you trust the new fingerprint. |
| RCH-E104 | SSH command timed out | Network or remote slowdown. Bump `RCH_SSH_SERVER_ALIVE_INTERVAL_SECS=15` and retry. |
| RCH-E108 | Connection refused | sshd not running or wrong port. |
| RCH-E109 | TCP connect timeout | Firewall, NAT, or worker down. |

### Worker

| Code | Meaning | First action |
|---|---|---|
| RCH-E200 | No workers available for selection | See `FAIL_OPEN.md` selection-reasons table. |
| RCH-E202 | Worker failed health check | `rch workers probe <id>` reproduces; inspect `rch workers list --speedscore`. |
| RCH-E203 | Worker self-test failed | `rch self-test --worker <id>`; inspect `rch self-test history --limit 5`. |
| RCH-E204 | Worker at maximum capacity | Queueing is on by default; if seen, the wait timed out. Bump `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120` (autonomous); raising `total_slots` edits config — **(authorize first)**. |
| RCH-E205 | Worker missing required toolchain | `rch workers sync-toolchain --all` mutates the worker — **(authorize first)**. |
| RCH-E207 | Worker circuit breaker open | Triggered by repeated failures. Inspect daemon logs (autonomous); circuit auto-closes after cooldown, or force `rch workers enable <id>` after fixing the cause — **(authorize first)**. |
| **RCH-E210** | **Worker disk usage critically high** | **Hand off to the `sbh` skill.** See the disk rows in this table and `RECOVERY_PLAYBOOKS.md` Playbook G. |
| RCH-E211 | Worker disk usage above warning threshold | `sbh` recommended. |
| RCH-E212 | Disk pressure telemetry stale | Worker not reporting; wait one telemetry tick (autonomous). Restarting `rch-wkr` on the worker is remote mutation — **(authorize first)**. |
| RCH-E213 | Worker disk I/O too high | Transient — wait (autonomous), or `rch workers drain <id>` for maintenance — **(authorize first)**. |
| RCH-E214 | Worker memory pressure too high | Same; check what else is running on the worker. |
| RCH-E215 | Disk reclaim failed | sbh ran but couldn't free enough. Manual triage. |
| RCH-E216 | Insufficient disk headroom for build reservation | Free space, or steer to a different worker via `tags`. |
| RCH-E217 | Active build protection prevented reclaim | Wait for active build, then retry reclaim. |

### Build

| Code | Meaning | First action |
|---|---|---|
| RCH-E300 | Remote compilation failed | Read the actual rustc/cargo error in stderr. |
| RCH-E303 | Build operation timed out | Splitting the build is autonomous; raising `[compilation] build_timeout_sec` edits config — **(authorize first)**. |
| RCH-E305 | Remote working dir error | Often = mirror perms broken. See `RECOVERY_PLAYBOOKS.md` Playbook F — the chown fix is a remote `sudo` command, authorize first. |
| RCH-E307 | Build environment setup failed | Missing system package on worker. Detected automatically when stderr names `pkg-config` or `library .pc`. |

### Transfer

| Code | Meaning | First action |
|---|---|---|
| RCH-E400 | Rsync transfer failed | Check `rch daemon logs -n 200` for full rsync stderr. |
| RCH-E401 | Sync timed out | Big workspace + slow link. Tighten excludes or increase compression. |
| RCH-E404 | Insufficient disk on worker | `sbh` on worker. |
| RCH-E405 | Permission denied during transfer | Mirror ownership broken. See `RECOVERY_PLAYBOOKS.md` Playbook F — the chown fix is a remote `sudo` command, authorize first. |
| RCH-E406 | Transfer checksum mismatch | Re-run; if persistent, suspect concurrent agent writes during sync. Coordinate writers with file reservations via the `agent-mail` skill. |

### Internal

| Code | Meaning | First action |
|---|---|---|
| RCH-E500 | Failed to connect to daemon socket | Wait out the auto-start cooldown and retry; manually starting the daemon needs caller authorization. If it spins, see `RECOVERY_PLAYBOOKS.md` Playbook B (cooldown/stale-socket). |
| RCH-E502 | Daemon not running | Same. |
| RCH-E506 | Hook execution failed | `rch hook test` reproduces; capture `RCH_LOG_LEVEL=debug rch hook test`. |

---

## Cross-References

- Path-dep family (RCH-E013–E024): the path-dep section of `TROUBLESHOOTING.md`
- Disk-pressure family (RCH-E210–E217): the disk rows above + the `sbh` skill
- SSH family (RCH-E100–E109): `RECOVERY_PLAYBOOKS.md` Playbook C
- Selection family (RCH-E200–E209): `FAIL_OPEN.md`
- Daemon/internal (RCH-E500–E509): `RECOVERY_PLAYBOOKS.md` Playbook B + `TROUBLESHOOTING.md`

---

## Schema Discovery

For agents that need to consume the catalog programmatically (e.g., to build a remediation table at runtime):

```bash
rch schema list                  # human-readable
rch schema export -o ./schemas   # writes api-response, api-error, error-codes
rch --schema config lint         # JSON Schema for one command's output
rch --capabilities               # full capability description
```

Every command also accepts `--help-json` to dump its argument tree as JSON.

# Detecting and Responding to RCH Fail-Open

## Contents

- [Golden Rule](#golden-rule)
- [The Fail-Open Surface](#the-fail-open-surface)
- [Fail-Open Reasons (and What To Do)](#fail-open-reasons-and-what-to-do)
- [Detection Snippets](#detection-snippets)
- [Force the Issue](#force-the-issue)
- [Autonomous Remediation Envelope (and Where It Stops)](#autonomous-remediation-envelope-and-where-it-stops)

RCH's most expensive failure mode for agents is **silent fall-back to local execution**. The build "succeeded" — but it ran on the local machine, slowly, while the worker fleet sat idle. If you don't notice, you bake hours of extra latency into every iteration.

This file is the canonical guide for: (1) how to *see* a fail-open, (2) what each fail-open reason means, and (3) how to remediate it within the autonomous envelope — and where that envelope stops and explicit caller authorization begins.

---

## Golden Rule

**Never say "build done" until you've checked stderr for `[RCH] local (...)`.**

If you see that string, the build did not run remotely. It might still be a correct build, but RCH chose to fall back, and the parenthetical reason is a contract telling you exactly why.

---

## The Fail-Open Surface

`rch exec` and the PreToolUse hook print exactly one summary line on stderr at the end of every routed compilation. The visibility is controlled by `[output] visibility = "summary"|"verbose"|"none"` (env: `RCH_VISIBILITY`).

There are five summary forms:

| Pattern | Meaning |
|---------|---------|
| `[RCH] remote <worker> (<ms>)` | Successful remote build. Worker name + wall-clock time. |
| `[RCH] remote <worker> failed (exit <N>)` | Build ran remotely and the build itself failed. Treat as a normal compiler error. |
| `[RCH] remote <worker> failed [RCH-Exxx] <summary>` | Build environment failure on the worker (missing system package, etc.). See `ERROR_CODES.md`. |
| `[RCH] local (<reason>)` | **Fail-open.** Compilation ran locally instead of remotely. Read the reason. |
| *(no summary)* | Visibility is `none` or RCH never engaged. Re-run with `RCH_VISIBILITY=summary` to confirm. |

To force a summary banner without changing config:

```bash
RCH_VISIBILITY=verbose cargo check
```

---

## Fail-Open Reasons (and What To Do)

Every reason in the parens comes from one of two sources:

1. **Hook decision points** in `rch/src/hook.rs::process_hook` and `run_exec` — short, hand-written reason strings
2. **Daemon selection reasons** (`SelectionReason` enum in `rch-common/src/types.rs`) — stable machine reasons from worker selection

### Hook-decision fail-opens

| Reason text | Triggered when | Self-fix |
|---|---|---|
| `daemon unavailable` | Daemon socket can't be reached (and auto-start failed or is disabled) | Wait out the auto-start cooldown and retry (autonomous). If still failing, confirm with `rch --json daemon status` and check `~/.cache/rch/rch.sock`; manually starting the daemon needs caller authorization. See `RECOVERY_PLAYBOOKS.md` Playbook B. |
| `force_local` | `[general] force_local = true` is set | This is intentional. `rch config get general.force_local --sources` shows where it came from (autonomous). Reverting with `rch config set general.force_local false` edits config — **(authorize first)**. |
| `invalid config: force_local+force_remote` | Both flags set simultaneously | `rch config edit` to unset one, then `rch daemon reload` — both mutate config/daemon, **(authorize first)**. |
| `confidence below threshold` | Classifier flagged the command but only weakly (e.g., wrapped in shell pipelines). Threshold is `[compilation] confidence_threshold` (default 0.85). | Run `rch diagnose "<the command>"` to see classifier confidence (autonomous). Lowering the threshold or setting `[general] force_remote = true` in `.rch/config.toml` are config mutations — **(authorize first)**. |
| `command '<base>' not in allowlist` | `[execution] allowlist` excludes this command base | `rch --json config get execution.allowlist` to inspect (autonomous). Adding the command base to `.rch/config.toml` is a config mutation — **(authorize first)**. |
| `dependency preflight <RCH-Exxx>: <remediation>` | The closure planner refused to ship the workspace (cycle, missing manifest, off-canonical-root path dep). See the RCH-E013–E020 rows in `ERROR_CODES.md` and the path-dep section of `TROUBLESHOOTING.md`. | The remediation message is actionable; follow it. Then re-run `rch diagnose --dry-run "<command>"`. |
| `<TransferSkipped reason>` | Transfer pipeline opted out (e.g., empty workspace, all paths excluded). | Run `RCH_LOG_LEVEL=debug rch exec -- <command>` and look for `Transfer skipped:` log lines. |
| `remote execution failed` | Generic catch-all for transfer/exec errors | Re-run with `RCH_LOG_LEVEL=debug` to surface the real error, and check `rch daemon logs -n 200` for the daemon side. |
| `toolchain missing on <worker>` | Remote `rustup`/`cargo` not present, or no default toolchain | Diagnose the gap (autonomous). `rch workers sync-toolchain --all` (or one worker) mutates the worker — get caller authorization first; then `rch workers capabilities --refresh`. |

### Daemon-decision fail-opens (selection reasons)

These come from the daemon's `SelectionReason` enum. The `[RCH] local (...)` summary uses the **human Display** form (verbatim from `Display for SelectionReason` in `rch-common/src/types.rs`). Machine-readable JSON output (`rch --json`) uses the **snake_case tag** instead. Match either when you grep — the human form is what appears on stderr.

| Snake_case tag (JSON) | Human form in `local (...)` summary | Self-fix |
|---|---|---|
| `no_workers_configured` | `no workers configured` | `rch workers discover --add --yes && rch workers setup --all` adds and provisions workers — a fleet mutation; get caller authorization first. |
| `all_workers_unreachable` | `all workers unreachable` | `rch workers probe --all` and read the SSH errors (autonomous). Any fix beyond reading — repairing a local key's permissions included — is a host mutation → **(authorize first)**; changing anything on a worker is remote mutation → **(authorize first)**. See `RECOVERY_PLAYBOOKS.md` Playbook C and the Network rows of `ERROR_CODES.md`. |
| `all_circuits_open` | `all worker circuits open` | A worker hit repeated failures and tripped its circuit. Inspect with `rch --json status --workers \| jq '.data.daemon.workers[] \| {id, circuit_state, last_error, recovery_in_secs}'` and `rch daemon logs -n 200` (autonomous). Circuits auto-close after the cooldown; forcing `rch workers enable <id>` mutates worker state — **(authorize first)**. |
| `all_workers_busy` | `all workers at capacity` | Queueing is **on by default** (`RCH_QUEUE_WHEN_BUSY=1`); seeing this means the wait timed out. Bump `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120` for the next invocation (autonomous). Raising `total_slots` edits worker config — **(authorize first)**. Check `rch queue --watch` to see backlog. |
| `all_workers_failed_preflight` | `all workers failed preflight checks` | Path-topology check, repo presence, or toolchain probe failed on every candidate. Re-run with `rch diagnose --dry-run "<command>"` to see the preflight pipeline; hits `RCH-E013..024`, `RCH-E205`, `RCH-E305`. |
| `all_workers_failed_convergence` | `all workers failed repo convergence checks` | The repo updater contract couldn't bring required repos to a target state on any worker. Check that the sibling repos exist on workers under the canonical root. See the RCH-E013–E020 rows in `ERROR_CODES.md` and the path-dep section of `TROUBLESHOOTING.md`. |
| `no_matching_workers` | `no matching workers found` | The project requires tags (e.g., `tags = ["bun"]`) and no worker carries them. `rch workers list --json` to inspect tags (autonomous); adding the tag edits worker config — **(authorize first)**. |
| `no_workers_with_runtime` (value = runtime name) | `no workers with bun installed` (or `node`, `rust`, …) | Installing the runtime on a worker and `rch workers capabilities --refresh` both mutate worker state — **(authorize first)**. |
| `selection_error` (value = error text) | `selection error: <msg>` | An internal error during selection. Check `rch daemon logs -n 200`. Likely a code-side bug; capture `rch doctor --json` and `rch --json daemon status` for escalation. |

Two more variants — `affinity_pinned` and `affinity_fallback` — are *success* paths (a worker was assigned via affinity), not fail-opens, so they never appear in `[RCH] local (...)` output.

### "Build" succeeded but nothing went remote

If the command exited 0, the agent often calls the work done. **Check the summary line first.** A common pathology:

```
   Compiling foo v0.1.0
    Finished `dev` profile in 38.41s
[RCH] local (all workers at capacity)
```

That's a 38-second build that should have been 2 seconds remote. Queueing is on by default; if you still see this, bump `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120` so the next iteration waits longer for a slot rather than burning local CPU.

---

## Detection Snippets

Single-shot check after a build:

```bash
# Show the last RCH summary line from this shell's stderr capture
grep -E '^\[RCH\] (remote|local)' /tmp/rch_last_run.stderr | tail -1
```

Wrapper that runs a build and asserts remote:

```bash
out=$(RCH_VISIBILITY=summary cargo check 2>&1)
if grep -qE '^\[RCH\] local' <<<"$out"; then
  echo "RCH FELL BACK TO LOCAL:" >&2
  grep -E '^\[RCH\] local' <<<"$out" >&2
  exit 99
fi
echo "$out" | tail -3
```

JSON-mode check (for scripted agents):

```bash
rch --json check | jq -r '.data.status'   # ready | degraded | unhealthy
```

---

## Force the Issue

When you absolutely need to know whether remote works at all (for example, before deciding to file an issue), bypass the hook entirely:

```bash
rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets
```

If that prints `[RCH] remote <worker> (...)`, the offload path is healthy and the failures are coming from the hook classifier (or some shell wrapper around your command). If it also prints `[RCH] local (...)`, follow the reason above.

---

## Autonomous Remediation Envelope (and Where It Stops)

The skill resolves diagnosis autonomously — but the SKILL.md authority boundary
governs, and this file does not widen it. Two tiers:

**Autonomous (no need to ask) — the blast-radius ceiling is read-only.** Only
diagnosis and inspection that mutates no host, worker, daemon, config, or
filesystem state:

- reading summary lines, stderr, and `rch daemon logs`;
- status/inspection commands: `rch check`, `rch status`, `rch --json daemon
  status`, `rch diagnose`/`--dry-run`, `rch --json workers probe|list`, `rch
  config get|show|validate|doctor|diff`, `rch hook status|test`, `rch fleet
  status|verify`, `rch queue`, `rch self-test`;
- tuning env vars for the *next* invocation (`RCH_VISIBILITY`, `RCH_LOG_LEVEL`,
  `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120`, `RCH_SSH_SERVER_ALIVE_INTERVAL_SECS`);
- re-running the original command *only when that command is itself read-only*,
  including waiting out an auto-start cooldown (the built-in
  `try_auto_start_daemon` handles stale sockets — do not `rm` them). A
  mutating original command re-runs only under the authorization that covered
  it in the first place.

**Requires explicit caller authorization first — even when a reason maps
straight to one of these commands.** Every host, worker, daemon, config, or
filesystem mutation crosses the ceiling. It does not matter that a table cell
below lists the command — a listed command is not a pre-authorized one:

- **local host mutation** — `rch hook install|uninstall` (writes
  `~/.claude/settings.json`), `chmod` on a local key file, `rch config
  set|edit|init`, and any edit to `~/.config/rch/*.toml`;
- **remote host mutation** — any `ssh … sudo …`, remote `chown`/`chmod`/`rm`,
  remote installs, restarting `sshd` or `rch-wkr` on a worker, changing a
  worker's `authorized_keys` or its host-key entry;
- **worker / fleet operations** — `rch workers
  sync-toolchain|setup|deploy-binary|discover --add|drain|disable|enable|capabilities --refresh`,
  `rch fleet deploy|rollback`;
- **daemon lifecycle** — `rch daemon start|restart|reload|stop`;
- **destructive cleanup** — removing sockets, cooldown/lock files, caches, or
  moving the telemetry DB aside.

For each reason, run the autonomous diagnosis, then, if the fix is above the
ceiling, capture the evidence and get authorization before running it:

- **"daemon unavailable"** → confirm with `rch --json daemon status`; wait out
  the auto-start cooldown and retry (autonomous). Manually starting/restarting
  the daemon, or removing the cooldown file, needs authorization.
- **"all workers unreachable"** → `rch workers probe --all` and read the SSH
  errors (autonomous). Every fix mutates state and needs authorization first:
  `chmod` on a local key, and anything **on** a worker (host-key entry,
  `authorized_keys`, restarting `sshd`).
- **"all workers at capacity"** → queueing is on by default; bump
  `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120` for the next run (autonomous).
  Raising `total_slots` edits worker config → authorization.
- **"toolchain missing on X"** → diagnose the gap (autonomous); running `rch
  workers sync-toolchain` mutates the worker → get authorization first.
- **Permission denied on `/data/projects/<repo>` over rsync** → the fix is a
  remote privileged command (`ssh … 'sudo chown -R …'`). Report it to the
  caller with the failing `stat` and get explicit authorization before running
  it — remote `sudo` is never autonomous.

When a fix is above the ceiling or you are in genuine doubt, surface the packet —
`rch doctor --json`, `rch --json daemon status`, `rch --json workers probe
--all`, the failing command's stderr, and the exact command you propose — so the
caller can authorize (or decline) in one round trip.

# Machine-Readable Surfaces (For Agents)

## Contents

- [Three Levels of Discovery](#three-levels-of-discovery)
- [Per-Command JSON Mode](#per-command-json-mode)
- [Unified Response Envelope](#unified-response-envelope)
- [Output Format Modes](#output-format-modes)
- [Useful jq Recipes](#useful-jq-recipes)
- [Wire-Level Hook Protocol](#wire-level-hook-protocol)
- [Built-In Robot Docs](#built-in-robot-docs)
- [Where Agents Trip Up](#where-agents-trip-up)

`rch` is built for agents. Every command returns structured output, every command exposes its schema, and the whole CLI is queryable as JSON. This file is the canonical guide for using those surfaces — so an agent can discover capability instead of guessing.

---

## Three Levels of Discovery

### 1. `--capabilities` — what does this `rch` know how to do?

```bash
rch --capabilities
```

Returns a JSON object describing version, build info, supported runtimes (rust, bun, node), available subcommands, and feature flags. Use this once at session start to confirm you're talking to the rch you expect.

### 2. `--help-json` — full CLI tree as JSON

```bash
rch --help-json                  # entire CLI
rch --help-json workers          # one subcommand subtree
rch --help-json workers probe
```

Parse this to drive an agent that needs to construct flags it hasn't seen before.

### 3. `--schema` — JSON Schema for a specific command's output

```bash
rch --schema config lint
rch --schema workers list
rch --schema daemon status
```

Validate the JSON you get back, or use it to build typed clients.

---

## Per-Command JSON Mode

Every subcommand accepts `--json` (and `-F json|toon`):

```bash
rch --json check                          # quick health
rch --json daemon status                  # daemon state
rch --json workers list                   # configured workers
rch --json workers probe --all            # connectivity
rch --json status --workers --jobs        # full status with workers and jobs
rch --json queue                          # build backlog
rch --json hook status                    # hook install state across agents
rch --json agents status                  # agent detection result
rch --json self-test --all                # end-to-end verification
rch --json speedscore --all               # composite score per worker
rch --json fleet status                   # fleet deploy state
rch --json fleet history --limit 20       # deployment timeline
rch --json doctor                         # diagnostic report
rch --json config show --sources          # effective config + provenance
rch --json config get general.socket_path # one value with source
rch --json config diff                    # delta from defaults
rch --json diagnose --dry-run "<cmd>"     # explain routing decision
```

**stdout is always data-only.** Diagnostics go to stderr. Exit code 0 means success.

---

## Unified Response Envelope

Every JSON response follows this shape:

```json
{
  "kind": "ok" | "error",
  "command": "workers.probe",
  "data":  { ... },        // present when kind=ok
  "error": { ... },        // present when kind=error
  "elapsed_ms": 123,
  "request_id": "...",
  "version": "1.0.18"
}
```

Schema: `rch schema export -o ./schemas` produces:

- `api-response.schema.json` — the success envelope
- `api-error.schema.json` — the error envelope
- `error-codes.json` — the full RCH-Exxx catalog

Errors carry `error.code` (`RCH-Exxx`), `error.message`, and `error.remediation` (an array of strings). Use the code as the stable handle.

---

## Output Format Modes

```bash
RCH_OUTPUT_FORMAT=json rch status     # JSON (implies --json)
RCH_OUTPUT_FORMAT=toon rch status     # TOON (compact text-overlay format)
TOON_DEFAULT_FORMAT=toon rch --json status   # Switch JSON-flagged calls to TOON
```

For agent pipelines, JSON is universally safest. TOON is useful for terminals.

`NO_COLOR=1` and `FORCE_COLOR=1` work as expected.

---

## Useful jq Recipes

These jq paths reflect the actual response shapes in rch v1.0.18. Each path
was verified against live output, not assumed.

```bash
# Daemon health summary
rch --json check | jq -r '.data.status'   # "ready" | "degraded" | "unhealthy"

# Daemon version (NOT in 'rch --json daemon status' — that endpoint is minimal)
rch --json status | jq -r '.data.daemon.daemon.version'

# Worker IDs that are reachable. `rch --json workers probe --all` returns
# .data as a flat array, not nested under .workers.
rch --json workers probe --all \
  | jq -r '.data[] | select(.status == "ok") | .id'

# Workers that aren't healthy (any non-"ok" status surfaces an error string)
rch --json workers probe --all \
  | jq -r '.data[] | select(.status != "ok") | "\(.id) [\(.status)] \(.error // "")"'

# Workers under pressure. Pressure fields live FLAT on each worker record
# under .data.daemon.workers[] inside `rch --json status`.
rch --json status --workers \
  | jq -r '.data.daemon.workers[]
           | select(.pressure_state != "healthy")
           | "\(.id) [\(.pressure_state)] \(.pressure_reason_code)"'

# Active builds (lives in `rch --json queue`, NOT `daemon status`)
rch --json queue | jq -r '.data.active_builds[]? | "\(.id) \(.worker_id) \(.project_id)"'

# Queue depth
rch --json queue | jq '.data.active_builds | length'

# Configured workers (canonical shape: .data.workers[].{id, host, user, total_slots, priority, tags})
rch --json workers list | jq -r '.data.workers[] | "\(.id)\t\(.host)\t\(.tags|join(","))"'

# Hook install state across detected agents
rch --json hook status | jq -r '.data.agents[] | "\(.agent)\t\(.status)"'

# All known error codes for a category (after `rch schema export -o ./schemas`)
jq -r '.errors[] | select(.category == "transfer") | "\(.code)\t\(.message)"' schemas/error-codes.json
```

---

## Wire-Level Hook Protocol

`rch` is itself a Claude Code PreToolUse hook. You can hand-craft requests to it (useful for tests):

```bash
printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"cargo build --release"}}' \
  | rch
```

Three response shapes:

- Empty stdout → allow unchanged
- `{"hookSpecificOutput": {"permissionDecision": "allow", "updatedInput": {"command": "rch exec -- ..."}}}` → allow with rewrite
- `{"hookSpecificOutput": {"permissionDecision": "deny", "permissionDecisionReason": "..."}}` → block

The hand-crafted `printf ... | rch` request above is the protocol probe: empty
stdout means the classifier rejected the command.

---

## Built-In Robot Docs

```bash
rch --help                       # human help
rch --help-json                  # everything as JSON
rch schema list                  # what schemas are available
rch schema export -o ./schemas   # write them to disk
```

For comparison, `cass robot-docs guide` is the cass equivalent (used by the `cass` skill).

---

## Where Agents Trip Up

- **Forgetting `--json`.** Human-readable rch output is nice but reformats. Always use `--json` (or `--schema`) when piping into other tools.
- **Conflating "exit 0" with "build was remote".** It isn't. See `FAIL_OPEN.md`.
- **Bare `rch dashboard` / `rch web`.** Both launch interactive UIs that block your session. Don't run them from automation.
- **Bare `rch tui`-like commands.** RCH does not currently ship a `tui` subcommand; the dashboard is `rch dashboard`. The general anti-pattern is the same: anything interactive blocks.
- **Reading `--json` output with grep instead of jq.** Field names are stable. Use jq.

# Recovery Playbooks (Symptom → Fix in ≤ 90 Seconds)

## Contents

- [Playbook A: "My build feels slow"](#playbook-a-my-build-feels-slow)
- [Playbook B: "Daemon unavailable"](#playbook-b-daemon-unavailable)
- [Playbook C: "All workers unreachable"](#playbook-c-all-workers-unreachable)
- [Playbook D: "All workers at capacity"](#playbook-d-all-workers-at-capacity)
- [Playbook E: "All workers failed preflight"](#playbook-e-all-workers-failed-preflight)
- [Playbook F: "Permission denied" during sync](#playbook-f-permission-denied-during-sync)
- [Playbook G: Worker disk pressure](#playbook-g-worker-disk-pressure)
- [Playbook H: Hook isn't being called](#playbook-h-hook-isnt-being-called)
- [Playbook I: Hook timeout / slow classification](#playbook-i-hook-timeout--slow-classification)
- [Playbook J: Worker version drift after upgrade](#playbook-j-worker-version-drift-after-upgrade)
- [Playbook K: `rch self-test` won't finish](#playbook-k-rch-self-test-wont-finish)
- [Playbook L: TOML/config edit broke things](#playbook-l-tomlconfig-edit-broke-things)
- [When the Playbook Doesn't Apply](#when-the-playbook-doesnt-apply)

Each playbook is structured as: **observed signal → one-shot diagnostic → ordered fix attempts → verification**. Run them in order; don't skip the diagnostic.

Diagnostics and reversible, local-only fixes run autonomously. Any step that
mutates a remote worker, runs `sudo`, starts/restarts/reconfigures the daemon,
deploys binaries or toolchains, or deletes state requires explicit caller
authorization **first** — the SKILL.md authority boundary governs, and a playbook
listing a command does not pre-authorize it (see `FAIL_OPEN.md` §"Autonomous
Remediation Envelope"). Where a step below crosses that ceiling it is marked
**(authorize first)**.

For unknown symptoms or a stuck loop, fall through to the bottom of this file ("When the Playbook Doesn't Apply") for the escalation packet.

---

## Playbook A: "My build feels slow"

**Signal:** Build wall time is much higher than expected; you suspect rch fell back to local.

```bash
# Diagnostic
RCH_VISIBILITY=verbose cargo check 2>&1 | grep -E '^\[RCH\]'
```

**Fix attempts:**

1. If you see `[RCH] local (...)` — open `references/FAIL_OPEN.md` and look up the parenthetical reason. Apply the matching self-fix.
2. If you see no `[RCH]` line at all — the hook isn't intercepting. Run `rch hook status` and `rch hook test` (autonomous). If not installed, `rch hook install` writes `~/.claude/settings.json` — **(authorize first)**.
3. If you see `[RCH] remote ...` — RCH is doing what it can; the slowness is real. Inspect `rch speedscore --all` and consider whether the project warrants a faster worker.

**Verify:**

```bash
rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check --workspace --all-targets 2>&1 | tail -5
```

The summary line should say `[RCH] remote <worker> (...)`.

---

## Playbook B: "Daemon unavailable"

**Signal:** `[RCH] local (daemon unavailable)` or `RCH-E500 / RCH-E502`.

```bash
# Diagnostic
rch --json daemon status 2>&1 | head -20
ls -la "${XDG_RUNTIME_DIR:-/tmp}/rch/" 2>&1
```

**Fix attempts:**

1. If `which rchd` is empty → daemon binary missing. Install rch (see project README).
2. If autostart cooldown is recent → wait 5–10 seconds and retry the original command.
3. If autostart lock is held by no process → it's stale. Ask user authorization, then `rm "${XDG_RUNTIME_DIR:-/tmp}/rch/hook_autostart.lock"`.
4. Foreground spawn to surface the real error: `rch daemon start` **(authorize first)** — starting the daemon is a lifecycle mutation. Read its stderr.
5. If `rch daemon start` succeeds but the hook still doesn't see the daemon, check socket consistency: `rch --json config get general.socket_path` and `rch --json daemon status | jq '.data.socket_path'` must match (autonomous inspection). If they don't: `rch daemon restart -y` **(authorize first)**.

**Verify:** `rch check` returns `ready`.

---

## Playbook C: "All workers unreachable"

**Signal:** `[RCH] local (all_workers_unreachable)` or `RCH-E100 / RCH-E101 / RCH-E108`.

```bash
# Diagnostic
rch --json workers probe --all | jq '.data[] | {id, status, last_error}'
```

**Fix attempts (per failing worker):**

1. SSH directly: `ssh -v -i <identity_file> ubuntu@<host> 'echo OK'`. The first error you see is the real one.
2. Auth error → check `identity_file` permissions (`chmod 600`), key on agent (`ssh-add -l`), and authorized_keys on the worker.
3. Connection refused → `sshd` not running on worker, or wrong port.
4. DNS / network unreachable → host moved or networking broken.
5. Host key changed → with explicit user authorization, refresh the entry. Compare fingerprints first.

**Verify:** `rch workers probe <id>` returns ok.

---

## Playbook D: "All workers at capacity"

**Signal:** `[RCH] local (all workers at capacity)` (snake-case tag in JSON: `all_workers_busy`) or `RCH-E204` repeatedly.

**Fix attempts (in order, escalating):**

1. **Verify queueing is on** — `RCH_QUEUE_WHEN_BUSY` is **already enabled by default** in current rch (only set `=0` to disable). If you still see `all workers at capacity`, queueing didn't help — the wait timed out.
2. Bump the wait timeout for the next invocation: `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS=120 <your-command>`.
3. Check whether one worker is hot and others are cold (uneven distribution): `rch --json status --workers | jq '.data.daemon.workers[] | {id, used: .used_slots, total: .total_slots}'`.
4. If aggregate capacity is the problem, raising `total_slots` on top workers in `~/.config/rch/workers.toml` and `rch daemon reload` both mutate config/daemon — **(authorize first)**.
5. Watch the backlog drain: `rch queue --watch` (this is an interactive polling view, not a TUI — Ctrl-C exits cleanly).

**Verify:** Next `rch exec` lands `[RCH] remote <worker> (...)`.

---

## Playbook E: "All workers failed preflight"

**Signal:** `[RCH] local (all workers failed preflight checks)` (snake-case tag: `all_workers_failed_preflight`) or `RCH-E013..024 / RCH-E205 / RCH-E305`.

```bash
# Diagnostic
rch diagnose --dry-run "<the command>" 2>&1 | head -50
RCH_LOG_LEVEL=debug rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_diag" cargo check 2>&1 | tail -40
```

**Fix attempts:**

1. If a path-dep error (RCH-E013..016) → see the RCH-E013–E020 rows in `ERROR_CODES.md` and the path-dep section of `TROUBLESHOOTING.md` for the exact code.
2. If `RCH_TOPOLOGY_ERR_CANONICAL_NOT_DIRECTORY` or `_ALIAS_NOT_SYMLINK` in worker stderr → fixing the topology on the worker (`/data/projects` should be a directory; `/dp` should be a symlink to it) is remote mutation **(authorize first)**.
3. If `RCH-E205 Worker missing toolchain` → `rch workers sync-toolchain --all` **(authorize first)** — it mutates the worker.
4. If `RCH-E305 Remote working dir error` → typically mirror perms broken; see Playbook F.

**Verify:** `rch diagnose --dry-run "<command>"` reports `Ready` for the closure plan.

---

## Playbook F: "Permission denied" during sync

**Signal:** `[RCH] local (remote execution failed)` followed by stderr lines mentioning `Permission denied` or `Operation not permitted` under `/data/projects/<repo>`.

```bash
# Diagnostic
ssh ubuntu@<worker> "stat -c '%U:%G %a %n' /data/projects/<repo>"
```

**Fix (authorize first):** the repair is a remote privileged command. Report the
failing `stat` and the exact command to the caller and get explicit
authorization before running it — remote `sudo` is never autonomous.

```bash
# after explicit caller authorization only:
ssh ubuntu@<worker> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'
rch exec -- env CARGO_TARGET_DIR="${TMPDIR:-/tmp}/rch_target_$(basename "$PWD")" cargo check
```

If you see this on multiple repos, audit who's doing `sudo git clone` or running things as root in `/data/projects`.

---

## Playbook G: Worker disk pressure

**Signal:** `[RCH] remote <worker> failed [RCH-E210]` (or `E211/E215/E216/E217`); or `rch status` calls out a worker as critical.

See the RCH-E210–E217 rows in `ERROR_CODES.md` and hand disk reclaim to the `sbh` skill. TL;DR:

```bash
rch --json status --workers | jq '.data.daemon.workers[] | select(.pressure_state != "healthy") | {id, pressure_state, pressure_reason_code, pressure_disk_free_gb}'
ssh ubuntu@<worker> 'df -h / /tmp && free -h && cat /proc/pressure/memory'
ssh ubuntu@<worker> 'sbh status --json'      # let sbh handle it
```

If `sbh` isn't installed on the worker: install it, or escalate. Don't `rm -rf` build artifacts blindly — other agents might be mid-build.

---

## Playbook H: Hook isn't being called

**Signal:** Builds run locally; `[RCH]` summary lines never appear; the hook seems silent.

```bash
# Diagnostic
rch hook status --json
rch agents status --json
which rch
cat ~/.claude/settings.json 2>/dev/null | jq '.hooks.PreToolUse'
```

**Fix attempts:**

1. Hook not installed → `rch hook install` (or `rch agents install-hook claude-code`) writes `~/.claude/settings.json` — **(authorize first)**.
2. Hook command path is wrong → `rch hook install` re-resolves to the current absolute path — same host write, **(authorize first)**.
3. The Claude Code session predates the hook install → restart Claude Code (the harness only loads hooks on startup).
4. Hook installed but fires for a different agent → confirm `rch agents list --json` includes the agent you're running under.

**Verify:**

```bash
rch hook test
printf '%s\n' '{"tool_name":"Bash","tool_input":{"command":"cargo check"}}' | rch
```

The second should produce a JSON `updatedInput` rewriting to `rch exec -- cargo check`.

---

## Playbook I: Hook timeout / slow classification

**Signal:** Claude Code reports the hook timed out, or the hook is logging classification budget warnings.

```bash
# Diagnostic
RCH_LOG_LEVEL=debug printf '%s\n' '{"tool_name":"Bash","tool_input":{"command":"cargo build"}}' | rch 2>&1 | grep -iE 'budget|classif'
```

**Likely causes:**

- Massive `Cargo.toml`/`metadata` and the closure preflight is slow → cache should warm; if not, the cache may be invalid: `rm -rf ~/.cache/rch/classify_cache_v*` (with explicit user authorization).
- A misbehaving CI invocation is running rch in a loop and starving the daemon.

If the hook's compilation decision exceeds 5ms, that's a budget regression worth filing.

---

## Playbook J: Worker version drift after upgrade

**Signal:** New rch features behave inconsistently across workers; `rch fleet status` shows mixed versions.

```bash
rch fleet status --json    # exact JSON shape varies by version; inspect first
rch fleet verify           # human-readable comparison of installed binaries
```

**Fix (authorize first):** fleet deployment mutates every worker binary. Present
the `rch fleet status`/`verify` evidence and the proposed rollout to the caller
and get explicit authorization before any `deploy`/`rollback`.

```bash
# after explicit caller authorization only:
rch fleet deploy --canary 25 --canary-wait 60 --verify
# observe output, then
rch fleet deploy --verify           # full rollout
rch fleet verify                    # confirm uniform
```

If rollback is needed (also authorize first): `rch fleet rollback --verify`.

For a single worker, `rch fleet deploy --worker <id> --verify` (deploy, single host) — still authorize first.

---

## Playbook K: `rch self-test` won't finish

**Signal:** `rch self-test --all` runs forever or returns no output for minutes.

```bash
# Diagnostic
rch self-test --worker <id> --timeout 120 --debug 2>&1 | tail -30
rch self-test history --limit 5 --json
```

**Fix attempts:**

1. Try a single worker with `--timeout 120 --debug`. If that works, the `--all` mode is hitting a slow worker — narrow down with `rch speedscore --all`.
2. If self-test hangs against any single worker, that worker has a deeper problem. `rch workers probe <id>` (autonomous); draining it (`rch workers drain <id>`) mutates worker state — **(authorize first)** — then continue without it.
3. Capture a full doctor report: `rch doctor --json > /tmp/rch-doctor.json`. The pre-v1.0.16 self-test hang bug is fixed; if you reproduce on current rch, escalate with the doctor output.

---

## Playbook L: TOML/config edit broke things

**Signal:** Things were working; you edited `~/.config/rch/config.toml` or `workers.toml`; now nothing works.

```bash
rch config validate
rch config doctor
rch config show --sources
rch config diff                      # what differs from defaults
```

The four `rch config` inspection commands above are autonomous. If `rch config validate` flags an issue, fixing the indicated line and `rch daemon reload` both mutate config/daemon — **(authorize first)**. `git diff` of the config (if version-controlled) is a read-only aid; `rch config init` overwrites a clean baseline — **(authorize first)**.

---

## When the Playbook Doesn't Apply

Capture the escalation packet before pinging the human:

```bash
mkdir -p /tmp/rch-escalation && cd /tmp/rch-escalation
rch doctor --json                         > doctor.json 2>&1 || true
rch --json daemon status                  > daemon-status.json 2>&1 || true
rch --json workers probe --all            > workers-probe.json 2>&1 || true
rch --json status --workers --jobs        > status.json 2>&1 || true
rch --json queue                          > queue.json 2>&1 || true
rch --json config show --sources          > config.json 2>&1 || true
rch --json hook status                    > hook-status.json 2>&1 || true
rch --json agents status                  > agents-status.json 2>&1 || true
rch --json self-test --all --timeout 120  > selftest.json 2>&1 || true
rch daemon logs -n 500                    > daemon.log 2>&1 || true
{ echo "rch=$(rch --version)"; echo "rchd=$(rchd --version 2>/dev/null || true)"; \
  echo "daemon-reported-version=$(rch --json status 2>/dev/null | jq -r '.data.daemon.daemon.version // ""')"; } > versions.txt
ls -lah
```

Then surface a one-paragraph synthesis to the human: what symptom you saw, what you tried (with playbook letter), where the packet lives. Don't ship a wall of text; the packet is the data, your message is the signal.

# RCH Troubleshooting

## Contents

- [Diagnostic Flow](#diagnostic-flow)
- [Common Errors](#common-errors)
- [Debug Mode](#debug-mode)
- [Safe Reset Sequence](#safe-reset-sequence)
- [Reading `rch status` Output Correctly](#reading-rch-status-output-correctly)
- [Daemon Version Drift After Upgrade](#daemon-version-drift-after-upgrade)
- [Telemetry Corruption](#telemetry-corruption)
- ["Why did my command run locally?" (Silent Fail-Open)](#why-did-my-command-run-locally-silent-fail-open)
- [See Also](#see-also)

## Diagnostic Flow

```text
Compilation running locally instead of remotely?
│
├─ Quick health gate:
│  $ rch check
│  │
│  ├─ Not ready/degraded?
│  │   ├─ Check daemon:
│  │   │  $ rch --json daemon status
│  │   │
│  │   ├─ Check workers:
│  │   │  $ rch workers probe --all
│  │   │
│  │   └─ Check hook install:
│  │      $ rch hook status
│  │
│  └─ Ready?
│      continue below
│
└─ Ready but behavior is wrong?
   ├─ Socket alignment:
   │  $ rch --json config get general.socket_path
   │  $ rch --json daemon status
   │
   ├─ Explain routing decision:
   │  $ rch diagnose "cargo build --release"
   │
   ├─ Validate hook protocol path:
   │  $ rch hook test
   │
   └─ Force direct offload proof:
      $ rch exec -- cargo check --workspace --all-targets
```

---

## Common Errors

> **Authority:** only read-only diagnosis is autonomous. Any recipe below that
> starts/restarts/reloads the daemon, adds/provisions/drains a worker or its
> toolchain, edits config, installs the hook, or mutates a remote host requires
> explicit caller authorization first — see `FAIL_OPEN.md` §"Autonomous
> Remediation Envelope". Such steps are called out per block.

### Daemon not running / `check` says not ready

**Cause:** daemon process absent or startup failure.

```bash
rch --json daemon status          # autonomous
rch daemon logs -n 200            # autonomous
rch daemon start                  # (authorize first) — daemon lifecycle
```

### Socket mismatch between config and daemon

**Cause:** `general.socket_path` differs from active daemon socket.

```bash
rch --json config get general.socket_path   # autonomous
rch --json daemon status                     # autonomous
# then align and restart AFTER explicit caller authorization:
rch daemon restart -y                        # (authorize first)
```

### "No workers available" / probe failures

**Cause:** no workers configured, SSH/auth failures, or workers are disabled/drained.

```bash
rch workers list                  # autonomous
rch workers probe --all           # autonomous
rch workers discover --probe      # autonomous (probe only)
# adding/provisioning workers mutates the fleet — authorize first:
rch workers discover --add --yes  # (authorize first)
rch workers setup --all           # (authorize first)
```

### "rustup: not found" / "cargo: not found" on worker

**Cause:** missing toolchain on one or more workers.

`rch workers sync-toolchain --all` mutates the workers — get explicit caller
authorization first (`rch workers capabilities --refresh` after is read-only):

```bash
# after explicit caller authorization only:
rch workers sync-toolchain --all
rch workers capabilities --refresh
```

If still failing, SSH to the specific worker and validate `rustup`, `cargo`, and PATH.

### Hook not intercepting

**Cause:** hook missing, wrong binary path, or command classified as local.

```bash
rch hook status                        # autonomous
rch hook test                          # autonomous
rch diagnose "cargo build --release"   # autonomous
rch hook install                       # (authorize first) — writes ~/.claude/settings.json
```

### Sync/transfer fails under active target churn

**Cause:** build artifacts changing during rsync.

```bash
# Editing ~/.config/rch/config.toml [transfer].exclude_patterns and reloading
# both mutate config/daemon — (authorize first):
rch daemon reload                 # (authorize first)
rch config show --sources         # autonomous
```

Also inspect the worker directly:

```bash
ssh ubuntu@<host> 'df -h / /tmp'
ssh ubuntu@<host> 'du -sh /tmp/rch-* /tmp/rch_target_* 2>/dev/null | sort -h'
```

If cleanup is needed, verify inactivity first:

```bash
ssh ubuntu@<host> 'sudo lsof +D /tmp/rch_target_<name>'
```

If the directory is inactive, prefer targeted stale-artifact cleanup over broad cache deletion.

### Sync fails with `Permission denied` or `Operation not permitted` inside `/data/projects/<repo>`

**Cause:** the canonical mirror on the worker is not writable by the SSH user. This commonly happens when a repo under `/data/projects` was created or updated as `root`.

Check:

```bash
ssh ubuntu@<host> "stat -c '%U:%G %a %n' /data/projects/<repo>"
```

Fix (authorize first): this is a remote privileged command. Report the failing
`stat` output and the exact command to the caller and get explicit authorization
before running it — remote `sudo` is never autonomous.

```bash
# after explicit caller authorization only:
ssh ubuntu@<host> 'sudo chown -R ubuntu:ubuntu /data/projects/<repo> && sudo chmod 775 /data/projects/<repo>'
```

Then retry:

```bash
rch exec -- cargo check --workspace --all-targets
```

### `rch exec` fails open for workdirs outside `/data/projects`

**Cause:** canonical-root normalization rejects workdirs outside the configured project root.

Symptoms include errors mentioning `input resolves outside canonical root`.

Fix:

```bash
pwd
rch diagnose --dry-run "cargo build --release"
```

Then run the build from a workspace under `/data/projects`. If you need a clean copy for testing, stage it under `/data/projects/<temp-repo>` instead of `/tmp/<temp-repo>`.

### Worker shows storage pressure even after cleanup

**Cause:** telemetry lag, large ballast allocation, or active live build churn.

Check:

```bash
rch status --workers --jobs
ssh ubuntu@<host> 'df -h / /tmp && free -h'
ssh ubuntu@<host> 'journalctl -u sbh -n 50 --no-pager'
```

Interpretation:

- If `df` is healthy but `rch status` still warns, give telemetry a minute and refresh.
- If `/tmp` is healthy but `/` is still low, inspect large project `target_*` trees under `/data/projects`.
- If `sbh` is active but repeatedly logging `scan channel saturated` or `scan timed out`, inspect stale build artifacts and verify the host is running the current `sbh` binary and the narrowed worker config.

### Path dependency missing remotely (`../.../Cargo.toml`)

**Cause:** required sibling repositories are not available in worker topology.

```bash
rch diagnose --dry-run "cargo test --workspace"
rch exec -- env CARGO_TARGET_DIR=/tmp/rch_target_<name> cargo check --workspace --all-targets
```

Then ensure sibling repos exist on workers under canonical roots and retry.

---

## Debug Mode

```bash
RCH_LOG_LEVEL=debug rch check
RCH_LOG_LEVEL=debug rch diagnose "cargo test --workspace"
RCH_LOG_LEVEL=debug rch exec -- cargo check --workspace --all-targets
```

Protocol-level hook test:

```bash
RCH_LOG_LEVEL=debug printf '%s\n' \
  '{"tool_name":"Bash","tool_input":{"command":"cargo check"}}' | rch
```

---

## Safe Reset Sequence

The `rch daemon restart -y` that opens this sequence is a daemon-lifecycle
mutation — get explicit caller authorization before running it; the remaining
steps are read-only.

```bash
rch daemon restart -y          # authorize first
rch config validate
rch config doctor
rch workers probe --all
rch hook status
rch hook test
rch check
```

If still failing, capture artifacts for escalation:

```bash
rch doctor --json > /tmp/rch-doctor.json
rch --json daemon status > /tmp/rch-daemon-status.json
rch --json workers probe --all > /tmp/rch-workers-probe.json
```

---

## Reading `rch status` Output Correctly

`rch status` (and `rch check`) can simultaneously show `✓ RCH is ready (9/9 workers healthy)` AND a list of `[warning] Circuit opened for worker '<id>'` alerts. **The alerts are informational** — circuit breakers are self-healing once the worker is healthy and the half-open probe succeeds. Don't over-react.

**Wrong:** "I see warnings — better restart the daemon and reload the config."

**Right:** `rch workers probe --all && rch status --workers --jobs` — the alert clears within the next status refresh.

If a circuit doesn't auto-clear after 60 seconds and the underlying probe is healthy, then there's a real bug; capture `rch --json daemon status | jq '.data.circuit_breakers'` and `rch daemon logs -n 100`.

---

## Daemon Version Drift After Upgrade

**Symptom:** New rch CLI features behave inconsistently; `rch --version` differs from the daemon's reported version.

**Diagnosis is autonomous; the restart is a daemon-lifecycle mutation — authorize first.**

The daemon's running version is reported by `rch --json status` at `.data.daemon.daemon.version` (the `rch --json daemon status` endpoint deliberately returns only running/socket/uptime — not version). Compare (read-only):

```bash
rch --version | awk '{print $2}'
rch --json status | jq -r '.data.daemon.daemon.version'
# If they differ, restart AFTER explicit caller authorization:
rch daemon restart -y                                    # drains in-flight builds gracefully
rch --json status | jq -r '.data.daemon.daemon.version'  # confirm equal
```

`rch daemon restart -y` is the **documented upgrade path**. It drains active builds before stopping. The `-y` skips the interactive prompt — but it does *not* skip the drain, and it does *not* skip the caller-authorization requirement.

If a worker shows mismatched binary version after a host upgrade, diagnose with
`status`/`verify` (read-only), then deploy only after explicit caller
authorization — fleet deploy mutates every worker binary:

```bash
rch fleet status              # human-readable per-worker status (read-only)
rch fleet verify              # compare installed vs expected (read-only)
# after explicit caller authorization only:
rch fleet deploy --canary 25 --canary-wait 60 --verify
rch fleet deploy --verify
```

---

## Telemetry Corruption

**Symptom:** Recurring `RCH-E507`, `Telemetry database integrity check failed`, empty `rch speedscore --history`, daemon log lines mentioning `database disk image is malformed`.

**Fix (authorize first):** stopping the daemon, moving `~/.local/share/rch/telemetry/telemetry.db*` aside, and restarting is a daemon-lifecycle mutation plus local file removal — get explicit caller authorization first. Telemetry is derived data; you lose history but nothing operational.

---

## "Why did my command run locally?" (Silent Fail-Open)

**Symptom:** `rch hook status` says installed; `rch exec` works in isolation; but a particular `cargo build` invocation runs locally without the rch wrapper. No `[RCH] local (...)` line appears because `RCH_VISIBILITY=none` is set, or because the hook never engaged at all.

**Self-fix:**

1. Force visibility: `RCH_VISIBILITY=verbose <your-command>`. If you now see `[RCH] local (...)`, follow `references/FAIL_OPEN.md` to map the reason to a fix.
2. If still no `[RCH]` line, the hook never fired. Probe the protocol directly with a hand-crafted request (see the Wire-Level Hook Protocol section of `references/MACHINE_INTROSPECTION.md`):
   ```bash
   printf '%s\n' '{"tool_name":"Bash","tool_input":{"command":"<your-command>"}}' | rch
   ```
   If stdout is empty, the classifier is rejecting your command. Common causes: shell pipe (`cargo build | tee log`), backgrounded with `&`, env-prefixed in an unusual form. Restructure or use `rch exec -- <cmd>` directly.
3. If the hook fires but the command still runs locally, the rewrite isn't being honored — check that `~/.claude/settings.json` has the right hook command path (reading it is autonomous; `rch hook install` re-resolves it but writes that file — **(authorize first)**).

See `references/FAIL_OPEN.md` for the full taxonomy.

---

## See Also

- `references/FAIL_OPEN.md` — the canonical guide for `[RCH] local (...)` reasons
- `references/ERROR_CODES.md` — the full RCH-Exxx catalog
- `references/RECOVERY_PLAYBOOKS.md` — symptom→fix in ≤90s
- `references/MACHINE_INTROSPECTION.md` — JSON/schema/capability surfaces, wire-level hook protocol
- `references/CONFIGURATION.md` — config keys and precedence
- `references/WORKERS.md` — worker inventory and health

# Worker Management

## Contents

- [Worker Lifecycle](#worker-lifecycle)
- [Add a Worker Manually](#add-a-worker-manually)
- [Drain / Disable / Enable](#drain--disable--enable)
- [Toolchain and Binary Management](#toolchain-and-binary-management)
- [Fleet-Level Rollout Commands](#fleet-level-rollout-commands)
- [Worker Selection Notes](#worker-selection-notes)
- [SSH Verification Shortcuts](#ssh-verification-shortcuts)

> **Authority:** this is a command catalog, not a licence to run it. Only the
> read-only probes here (`rch workers list`, `rch workers probe`, `rch fleet
> status`, `rch fleet verify`) are autonomous. Everything that adds, sets up,
> drains, disables, enables, syncs a toolchain to, or deploys a binary to a
> worker — and every `rch fleet deploy|rollback` — is a host mutation requiring
> explicit caller authorization first (see `FAIL_OPEN.md` §"Autonomous
> Remediation Envelope"). The `--dry-run` variants shown are the read-only way to
> preview before you ask.

## Worker Lifecycle

### 1) Discover and add workers

```bash
rch workers discover
rch workers discover --probe
rch workers discover --add --yes
```

### 2) Complete setup

```bash
rch workers setup --all
```

This performs the standard bootstrap path (binary/toolchain setup, validation) for configured workers.

### 3) Validate runtime health

```bash
rch workers list --speedscore
rch workers probe --all
rch workers capabilities --refresh
rch check
```

---

## Add a Worker Manually

Edit `~/.config/rch/workers.toml`:

```toml
[[workers]]
id = "new-worker"
host = "203.0.113.20"
user = "ubuntu"
identity_file = "~/.ssh/new_worker_ed25519"
total_slots = 16
priority = 90
tags = ["rust", "bun"]
```

Then validate and setup:

```bash
rch config validate
rch workers probe new-worker
rch workers setup new-worker
```

---

## Drain / Disable / Enable

Use these for maintenance windows and incident isolation.

```bash
rch workers drain <worker> -y
rch workers disable <worker> --reason "maintenance" --drain -y
rch workers enable <worker>
```

State model:

- `HEALTHY`: accepting jobs
- `DRAINING`: finishing active jobs, no new jobs
- `DRAINED`: idle and not accepting jobs
- `DISABLED`: explicitly offline from scheduler

---

## Toolchain and Binary Management

```bash
rch workers sync-toolchain --all
rch workers deploy-binary --all
```

Use `--dry-run` before broad changes:

```bash
rch workers sync-toolchain --all --dry-run
rch workers deploy-binary --all --dry-run
```

---

## Fleet-Level Rollout Commands

```bash
rch fleet status
rch fleet deploy --verify
rch fleet deploy --canary 25 --canary-wait 60 --verify
rch fleet rollback --verify
rch fleet history --limit 20
```

---

## Worker Selection Notes

Selection favors availability and execution quality signals (slot capacity, health, and policy strategy).

Operational guidance:

- Keep `total_slots` realistic for CPU and memory limits.
- Prefer explicit `priority` shaping for known fast/reliable workers.
- Drain before disruptive operations.
- Keep worker toolchains synchronized to avoid fallback churn.

---

## SSH Verification Shortcuts

Single worker:

```bash
rch workers probe <worker>
```

All workers with machine-readable output:

```bash
rch --json workers probe --all
```

If probes fail:

1. Verify `identity_file` exists and permissions are restrictive.
2. Verify worker host reachability and SSH service.
3. Re-run `rch workers setup <worker>` after connectivity is restored.