codex-qa
QA the omo Codex Light edition (lazycodex / packages/omo-codex) itself, in strict isolation so ONLY our plugin is exercised, never the user's real ~/.codex. The first-party method drives the real `codex app-server` against an isolated CODEX_HOME plus a LOCAL mock model (no real A
Install
npx skills add https://github.com/code-yeongyu/oh-my-openagent/tree/dev/.agents/skills/codex-qa
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install code-yeongyu-oh-my-openagent@llmmart
git clone https://github.com/code-yeongyu/oh-my-openagent.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole code-yeongyu/oh-my-openagent collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Codex QA
QA the omo Codex Light edition (packages/omo-codex/, shipped as lazycodex). We
exercise OUR plugin in a REAL Codex while touching nothing of the user's setup:
an isolated CODEX_HOME + a local mock model means no real API call and the real
~/.codex is never read or written. Each helper script ships a --self-test
that asserts its scenario against the live machine, so the scripts are both the
QA tools and their own regression checks.
Verified against codex-cli 0.140.0 (node, jq, tmux, bun on macOS). Confirm with
codex --version; check a flag with codex <cmd> --help.
Golden rules (read before running anything)
- QA ONLY our plugin. Everything that spawns codex uses an isolated
CODEX_HOME(created bycqa_mk_isolated_home) and a LOCAL mock model provider (cqa_start_mock). Never QA against the real~/.codex, never hit a real model API. The bundled scripts enforce this; if you run codex by hand,export CODEX_HOME="$(mktemp -d)/codex"; mkdir -p "$CODEX_HOME"FIRST (a setCODEX_HOMEmust already exist or codex hard-errors). - Prove the real home stayed clean. Every script shasums
~/.codex/config.tomlbefore and after and asserts it is unchanged. If you script by hand, do the same. - The interactive
codexis a shell function that injects--profile quotio. Bash scripts bypass it and get the real binary; never rely on the interactive alias. See references/isolation.md. - The first-party way to prove a hook fired is the app-server notification
stream (
hook/started/hook/completed), not log scraping. See references/app-server.md. - The captured JSON / pane IS the evidence — write it under
.omo/evidence/<YYYYMMDD>-<slug>/(no evidence file == the QA did not happen). That directory is gitignored: the files stay local, the PR body carries the summary and decisive excerpts, and nothing under it is ever committed.
Setup
cd <this-skill-dir> # .agents/skills/codex-qa
bash scripts/lib/common.sh --self-check # confirm deps + isolation harness
Docker is the default QA surface. Run this QA inside a disposable container
that has the latest codex and a copy of your config, with the host ~/.codex
untouched: script/agent/qa-docker.sh (see references/docker-qa.md).
The local scripts below are the fallback for when Docker is unavailable or on
Windows.
Router: pick your case
| You need to… | Run | Deep dive |
|---|---|---|
| Prove a plugin hook fires in a LIVE Codex turn (first-party) | scripts/app-server-drive.sh --plugin |
app-server.md |
| Prove the app-server driver itself works (no plugin, fast) | scripts/app-server-drive.sh --self-test |
app-server.md |
| Install the LOCAL build into an isolated home + assert it landed | scripts/install-verify.sh --self-test |
install-verify.md |
| Pin ONE component's hook logic deterministically (no codex) | scripts/hook-unit-probe.sh --self-test |
components-hooks.md |
| Smoke the real TUI under tmux (boots, renders, survives) | scripts/tui-smoke.sh --self-test |
logging-debug.md |
| Watch runtime logs while QAing | (see reference; RUST_LOG / logs DB / /debug-config) |
logging-debug.md |
Scripts index (each is its own regression test)
| Script | --self-test asserts |
|---|---|
scripts/lib/common.sh --self-check |
deps present; isolated CODEX_HOME is created inside a sandbox and auto-removed on exit; mock model serves the Responses SSE; real ~/.codex unchanged |
scripts/app-server-drive.sh |
--self-test: a bare turn completes and the mock assistant text comes back. --plugin: installs local omo, drives a turn, and asserts hook/completed for sessionStart,userPromptSubmit |
scripts/install-verify.sh |
local omo installs into the isolated home; config.toml enables omo@sisyphuslabs; component bins + agent TOMLs linked in the sandbox; real ~/.codex unchanged |
scripts/hook-unit-probe.sh |
the ultrawork component injects <ultrawork-mode> on an ulw UserPromptSubmit (also a manual --component/--event mode) |
scripts/tui-smoke.sh |
the real codex TUI boots in the isolated home, renders, and survives (no early exit); captures the pane |
To tell a dev dogfood build apart from a published one on a REAL ~/.codex (NOT the isolated QA home), the repo ships bun run install:codex-dev, which stamps the plugin version as dev — visible as the (OmO dev) hook-status prefix every turn and as a [DEV] badge in omo get-local-version. Use it to confirm which build is loaded during manual dogfooding; it writes to the real home, so it is NEVER part of the isolated QA flow above.
When TUI visual QA evidence is needed, follow
docs/reference/web-terminal-visual-qa.md: render the TUI through the real
xterm.js web terminal - NEVER the tmux capture-pane frame, which degrades
color and CJK width:
node script/qa/web-terminal-visual-qa.mjs --title "Codex TUI QA" \
--command "codex" --input "{Enter}" \
--evidence-dir .omo/evidence/<slug>/codex-web-terminal
The helper runs a real pty, renders it in xterm.js under Chrome, and writes
terminal.txt, terminal-ansi.txt, terminal.png (true color), and
metadata.json (--from-file <capture.ansi> replays a saved raw stream). Use
that artifact set for TUI visual QA; use app-server-drive.sh --plugin for
assertion-grade hook behavior.
Match QA to your change scope
- Component / hook logic (
packages/omo-codex/plugin/components/*):hook-unit-probe.shfor the exact stdout, THENapp-server-drive.sh --pluginto prove the live wiring. See components-hooks.md. - Installer / config.toml (
packages/omo-codex/src/install/*):install-verify.sh. - Anything that affects a live session (hooks, agents, MCP wiring):
app-server-drive.sh --plugin, andtui-smoke.sh --pluginif the TUI path matters.
Capturing evidence
ev=".omo/evidence/$(date +%Y%m%d)-codex-qa-<slug>"; mkdir -p "$ev"
bash scripts/app-server-drive.sh --plugin > "$ev/app-server-drive.json" 2>&1
bash scripts/install-verify.sh --self-test > "$ev/install-verify.txt" 2>&1
On /debugging
There is no /debugging command in Codex. To observe a run: the app-server
notification stream (above), RUST_LOG=debug on the app-server's stderr, the
logs SQLite under $CODEX_HOME, the TUI's /debug-config, and the
codex debug … subcommands. See logging-debug.md.
Files (oh-my-openagent)
-
references
-
app-server.md 3 KB
# Codex app-server — the first-party QA channel The app-server is how a host (IDE, our QA harness) drives Codex programmatically. We speak its protocol directly so we can read the **structured notification stream** — including `hook/started` / `hook/completed`, which is the authoritative proof that an omo plugin hook fired in a live turn. Verified against `codex-cli 0.139.0`. Source citations are `path:line` under `../codex/codex-rs/`. ## Transport & framing - Start with `codex app-server` (no subcommand runs the server). Implemented by the `codex-app-server` crate; entry `app-server/src/lib.rs:429`. - Default transport is **stdio**, framing is **newline-delimited JSON** (one message per line) — `app-server-transport/src/transport/stdio.rs:46-88`. - It is NOT standard JSON-RPC 2.0: there is **no `"jsonrpc"` field**. Requests are `{id, method, params}`; notifications are `{method, params}` (`app-server-protocol/src/jsonrpc_lite.rs`). Field names are camelCase. Confirm the method set for the installed binary: ```bash codex app-server generate-json-schema --out "$(mktemp -d)" # ClientRequest.json / ServerNotification.json ``` ## Drive one turn (the sequence the driver uses) ```jsonc {"id":1,"method":"initialize","params":{"clientInfo":{"name":"codex-qa","version":"0.1.0"},"capabilities":{"experimentalApi":true,"requestAttestation":false}}} {"method":"initialized"} // notification, REQUIRED, no id {"id":2,"method":"thread/start","params":{"cwd":"/abs/workdir"}} // -> result.thread.id {"id":3,"method":"turn/start","params":{"threadId":"<id>","input":[{"type":"text","text":"say hello"}]}} // -> result.turn.id ``` Read stdout line-by-line and collect: - `hook/started` / `hook/completed` — `params.run.eventName` (e.g. `sessionStart`, `userPromptSubmit`, `stop`), `params.run.status` (`running` → `completed`), `params.run.source` (`plugin`). **This is the plugin-fired proof.** - `item/completed` where `item.type == "agentMessage"` — `item.text` is the assistant message. - `turn/completed` — stop when `turn.status == "completed"` (or `"failed"` with `turn.error`) for your `turnId`. `scripts/lib/app-server-client.mjs` implements exactly this and exits non-zero unless the turn completes and every `EXPECT_HOOK` event reaches `completed`. ## Why a mock model A turn needs a model. We point a custom `model_provider` at the local `scripts/lib/mock-model.mjs` (OpenAI Responses SSE), so the turn runs with NO real API call. A non-OpenAI provider needs no auth (`requires_openai_auth` defaults false). The driver injects the provider via `-c` overrides — see [isolation.md](./isolation.md). ## Observed result on 0.139.0 With omo installed in an isolated `CODEX_HOME`, one `ulw: say hello` turn emits `hook/*` for `sessionStart` (rules, telemetry, bootstrap, auto-update), `userPromptSubmit` (rules, ultrawork, ulw-loop), and `stop` (start-work-continuation), then the mock assistant message and `turn/completed`. `scripts/app-server-drive.sh --plugin` asserts this end to end. -
components-hooks.md 2.7 KB
# omo-codex components → events → observable proof The plugin's hook wiring lives in `packages/omo-codex/plugin/hooks/hooks.json`. Each hook runs `node "${PLUGIN_ROOT}/components/<c>/dist/cli.js" hook <event>`, reading the event JSON on stdin and writing zero-or-one line of JSON on stdout. Use this table to pick what to assert. Two proof tiers: - **Unit** (`hook-unit-probe.sh`): pipe a synthetic event into a component's `dist/cli.js`, assert stdout/disk. Deterministic, no codex process. - **Live** (`app-server-drive.sh --plugin`): drive a real turn, assert the `hook/completed` notification fires for the event. Proves Codex WIRES it. | Component | Codex events | Observable proof it fired | |---|---|---| | `rules` | SessionStart; UserPromptSubmit; PostToolUse `apply_patch`; PostCompact | `hookSpecificOutput.additionalContext` (rule body) on stdout; session cache at `$PLUGIN_DATA/sessions/<id>.json` | | `ultrawork` | UserPromptSubmit | stdout `additionalContext` contains `<ultrawork-mode>` **only** when prompt matches `/ultrawork|ulw/i`; empty otherwise | | `ulw-loop` | UserPromptSubmit; PreToolUse `create_goal` | steer JSON on a steer prompt; `permissionDecision:"deny"` when `create_goal` carries keys beyond `objective` | | `comment-checker` | PostToolUse (write/edit/apply_patch) | warning text on stdout when an edited file has banned comments; empty when clean | | `lsp` | PostToolUse (write/edit/apply_patch); PostCompact | LSP diagnostics as `additionalContext` for mutated files | | `start-work-continuation` | Stop; SubagentStop | `{"decision":"block","reason":...}` **only** when a continuation/boulder state exists for the session | | `git-bash` | PreToolUse `Bash`; PostCompact | **Windows-only**: reminder + marker `$PLUGIN_DATA/git-bash-reminder/<id>.seen`; no-op elsewhere | | `telemetry` | SessionStart | empty stdout; side effect is a PostHog event (or a diagnostic file on failure) | | `bootstrap` | SessionStart | `BOOTSTRAP_RESTART_NOTICE` additionalContext on first run (gated on `PLUGIN_ROOT`+`PLUGIN_DATA`) | Many components are **conditional** (only emit on a matching prompt / OS / state). For a stable always-fires assertion, prefer: - Live: `sessionStart` and `userPromptSubmit` `hook/completed` (several components wire them, so the events always fire). `app-server-drive.sh --plugin` defaults to `--expect sessionStart,userPromptSubmit`. - Unit: `ultrawork` on an `ulw` prompt deterministically injects `<ultrawork-mode>`. `hook/*` notification eventNames are camelCase (`sessionStart`, `userPromptSubmit`, `postToolUse`, `stop`, …); the hooks.json matchers use snake_case (`session_start`, `user_prompt_submit`, …). The component CLI takes the kebab form (`hook user-prompt-submit`). -
docker-qa.md 2.6 KB
# Docker QA (default path) Run Codex QA inside a DISPOSABLE container so the real `~/.codex` is never touched and you always test against the latest codex. The container is the sandbox: latest released codex (and opencode) are baked in, a COPY of your config is loaded, and the container is removed on exit (`docker run --rm`). This is the DEFAULT; fall back to running the scripts locally (see SKILL.md) only when Docker is unavailable or on Windows. ## Use it `qa-docker.sh` brings up a disposable box (builds `omo-dev` then `omo-qa` on first use, reused after) and drops you into it. From the repo root: ```bash # drive codex via the FIRST-PARTY app-server (no acp): a real turn in the box script/agent/qa-docker.sh codex # fallback: the interactive codex TUI in the box (uses your mounted config) script/agent/qa-docker.sh codex --tui # a shell inside the box: codex (and opencode) are on PATH script/agent/qa-docker.sh script/agent/qa-docker.sh shell # one-off command, or a codex-qa self-test inside the box: script/agent/qa-docker.sh exec codex --version script/agent/qa-docker.sh exec bash .claude/skills/codex-qa/scripts/tui-smoke.sh --self-test script/agent/qa-docker.sh --clean # remove the QA images ``` `omo-qa` is `omo-dev` (`.devcontainer/Dockerfile`) plus the latest `@openai/codex` and `opencode-ai` npm packages and `sqlite3 jq curl rsync`. ## Isolation still applies inside The codex-qa scripts already isolate via an mktemp `CODEX_HOME` and a local mock model (no real API call). In Docker that runs inside a throwaway container too, so there are two layers: the scripts never touch the mounted real `~/.codex`, and the container is discarded on exit. `qa-docker.sh` mounts `~/.codex` READ-ONLY at `/mnt/host/codex`; the entrypoint copies it into the container's writable home for any case that wants the real config. The host `~/.codex` (including `config.toml`) is never written. ## Credentials codex-qa uses a mock model, so no real key is needed for the first-party hook proof. For runs that do need auth, provide it at run time only: a gitignored `.env` / `.env.local`, Codespaces secrets, or the devcontainer `remoteEnv` passthrough - never baked into the image. ## Fallback: local / Windows `qa-docker.sh` exits 3 with guidance when Docker is unavailable or on Windows; run the scripts directly on the host there (they isolate via mktemp `CODEX_HOME`). Windows has no Docker QA path here by design. ## Cleanup Each run auto-removes its container (`--rm`). The `omo-dev` / `omo-qa` images persist for fast re-runs; drop them with `script/agent/qa-docker.sh --clean`. -
install-verify.md 2.6 KB
# Installing the LOCAL omo build into an isolated CODEX_HOME QA must run THIS repo's local build, not the published package. The installer respects `CODEX_HOME` for everything, so a non-default home is fully self contained. ## Command ```bash export CODEX_HOME="$(mktemp -d)/codex"; mkdir -p "$CODEX_HOME" # must exist first export OMO_DISABLE_POSTHOG=1 OMO_CODEX_DISABLE_POSTHOG=1 export OMO_CODEX_PROJECT="$(mktemp -d)/project" # keep project-local cleanup off your tree node packages/omo-codex/scripts/install-local.mjs install ``` `cqa_install_local_omo` wraps this (logs to `$CQA_HOME_ROOT/install.log`). ## What it writes (all under CODEX_HOME) Source: `packages/omo-codex/src/install/install-codex.ts`. 1. Builds + copies the plugin to `$CODEX_HOME/plugins/cache/sisyphuslabs/omo/<version>/` (then `npm ci --omit=dev`). 2. Links component bins into `$CODEX_HOME/bin/omo-*` (8: comment-checker, git-bash-hook, lsp, rules, start-work-continuation, telemetry, ultrawork, ulw-loop). 3. Links agent TOMLs into `$CODEX_HOME/agents/*.toml`. 4. Writes a marketplace snapshot under `$CODEX_HOME/.tmp/marketplaces/sisyphuslabs/`. 5. Edits `$CODEX_HOME/config.toml`: enables `[plugins."omo@sisyphuslabs"]`, the `[marketplaces.sisyphuslabs]` local source, `[features]` (plugins/plugin_hooks/multi_agent/child_agents_md), and one `[hooks.state."omo@sisyphuslabs:hooks/hooks.json:<event>:i:j"] trusted_hash` per hook (so Codex trusts them — no `--dangerously-bypass-hook-trust` needed for the app-server turn). ## Assertions (what install-verify.sh checks) ```bash ls "$CODEX_HOME"/plugins/cache/sisyphuslabs/omo/*/ # cache present grep -A2 '\[plugins."omo@sisyphuslabs"\]' "$CODEX_HOME/config.toml" | grep 'enabled = true' ls "$CODEX_HOME"/bin/omo-* # component bins ls "$CODEX_HOME"/agents/*.toml # agent links ``` Plus the cross-cutting invariant every script enforces: the real `~/.codex/config.toml` shasum is unchanged. ## Notes - The only thing outside CODEX_HOME is the `omo` runtime wrapper, which targets the repo's `dist/cli/index.js` (the CLI ships from the repo). `dist/cli/index.js` must exist (run `bun run build` if missing) or that link is skipped. - Cleanup: for an isolated home just `rm -rf "$CODEX_HOME"` (the harness does this on exit). For a normal home, `node packages/omo-codex/scripts/install-local.mjs uninstall`. - `bun run test:codex` is the hermetic unit gate (installer/config/component build) and does NOT launch a real codex — this skill is what proves the live session. -
isolation.md 2.5 KB
# Isolation — QA ONLY our plugin, never the user's real Codex The whole point of this skill: exercise the omo plugin in a real Codex without reading or writing the user's `~/.codex`, and without a real model API call. Two levers do all the work. ## Lever 1 — an isolated `CODEX_HOME` `CODEX_HOME` is Codex's master state root: `config.toml`, `auth.json`, sessions, the state SQLite, plugins, and logs all hang off it (`utils/home-dir/src/lib.rs`). Point it at a fresh temp dir and Codex reads/writes nothing else. Gotcha: when `CODEX_HOME` is set it **must already exist** or Codex hard-errors. `cqa_mk_isolated_home` creates it first. `cqa_mk_isolated_home` also exports: - `OMO_CODEX_PROJECT` + `QA_CWD` → a sandbox project dir, so the installer's project-local cleanup and the TUI's cwd never touch your real tree. - `CODEX_LOCAL_BIN_DIR=$CODEX_HOME/bin` → component bins land in the sandbox. (Even without this, a non-default `CODEX_HOME` already routes bins to `$CODEX_HOME/bin`; with the DEFAULT home they would leak to `~/.local/bin`.) - `OMO_DISABLE_POSTHOG=1` + `OMO_CODEX_DISABLE_POSTHOG=1` → no install/telemetry network call. Proof it stayed clean: `cqa_guard_real_home` shasums `~/.codex/config.toml` before, `cqa_assert_real_home_unchanged` re-checks after. Every script runs it. ## Lever 2 — a local mock model (no real API) Codex must reach a model to run a turn. Instead of OpenAI, we run `scripts/lib/mock-model.mjs` (OpenAI **Responses** SSE) on localhost and point a custom provider at it via `-c` overrides: ``` -c model="mock-model" -c model_provider="mock_provider" -c model_providers.mock_provider.name="codex-qa mock" # REQUIRED: empty name fails config load -c model_providers.mock_provider.base_url="http://127.0.0.1:<PORT>/v1" -c model_providers.mock_provider.wire_api="responses" -c approval_policy="never" -c sandbox_mode="read-only" ``` A non-OpenAI provider needs no key/auth, so there is no real egress. `-c` overrides beat any value in `config.toml`, so even a misconfigured isolated home still lands on the mock. ## The `codex` shell-function trap The interactive shell here wraps `codex` in a function that injects `--profile quotio`. That breaks non-runtime subcommands like `generate-json-schema` and would point a turn at the quotio provider. **Bash scripts do not inherit that function**, so `codex` inside a `#!/usr/bin/env bash` script is the real binary on PATH. `cqa_codex_bin` resolves it explicitly; never rely on the interactive alias. Combined with the isolated `CODEX_HOME`, the real quotio config is never read. -
logging-debug.md 2.6 KB
# Observing Codex at runtime (logs + debug surfaces) The intent "use `/debugging` to watch logs while QAing" maps to the surfaces below. Codex has **no `/debugging` command**; these are the real ways to observe a run. ## 1. The notification stream (best signal for plugin QA) When you drive via the app-server, the stdout stream IS the live trace: `hook/started` / `hook/completed`, `item/*`, `mcpServer/*`, `error`, `warning`. This is structured and assertion-grade — prefer it over scraping text logs. `scripts/app-server-drive.sh` captures it; the JSON summary it prints is the evidence. ## 2. app-server stderr (`RUST_LOG`) The app-server writes tracing logs to **stderr** (not a file), filtered by `RUST_LOG` (`app-server/src/lib.rs:638-651`). Turn it up and capture: ```bash RUST_LOG=info # or debug LOG_FORMAT=json # optional: machine-parseable lines ``` The driver inherits the env; raise `RUST_LOG` before invoking it to see the plugin/hook subprocess accounting on stderr (surfaced in the summary's `stderrTail`). ## 3. The logs SQLite DB The app-server also writes structured logs to a SQLite DB under `$CODEX_HOME` (alongside `state_5.sqlite`). Query it post-run for a durable record: ```bash ls "$CODEX_HOME"/*.sqlite ``` ## 4. TUI `/debug-config` Inside the TUI, the slash command is **`/debug-config`** (NOT `/debugging`) — "show config layers and requirement sources" (`tui/src/slash_command.rs:107`). Useful to confirm which config layer enabled the plugin. Drive it under tmux: ```bash tmux send-keys -t <sess> "/debug-config" Enter tmux capture-pane -t <sess> -p -S - ``` For TUI visual QA evidence, render the live TUI through the real xterm.js web terminal instead of scraping the tmux pane (`tmux capture-pane` degrades color and CJK width): ```bash node script/qa/web-terminal-visual-qa.mjs --title "Codex TUI /debug-config" \ --command "codex" --input "/debug-config" --input "{Enter}" \ --evidence-dir .omo/evidence/<slug>/codex-debug-config-web-terminal ``` Attach the resulting `terminal.png` and keep `metadata.json` with the cleanup receipt. The tmux pane above proves the config text; this PNG proves the visual TUI surface in true color. ## 5. `codex debug` subcommands `codex debug models` (raw model catalog), `codex debug prompt-input` (the model-visible prompt list), and `codex debug app-server …` (a built-in app-server driver). Run them against the isolated `CODEX_HOME` for ad-hoc inspection. ## Component-level logs `rules` emits phase/timing lines to stderr under `NODE_DEBUG=codex-rules`. Most components prove themselves through their stdout `additionalContext` or a disk artifact — see [components-hooks.md](./components-hooks.md).
-
-
scripts
-
lib
-
app-server-client.mjs 6.3 KB · in bundle
-
app-server-client.test.js 1.9 KB
import { describe, expect, it } from "bun:test"; import { parseExpectedHooks, summarizeRun } from "./app-server-client.mjs"; describe("app-server-client summary", () => { it("#given a completed expected event also has a failed hook run #when summarized #then the QA run fails", () => { const summary = summarizeRun({ turnStatus: "completed", assistantText: "ok", threadId: "thread", turnId: "turn", expectHook: ["sessionStart", "userPromptSubmit"], hooks: [ { method: "hook/completed", eventName: "sessionStart", status: "completed", source: "plugin" }, { method: "hook/completed", eventName: "userPromptSubmit", status: "completed", source: "plugin" }, { method: "hook/completed", eventName: "userPromptSubmit", status: "failed", source: "plugin" }, ], stderr: "", }); expect(summary.ok).toBe(false); expect(summary.missingHooks).toEqual([]); expect(summary.failedHooks).toEqual([ { method: "hook/completed", eventName: "userPromptSubmit", status: "failed", source: "plugin" }, ]); }); it("#given all expected hooks complete #when summarized #then the QA run passes", () => { const summary = summarizeRun({ turnStatus: "completed", assistantText: "ok", threadId: "thread", turnId: "turn", expectHook: ["sessionStart", "userPromptSubmit"], hooks: [ { method: "hook/completed", eventName: "sessionStart", status: "completed", source: "plugin" }, { method: "hook/completed", eventName: "userPromptSubmit", status: "completed", source: "plugin" }, ], stderr: "", }); expect(summary.ok).toBe(true); expect(summary.failedHooks).toEqual([]); }); it("#given a comma-separated expectation #when parsed #then whitespace and empties are ignored", () => { expect(parseExpectedHooks(" sessionStart, ,userPromptSubmit ")).toEqual(["sessionStart", "userPromptSubmit"]); }); }); -
common.sh 7.1 KB
#!/usr/bin/env bash # common.sh - shared helpers for codex-qa scripts. # # Source it from a sibling script: # SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # . "$SCRIPT_DIR/lib/common.sh" # # SAFETY MODEL (read this): # - We QA ONLY our plugin, never the user's real codex. Everything that # spawns codex runs against an ISOLATED CODEX_HOME (cqa_mk_isolated_home) # plus a LOCAL mock model provider (cqa_start_mock) - so there is no real # API call and the real ~/.codex is never read or written. # - cqa_guard_real_home snapshots the real ~/.codex/config.toml up front; # cqa_assert_real_home_unchanged proves QA never touched it. # - cqa_cleanup runs on EXIT and tears down the app-server, mock model, tmux # sessions, and every temp dir the helpers created. # # The interactive shell may wrap `codex` in a function that injects # `--profile`; bash scripts do not see that function, so `codex` here is the # real binary on PATH. We still resolve it explicitly via cqa_codex_bin. set -uo pipefail CQA_TMPDIRS=() CQA_PIDS=() CQA_TMUX_SESSIONS=() CQA_REAL_HOME_SUM="" cqa_log() { printf '%s\n' "$*" >&2; } cqa_pass() { printf 'PASS: %s\n' "$*"; } cqa_fail() { printf 'FAIL: %s\n' "$*" >&2; return 1; } # cqa_require <bin>... -> 0 if all present, else 1 (names the missing ones). cqa_require() { local missing=0 b for b in "$@"; do command -v "$b" >/dev/null 2>&1 || { cqa_log "missing dependency: $b"; missing=1; } done return "$missing" } # Absolute path of the REAL codex binary (bypasses any interactive shell # function/alias). Override with CODEX_BIN. cqa_codex_bin() { if [ -n "${CODEX_BIN:-}" ]; then printf '%s' "$CODEX_BIN"; return 0; fi command -v codex 2>/dev/null } cqa_real_codex_home() { printf '%s' "${HOME}/.codex"; } # Snapshot the real ~/.codex/config.toml so we can prove QA never touched it. cqa_guard_real_home() { local cfg; cfg="$(cqa_real_codex_home)/config.toml" if [ -f "$cfg" ]; then CQA_REAL_HOME_SUM="$(shasum "$cfg" 2>/dev/null | awk '{print $1}')" else CQA_REAL_HOME_SUM="ABSENT" fi } cqa_assert_real_home_unchanged() { local cfg now; cfg="$(cqa_real_codex_home)/config.toml" if [ -f "$cfg" ]; then now="$(shasum "$cfg" 2>/dev/null | awk '{print $1}')"; else now="ABSENT"; fi if [ "$now" = "$CQA_REAL_HOME_SUM" ]; then cqa_pass "real ~/.codex/config.toml unchanged ($now)" return 0 fi cqa_fail "real ~/.codex/config.toml CHANGED ($CQA_REAL_HOME_SUM -> $now)" } # Create an isolated CODEX_HOME and project dir, export the env that keeps the # run hermetic, and register the temp root for cleanup. Sets globals # CQA_HOME_ROOT / CODEX_HOME / OMO_CODEX_PROJECT / QA_CWD. # # IMPORTANT: call this DIRECTLY, never via $(...). A subshell would discard the # exports and the cleanup registration. cqa_mk_isolated_home() { local root; root="$(mktemp -d -t cqa-home.XXXXXX)" || return 1 CQA_TMPDIRS+=("$root") # CODEX_HOME must EXIST before codex launches, or codex hard-errors. mkdir -p "$root/codex" "$root/proj" export CQA_HOME_ROOT="$root" export CODEX_HOME="$root/codex" export OMO_CODEX_PROJECT="$root/proj" export QA_CWD="$root/proj" # never leak install bins or telemetry out of the sandbox export CODEX_LOCAL_BIN_DIR="$root/codex/bin" export OMO_DISABLE_POSTHOG=1 export OMO_CODEX_DISABLE_POSTHOG=1 } # Start the local mock model server. Sets CQA_MOCK_PID + exports MOCK_PORT. # Call DIRECTLY (not via $(...)) so the PID + export land in the caller. cqa_start_mock() { local lib_dir log; lib_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" log="$(mktemp -t cqa-mock.XXXXXX)"; CQA_TMPDIRS+=("$log") node "$lib_dir/mock-model.mjs" >"$log" 2>&1 & CQA_MOCK_PID=$!; CQA_PIDS+=("$CQA_MOCK_PID") local i port="" for i in $(seq 1 100); do port="$(awk '/MOCK_LISTENING/{print $2; exit}' "$log" 2>/dev/null)" [ -n "$port" ] && break kill -0 "$CQA_MOCK_PID" 2>/dev/null || { cqa_log "mock model died:"; cat "$log" >&2; return 1; } sleep 0.1 done [ -n "$port" ] || { cqa_log "mock model never reported a port"; return 1; } export MOCK_PORT="$port" } # Install THIS repo's local omo build into the isolated CODEX_HOME. Requires # cqa_mk_isolated_home first. REPO_ROOT defaults to the repo containing this skill. cqa_install_local_omo() { local repo="${REPO_ROOT:-}" if [ -z "$repo" ]; then repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." && pwd)" fi local installer="$repo/packages/omo-codex/scripts/install-local.mjs" [ -f "$installer" ] || { cqa_fail "installer not found: $installer"; return 1; } node "$installer" install >"$CQA_HOME_ROOT/install.log" 2>&1 } # Teardown everything the helpers created. Safe to call multiple times. cqa_cleanup() { local p s d for p in "${CQA_PIDS[@]:-}"; do [ -n "$p" ] && kill "$p" 2>/dev/null || true done for s in "${CQA_TMUX_SESSIONS[@]:-}"; do [ -n "$s" ] && tmux kill-session -t "$s" 2>/dev/null || true done for d in "${CQA_TMPDIRS[@]:-}"; do [ -n "$d" ] && rm -rf "$d" 2>/dev/null || true done CQA_TMPDIRS=(); CQA_PIDS=(); CQA_TMUX_SESSIONS=() } trap cqa_cleanup EXIT # ---- self-check ------------------------------------------------------------ # Run: bash scripts/lib/common.sh --self-check cqa__self_check() { local fails=0 if cqa_require codex node jq tmux; then cqa_pass "dependencies present (codex node jq tmux)" else cqa_log "FAIL: missing dependencies"; fails=$((fails+1)); fi local bin; bin="$(cqa_codex_bin)" if [ -n "$bin" ]; then cqa_pass "codex binary -> $bin" else cqa_log "FAIL: codex binary not found"; fails=$((fails+1)); fi cqa_guard_real_home # isolation + trap teardown: an inner shell creates a sandbox (DIRECTLY) and # exits; the EXIT trap must remove it. Pass the path out via a marker file. local marker root home marker="$(mktemp -t cqa-marker.XXXXXX)" bash -c '. "'"${BASH_SOURCE[0]}"'"; cqa_mk_isolated_home; printf "%s\n%s\n" "$CQA_HOME_ROOT" "$CODEX_HOME" > "'"$marker"'"' root="$(sed -n '1p' "$marker" 2>/dev/null)"; home="$(sed -n '2p' "$marker" 2>/dev/null)" rm -f "$marker" if [ -n "$root" ] && [ ! -d "$root" ]; then cqa_pass "isolated CODEX_HOME auto-removed on exit ($root)" else cqa_log "FAIL: sandbox not cleaned: '$root'"; fails=$((fails+1)); fi if [ -n "$home" ] && [ "$home" = "$root/codex" ]; then cqa_pass "CODEX_HOME points inside sandbox, not ~/.codex" else cqa_log "FAIL: CODEX_HOME not isolated ('$home')"; fails=$((fails+1)); fi # mock model: start it, confirm it serves the Responses SSE, then cleanup. cqa_mk_isolated_home if cqa_start_mock; then if curl -s -X POST "http://127.0.0.1:$MOCK_PORT/v1/responses" -d '{}' 2>/dev/null | grep -q 'response.completed'; then cqa_pass "mock model serves Responses SSE on :$MOCK_PORT" else cqa_log "FAIL: mock model did not return response.completed"; fails=$((fails+1)); fi else cqa_log "FAIL: mock model did not start"; fails=$((fails+1)); fi cqa_assert_real_home_unchanged || fails=$((fails+1)) if [ "$fails" -eq 0 ]; then cqa_pass "common.sh self-check"; return 0; fi cqa_log "common.sh self-check had $fails failure(s)"; return 1 } if [ "${1:-}" = "--self-check" ]; then cqa__self_check exit $? fi -
mock-model.mjs 2.1 KB · in bundle
-
-
app-server-drive.sh 3.1 KB
#!/usr/bin/env bash # app-server-drive.sh - FIRST-PARTY codex QA: drive a real `codex app-server` # turn against an ISOLATED CODEX_HOME + a LOCAL mock model, and read the # structured notification stream. This is how you prove the omo plugin behaves # in a live Codex session without scripting the TUI and without a real API call. # # Modes: # --self-test Bare isolated home (no plugin). Proves the driver works: a # turn runs and the assistant message comes back from the mock. # Fast; no install. # --plugin Install THIS repo's local omo build into the isolated home, # then drive a turn and PROVE the plugin hooks fire by asserting # hook/completed notifications for the expected events. # Heavier (runs install-local). # # Options (any mode): # --prompt <text> user message (default: "say hello"; --plugin defaults to # "ulw: say hello" so the ultrawork userPromptSubmit hook fires) # --expect <ev,...> hook eventNames that MUST complete (default in --plugin: # "sessionStart,userPromptSubmit") # --keep do not delete the isolated home (for inspection) # # The captured JSON summary IS the evidence; redirect it into .omo/evidence/. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/lib/common.sh" cqa_drive() { local plugin="$1" prompt="$2" expect="$3" cqa_require codex node jq || return 1 cqa_guard_real_home cqa_mk_isolated_home if [ "$plugin" = "1" ]; then cqa_log "installing local omo build into $CODEX_HOME (this builds the plugin)..." if ! cqa_install_local_omo; then cqa_log "install failed; tail:"; tail -20 "$CQA_HOME_ROOT/install.log" >&2 2>/dev/null return 1 fi grep -q 'omo@sisyphuslabs' "$CODEX_HOME/config.toml" || { cqa_fail "omo not enabled in isolated config.toml"; return 1; } fi cqa_start_mock || return 1 local out out="$(EXPECT_HOOK="$expect" PROMPT="$prompt" DEADLINE_MS="${DEADLINE_MS:-90000}" \ node "$SCRIPT_DIR/lib/app-server-client.mjs")" local rc=$? printf '%s\n' "$out" cqa_assert_real_home_unchanged || rc=1 if [ "$rc" -eq 0 ]; then cqa_pass "app-server turn completed; assistant text: $(printf '%s' "$out" | jq -r '.assistantText')" [ -n "$expect" ] && cqa_pass "hooks fired: $(printf '%s' "$out" | jq -r '[.hooks[]|select(.method=="hook/completed")|.eventName]|unique|join(", ")')" else cqa_log "missing hooks: $(printf '%s' "$out" | jq -rc '.missingHooks? // []')" fi return "$rc" } MODE="--self-test"; PROMPT=""; EXPECT=""; KEEP=0 while [ $# -gt 0 ]; do case "$1" in --self-test|--plugin) MODE="$1"; shift ;; --prompt) PROMPT="$2"; shift 2 ;; --expect) EXPECT="$2"; shift 2 ;; --keep) KEEP=1; shift ;; -h|--help) sed -n '2,33p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; *) cqa_log "unknown option: $1"; shift ;; esac done if [ "$KEEP" = "1" ]; then trap - EXIT; fi if [ "$MODE" = "--plugin" ]; then cqa_drive 1 "${PROMPT:-ulw: say hello}" "${EXPECT:-sessionStart,userPromptSubmit}" else cqa_drive 0 "${PROMPT:-say hello}" "$EXPECT" fi exit $? -
hook-unit-probe.sh 3 KB
#!/usr/bin/env bash # hook-unit-probe.sh - deterministic, binary-free proof that a single omo # component's hook logic fires. Pipes a synthetic Codex hook event (the exact # stdin shape Codex sends) into the component's cached dist/cli.js and asserts # its stdout - no codex process, no model, no network. Fast and exact. # # Use this to pin a specific component's behavior; use app-server-drive.sh # --plugin to prove the app-server actually WIRES that hook in a live turn. # # --self-test install local omo (if needed), then assert the # ultrawork component injects <ultrawork-mode> on # an "ulw" UserPromptSubmit. (default) # --component <name> --event <kebab-event> [--prompt <text>] # run an arbitrary component/event by hand against # an already-installed isolated CODEX_HOME ($CODEX_HOME). SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/lib/common.sh" cqa_plugin_root() { ls -d "$CODEX_HOME"/plugins/cache/sisyphuslabs/omo/*/ 2>/dev/null | head -1 } cqa_run_component() { local comp="$1" event="$2" prompt="$3" root cli root="$(cqa_plugin_root)"; [ -n "$root" ] || { cqa_fail "no installed omo under $CODEX_HOME"; return 1; } cli="$root/components/$comp/dist/cli.js" [ -f "$cli" ] || { cqa_fail "component cli missing: $cli"; return 1; } local payload payload="$(jq -nc --arg p "$prompt" --arg cwd "${QA_CWD:-$PWD}" \ '{hook_event_name:"UserPromptSubmit",prompt:$p,cwd:$cwd,session_id:"cqa-unit",model:"mock-model"}')" printf '%s' "$payload" | PLUGIN_ROOT="$root" PLUGIN_DATA="$CODEX_HOME/plugins/data/omo-$comp" node "$cli" hook "$event" } cqa_self_test() { cqa_require codex node jq || return 1 cqa_guard_real_home cqa_mk_isolated_home cqa_log "installing local omo into $CODEX_HOME ..." cqa_install_local_omo || { tail -20 "$CQA_HOME_ROOT/install.log" >&2; return 1; } local out out="$(cqa_run_component ultrawork user-prompt-submit "ulw: do the thing")" cqa_assert_real_home_unchanged || return 1 if printf '%s' "$out" | jq -e '.hookSpecificOutput.additionalContext | test("ultrawork-mode")' >/dev/null 2>&1; then cqa_pass "ultrawork UserPromptSubmit injected <ultrawork-mode> on an ulw prompt" return 0 fi cqa_log "FAIL: ultrawork did not inject ultrawork-mode; got: $out"; return 1 } MODE="self"; COMP=""; EVENT=""; PROMPT="ulw: do the thing" while [ $# -gt 0 ]; do case "$1" in --self-test) MODE="self"; shift ;; --component) MODE="manual"; COMP="$2"; shift 2 ;; --event) EVENT="$2"; shift 2 ;; --prompt) PROMPT="$2"; shift 2 ;; -h|--help) sed -n '2,18p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; *) cqa_log "unknown option: $1"; shift ;; esac done if [ "$MODE" = "manual" ]; then [ -n "$COMP" ] && [ -n "$EVENT" ] || { cqa_log "manual mode needs --component and --event"; exit 2; } cqa_run_component "$COMP" "$EVENT" "$PROMPT" exit $? fi cqa_self_test exit $? -
install-verify.sh 2.5 KB
#!/usr/bin/env bash # install-verify.sh - install THIS repo's local omo build into an ISOLATED # CODEX_HOME and prove it landed correctly while the real ~/.codex is untouched. # # Asserts: plugin cache dir exists, config.toml enables omo@sisyphuslabs, the # component bins + agent TOMLs linked inside the sandbox, and the real # ~/.codex/config.toml shasum is unchanged. # # --self-test run the full isolated install + assertions (default) # --keep keep the isolated home and print its path for inspection SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/lib/common.sh" cqa_install_verify() { cqa_require codex node || return 1 cqa_guard_real_home cqa_mk_isolated_home cqa_log "installing local omo into $CODEX_HOME ..." if ! cqa_install_local_omo; then cqa_log "install failed; tail:"; tail -25 "$CQA_HOME_ROOT/install.log" >&2 2>/dev/null return 1 fi local fails=0 if ls "$CODEX_HOME"/plugins/cache/sisyphuslabs/omo/*/ >/dev/null 2>&1; then cqa_pass "plugin cache present ($(ls "$CODEX_HOME"/plugins/cache/sisyphuslabs/omo/ | head -1))" else cqa_log "FAIL: plugin cache missing"; fails=$((fails+1)); fi if grep -q '\[plugins."omo@sisyphuslabs"\]' "$CODEX_HOME/config.toml" 2>/dev/null \ && grep -A2 '\[plugins."omo@sisyphuslabs"\]' "$CODEX_HOME/config.toml" | grep -q 'enabled = true'; then cqa_pass "config.toml enables omo@sisyphuslabs" else cqa_log "FAIL: omo not enabled in isolated config.toml"; fails=$((fails+1)); fi if ls "$CODEX_HOME"/bin/omo-* >/dev/null 2>&1; then cqa_pass "component bins linked in sandbox ($(ls "$CODEX_HOME"/bin/omo-* | wc -l | tr -d ' ') bins)" else cqa_log "FAIL: no component bins under $CODEX_HOME/bin"; fails=$((fails+1)); fi if [ -d "$CODEX_HOME/agents" ] && ls "$CODEX_HOME"/agents/*.toml >/dev/null 2>&1; then cqa_pass "agent TOMLs linked in sandbox" else cqa_log "FAIL: no agent TOMLs under $CODEX_HOME/agents"; fails=$((fails+1)); fi cqa_assert_real_home_unchanged || fails=$((fails+1)) [ "$KEEP" = "1" ] && cqa_log "kept isolated home: $CODEX_HOME" if [ "$fails" -eq 0 ]; then cqa_pass "install-verify"; return 0; fi cqa_log "install-verify had $fails failure(s)"; return 1 } KEEP=0 while [ $# -gt 0 ]; do case "$1" in --self-test) shift ;; --keep) KEEP=1; shift ;; -h|--help) sed -n '2,11p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; *) cqa_log "unknown option: $1"; shift ;; esac done [ "$KEEP" = "1" ] && trap - EXIT cqa_install_verify exit $? -
lsp-e2e.sh 150.9 KB
#!/usr/bin/env bash # lsp-e2e.sh - isolated live Codex QA for the shared OMO LSP daemon. # # Normal mode installs this worktree's OMO plugin into a disposable CODEX_HOME, # drives a real `codex app-server` against the local codex-qa mock model, calls # the installed lsp MCP through `mcpServer/tool/call`, and records a PASS result # only after path-contract / rename, hook, isolation, and cleanup assertions # succeed. # # Usage: # lsp-e2e.sh --scenario <name> --evidence-dir <absolute-dir> # lsp-e2e.sh --self-test set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd -P)" SCENARIO="" EVIDENCE_DIR="" SELF_TEST=0 SANDBOX_ROOT="" OMO_TEST_ROOT="" EXPECTED_DAEMON_CLI="" MOCK_PID="" APP_SERVER_PID_FILE="" RESULT_STAGE="" CLEANUP_RUNNING=0 NORMAL_CLEANUP_COMPLETE=0 REAL_HOME="${HOME:-}" REAL_OMO_ROOT="${HOME:-}/.omo/lsp-daemon" REAL_CODEX_CONFIG="${HOME:-}/.codex/config.toml" REAL_OMO_BEFORE_HASH="" DAEMON_DIST_DIR="$REPO_ROOT/packages/lsp-daemon/dist" DAEMON_DIST_BACKUP="" DAEMON_DIST_WAS_PRESENT=0 DAEMON_DIST_PREPARED=0 BUILD_LOCK_DIR="" CANCELLATION_SMOKE_RELATIVE="packages/lsp-daemon/scripts/qa/cancellation-smoke.mjs" COMMIT_BARRIER_SMOKE_RELATIVE="packages/lsp-daemon/scripts/qa/commit-barrier-smoke.mjs" log() { printf '[codex-lsp-e2e] %s\n' "$*" >&2; } fail() { log "FAIL: $*"; return 1; } usage() { sed -n '2,12p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' } parse_args() { while [ "$#" -gt 0 ]; do case "$1" in --scenario) [ "$#" -ge 2 ] || { log "--scenario requires a value"; return 2; } [ -z "$SCENARIO" ] || { log "--scenario may be provided only once"; return 2; } SCENARIO="$2" shift 2 ;; --evidence-dir) [ "$#" -ge 2 ] || { log "--evidence-dir requires a directory"; return 2; } [ -z "$EVIDENCE_DIR" ] || { log "--evidence-dir may be provided only once"; return 2; } EVIDENCE_DIR="$2" shift 2 ;; --self-test) [ "$SELF_TEST" -eq 0 ] || { log "--self-test may be provided only once"; return 2; } SELF_TEST=1 shift ;; -h|--help) usage exit 0 ;; *) log "unknown option: $1" return 2 ;; esac done if [ "$SELF_TEST" -eq 1 ]; then if [ -n "$SCENARIO" ] || [ -n "$EVIDENCE_DIR" ]; then log "--self-test cannot be combined with normal-mode options" return 2 fi return 0 fi [ -n "$SCENARIO" ] || { log "--scenario is required"; return 2; } [ -n "$EVIDENCE_DIR" ] || { log "--evidence-dir is required"; return 2; } case "$SCENARIO" in [A-Za-z0-9]* ) ;; * ) log "invalid scenario: $SCENARIO"; return 2 ;; esac if ! printf '%s' "$SCENARIO" | grep -Eq '^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$'; then log "invalid scenario: $SCENARIO" return 2 fi case "$EVIDENCE_DIR" in /*) ;; *) log "--evidence-dir must be absolute"; return 2 ;; esac } require_bins() { local missing=0 bin for bin in "$@"; do if ! command -v "$bin" >/dev/null 2>&1; then log "missing dependency: $bin" missing=1 fi done [ "$missing" -eq 0 ] } verify_tracked_cancellation_probes() { local dependency ignored_evidence_root=".omo""/evidence" if grep -Fq "$ignored_evidence_root" "${BASH_SOURCE[0]}"; then fail "LSP QA driver references ignored evidence state" return 1 fi for dependency in "$CANCELLATION_SMOKE_RELATIVE" "$COMMIT_BARRIER_SMOKE_RELATIVE"; do [ -f "$REPO_ROOT/$dependency" ] || { fail "missing cancellation QA dependency: $dependency"; return 1; } git -C "$REPO_ROOT" ls-files --error-unmatch -- "$dependency" >/dev/null 2>&1 || { fail "cancellation QA dependency is not tracked: $dependency" return 1 } done } hash_path() { node --input-type=module - "$1" <<'NODE' import { createHash } from "node:crypto"; import { lstatSync, readFileSync, readlinkSync, readdirSync } from "node:fs"; import { basename, join } from "node:path"; const target = process.argv[2]; const hash = createHash("sha256"); function visit(path, relative) { const stat = lstatSync(path); const kind = stat.isDirectory() ? "dir" : stat.isFile() ? "file" : stat.isSymbolicLink() ? "link" : "special"; hash.update(`${kind}\0${relative}\0${stat.mode & 0o7777}\0`); if (kind === "file") hash.update(readFileSync(path)); if (kind === "link") hash.update(readlinkSync(path)); if (kind === "dir") { for (const name of readdirSync(path).sort()) visit(join(path, name), relative ? `${relative}/${name}` : name); } } try { visit(target, basename(target)); process.stdout.write(hash.digest("hex")); } catch (error) { if (error && error.code === "ENOENT") process.stdout.write("ABSENT"); else throw error; } NODE } run_bounded() { local seconds="$1" output="$2" shift 2 node --input-type=module - "$seconds" "$output" "$@" <<'NODE' import { closeSync, openSync } from "node:fs"; import { spawn } from "node:child_process"; const [secondsRaw, output, command, ...args] = process.argv.slice(2); const seconds = Number(secondsRaw); if (!Number.isFinite(seconds) || seconds <= 0 || !command) process.exit(125); const fd = openSync(output, "w"); const child = spawn(command, args, { stdio: ["ignore", fd, fd], detached: process.platform !== "win32", env: process.env, }); let timedOut = false; let forceTimer; const timer = setTimeout(() => { timedOut = true; try { if (process.platform !== "win32") process.kill(-child.pid, "SIGTERM"); else child.kill("SIGTERM"); } catch {} forceTimer = setTimeout(() => { try { if (process.platform !== "win32") process.kill(-child.pid, "SIGKILL"); else child.kill("SIGKILL"); } catch {} }, 3000); }, seconds * 1000); child.on("error", () => { clearTimeout(timer); if (forceTimer) clearTimeout(forceTimer); closeSync(fd); process.exit(126); }); child.on("exit", (code, signal) => { clearTimeout(timer); if (forceTimer) clearTimeout(forceTimer); closeSync(fd); if (timedOut) process.exit(124); if (typeof code === "number") process.exit(code); process.exit(signal ? 128 : 1); }); NODE } with_shared_build_lock() { local command_name="$1" attempts=0 rc shift BUILD_LOCK_DIR="$REPO_ROOT/.omo/locks/lsp-daemon-build.lock" mkdir -p "$(dirname "$BUILD_LOCK_DIR")" while ! mkdir "$BUILD_LOCK_DIR" 2>/dev/null; do [ "$attempts" -lt 600 ] || { fail "timed out waiting for shared LSP daemon build lock"; return 1; } sleep 0.2 attempts=$((attempts + 1)) done printf 'pid=%s\ncommand=%s\n' "$$" "$command_name" >"$BUILD_LOCK_DIR/owner.txt" "$@" rc=$? rm -rf "$BUILD_LOCK_DIR" BUILD_LOCK_DIR="" return "$rc" } safe_rm_tree() { local path="$1" local attempt=0 [ -n "$path" ] || return 0 case "$path" in /var/folders/*/T/cqa-lsp-e2e.*|/tmp/cqa-lsp-e2e.*|/private/tmp/cqa-lsp-e2e.*) while [ -e "$path" ] && [ "$attempt" -lt 100 ]; do rm -rf "$path" 2>/dev/null || true [ ! -e "$path" ] && return 0 sleep 0.1 attempt=$((attempt + 1)) done [ ! -e "$path" ] || fail "isolated sandbox remained after bounded cleanup: $path" ;; *) fail "refusing to remove unexpected sandbox path: $path" ;; esac } owned_sandbox_pids() { [ -n "$SANDBOX_ROOT" ] || return 0 /bin/ps ax -o pid=,command= 2>/dev/null | while read -r pid command; do case "$command" in *"$SANDBOX_ROOT"*) [ "$pid" = "$$" ] || printf '%s\n' "$pid" ;; esac done } stop_owned_sandbox_processes() { local pids pid command pids="$(owned_sandbox_pids)" [ -n "$pids" ] || return 0 for pid in $pids; do kill -0 "$pid" 2>/dev/null || continue command="$(process_command "$pid")" case "$command" in *"$SANDBOX_ROOT"*) ;; *) fail "sandbox process $pid changed identity before cleanup"; return 1 ;; esac kill "$pid" 2>/dev/null || true if ! wait_for_exit "$pid"; then command="$(process_command "$pid")" case "$command" in *"$SANDBOX_ROOT"*) kill -9 "$pid" 2>/dev/null || true ;; *) fail "sandbox process $pid changed identity during cleanup"; return 1 ;; esac wait_for_exit "$pid" || { fail "sandbox process $pid survived cleanup"; return 1; } fi if [ -n "$EVIDENCE_DIR" ] && [ -d "$EVIDENCE_DIR" ]; then printf 'sandbox_process_pid=%s alive_after=no\n' "$pid" >>"$EVIDENCE_DIR/owned-process-cleanup.txt" fi done } process_command() { /bin/ps -p "$1" -o command= 2>/dev/null || true } wait_for_exit() { local pid="$1" attempts=0 while [ "$attempts" -lt 50 ]; do kill -0 "$pid" 2>/dev/null || return 0 sleep 0.1 attempts=$((attempts + 1)) done return 1 } stop_known_pid_file() { local pid_file="$1" expected_fragment="$2" label="$3" [ -n "$pid_file" ] && [ -f "$pid_file" ] || return 0 local pid command pid="$(tr -d '[:space:]' <"$pid_file" 2>/dev/null || true)" case "$pid" in ''|*[!0-9]*) return 0 ;; esac kill -0 "$pid" 2>/dev/null || return 0 command="$(process_command "$pid")" case "$command" in *"$expected_fragment"*) ;; *) fail "refusing to stop unverified $label pid $pid"; return 1 ;; esac kill "$pid" 2>/dev/null || true if ! wait_for_exit "$pid"; then command="$(process_command "$pid")" case "$command" in *"$expected_fragment"*) kill -9 "$pid" 2>/dev/null || true ;; *) fail "$label pid $pid changed identity during cleanup"; return 1 ;; esac wait_for_exit "$pid" || { fail "$label pid $pid survived cleanup"; return 1; } fi } find_daemon_pid_file() { [ -n "$OMO_TEST_ROOT" ] && [ -d "$OMO_TEST_ROOT" ] || return 0 find "$OMO_TEST_ROOT" -type f -name daemon.pid -print 2>/dev/null | sort | head -1 } stop_known_daemon() { local pid_file pid command pid_file="$(find_daemon_pid_file)" [ -n "$pid_file" ] || return 0 pid="$(tr -d '[:space:]' <"$pid_file" 2>/dev/null || true)" case "$pid" in ''|*[!0-9]*) fail "daemon pid file is malformed: $pid_file"; return 1 ;; esac kill -0 "$pid" 2>/dev/null || return 0 command="$(process_command "$pid")" case "$command" in *"$EXPECTED_DAEMON_CLI"*" daemon"*) ;; *) fail "refusing to stop unverified daemon pid $pid"; return 1 ;; esac kill "$pid" 2>/dev/null || true if ! wait_for_exit "$pid"; then command="$(process_command "$pid")" case "$command" in *"$EXPECTED_DAEMON_CLI"*" daemon"*) kill -9 "$pid" 2>/dev/null || true ;; *) fail "daemon pid $pid changed identity during cleanup"; return 1 ;; esac wait_for_exit "$pid" || { fail "daemon pid $pid survived cleanup"; return 1; } fi } stop_owned_real_daemon_leak() { [ "$REAL_OMO_BEFORE_HASH" = "ABSENT" ] || return 0 [ "$OMO_TEST_ROOT" != "$REAL_OMO_ROOT" ] || return 0 [ -d "$REAL_OMO_ROOT" ] || return 0 local pid_file pid command state_dir pid_file="$(find "$REAL_OMO_ROOT" -type f -name daemon.pid -print 2>/dev/null | sort | head -1)" if [ -n "$pid_file" ]; then pid="$(tr -d '[:space:]' <"$pid_file" 2>/dev/null || true)" case "$pid" in ''|*[!0-9]*) fail "real-root leak pid file is malformed"; return 1 ;; esac command="$(process_command "$pid")" case "$command" in *"$EXPECTED_DAEMON_CLI"*" daemon"*) ;; *) fail "real OMO root changed by an unverified process; preserving it"; return 1 ;; esac kill "$pid" 2>/dev/null || true wait_for_exit "$pid" || { fail "own leaked daemon did not stop"; return 1; } fi if find "$REAL_OMO_ROOT" -type f \( -name daemon.pid -o -name daemon.endpoint \) -print 2>/dev/null | grep -q .; then fail "real OMO root still contains live markers after own-daemon cleanup" return 1 fi find "$REAL_OMO_ROOT" -type f -name daemon.log -delete 2>/dev/null || true while IFS= read -r state_dir; do rmdir "$state_dir" 2>/dev/null || true; done < <(find "$REAL_OMO_ROOT" -depth -type d -print 2>/dev/null) [ ! -e "$REAL_OMO_ROOT" ] || { fail "real OMO root could not be restored to ABSENT"; return 1; } } stop_mock() { [ -n "$MOCK_PID" ] || return 0 if kill -0 "$MOCK_PID" 2>/dev/null; then kill "$MOCK_PID" 2>/dev/null || true wait_for_exit "$MOCK_PID" || kill -9 "$MOCK_PID" 2>/dev/null || true fi MOCK_PID="" } cleanup_all() { local cleanup_rc=0 [ "$CLEANUP_RUNNING" -eq 0 ] || return 0 CLEANUP_RUNNING=1 stop_mock || cleanup_rc=1 if [ -n "$APP_SERVER_PID_FILE" ]; then stop_known_pid_file "$APP_SERVER_PID_FILE" "app-server" "Codex app-server" || cleanup_rc=1 fi stop_known_daemon || cleanup_rc=1 stop_owned_sandbox_processes || cleanup_rc=1 stop_owned_real_daemon_leak || cleanup_rc=1 restore_daemon_dist || cleanup_rc=1 if [ -n "$BUILD_LOCK_DIR" ]; then rm -rf "$BUILD_LOCK_DIR" 2>/dev/null || cleanup_rc=1 BUILD_LOCK_DIR="" fi [ -n "$RESULT_STAGE" ] && rm -f "$RESULT_STAGE" 2>/dev/null || true if [ -n "$EVIDENCE_DIR" ] && [ -d "$EVIDENCE_DIR" ]; then find "$EVIDENCE_DIR" -maxdepth 1 -type f -name '.result.json.*' -delete 2>/dev/null || true fi if [ -n "$SANDBOX_ROOT" ]; then safe_rm_tree "$SANDBOX_ROOT" || cleanup_rc=1 SANDBOX_ROOT="" fi CLEANUP_RUNNING=0 return "$cleanup_rc" } on_exit() { local rc=$? trap - EXIT INT TERM HUP if [ "$NORMAL_CLEANUP_COMPLETE" -eq 0 ]; then cleanup_all || rc=1 fi if [ "$rc" -ne 0 ] && [ -n "$EVIDENCE_DIR" ] && [ -d "$EVIDENCE_DIR" ]; then rm -f "$EVIDENCE_DIR/result.json" 2>/dev/null || true fi exit "$rc" } trap on_exit EXIT trap 'exit 130' INT trap 'exit 143' TERM trap 'exit 129' HUP prepare_evidence() { [ ! -L "$EVIDENCE_DIR" ] || { fail "evidence directory must not be a symlink"; return 1; } mkdir -p "$EVIDENCE_DIR" || return 1 EVIDENCE_DIR="$(cd "$EVIDENCE_DIR" && pwd -P)" rm -f "$EVIDENCE_DIR/result.json" find "$EVIDENCE_DIR" -maxdepth 1 -type f -name '.result.json.*' -delete 2>/dev/null || true printf 'bash %s --scenario %s --evidence-dir %s\n' \ "${BASH_SOURCE[0]}" "$SCENARIO" "$EVIDENCE_DIR" >"$EVIDENCE_DIR/invocation.txt" } daemon_dist_ready() { [ -f "$DAEMON_DIST_DIR/package.json" ] \ && [ -f "$DAEMON_DIST_DIR/index.js" ] \ && [ -f "$DAEMON_DIST_DIR/cli.js" ] } snapshot_daemon_dist() { local backup_root="$1" mkdir -p "$backup_root" || return 1 DAEMON_DIST_BACKUP="$backup_root/lsp-daemon-dist.backup" rm -rf "$DAEMON_DIST_BACKUP" if [ -e "$DAEMON_DIST_DIR" ]; then cp -a "$DAEMON_DIST_DIR" "$DAEMON_DIST_BACKUP" || return 1 DAEMON_DIST_WAS_PRESENT=1 else DAEMON_DIST_WAS_PRESENT=0 fi DAEMON_DIST_PREPARED=1 } restore_daemon_dist() { [ "$DAEMON_DIST_PREPARED" -eq 1 ] || return 0 rm -rf "$DAEMON_DIST_DIR" || return 1 if [ "$DAEMON_DIST_WAS_PRESENT" -eq 1 ]; then mkdir -p "$(dirname "$DAEMON_DIST_DIR")" || return 1 cp -a "$DAEMON_DIST_BACKUP" "$DAEMON_DIST_DIR" || return 1 fi if [ -n "$EVIDENCE_DIR" ] && [ -d "$EVIDENCE_DIR" ]; then { printf 'daemon_dist_was_present=%s\n' "$( [ "$DAEMON_DIST_WAS_PRESENT" -eq 1 ] && echo true || echo false )" printf 'daemon_dist_restored=true\n' printf 'daemon_dist_path=%s\n' "$DAEMON_DIST_DIR" } >>"$EVIDENCE_DIR/daemon-dist-cleanup.txt" fi DAEMON_DIST_BACKUP="" DAEMON_DIST_WAS_PRESENT=0 DAEMON_DIST_PREPARED=0 } prepare_daemon_dist() { if daemon_dist_ready; then printf 'daemon_dist_ready_before=true\nbuild_skipped=true\n' >"$EVIDENCE_DIR/lsp-daemon-prebuild-receipt.txt" return 0 fi if [ "$DAEMON_DIST_PREPARED" -eq 0 ]; then snapshot_daemon_dist "$SANDBOX_ROOT/daemon-dist-backup" || return 1 fi { printf 'daemon_dist_ready_before=false\n' printf 'daemon_dist_was_present=%s\n' "$( [ "$DAEMON_DIST_WAS_PRESENT" -eq 1 ] && echo true || echo false )" printf 'command=bun run build:lsp-daemon\n' } >"$EVIDENCE_DIR/lsp-daemon-prebuild-receipt.txt" run_bounded 300 "$EVIDENCE_DIR/lsp-daemon-prebuild.log" bun run build:lsp-daemon || return 1 daemon_dist_ready || { fail "lsp-daemon prebuild did not create required dist files"; return 1; } printf 'daemon_dist_ready_after=true\n' >>"$EVIDENCE_DIR/lsp-daemon-prebuild-receipt.txt" } write_path_contract_probe() { local probe_dir="$1" output="$2" script="$SANDBOX_ROOT/path-contract-probe.mjs" mkdir -p "$probe_dir" cat >"$script" <<'NODE' import { existsSync, mkdirSync, readFileSync, rmSync } from "node:fs"; import { dirname, join, resolve } from "node:path"; import { pathToFileURL } from "node:url"; const repoRoot = process.env.REPO_ROOT; const base = process.env.PROBE_BASE; const output = process.env.PROBE_OUTPUT; if (!repoRoot || !base || !output) throw new Error("missing probe environment"); const modulePath = join(repoRoot, "packages/lsp-daemon/dist/index.js"); const daemon = await import(pathToFileURL(modulePath).href + `?qa=${Date.now()}`); const cliPath = join(repoRoot, "packages/lsp-daemon/dist/cli.js"); const packagedVersion = JSON.parse(readFileSync(join(repoRoot, "packages/lsp-daemon/dist/package.json"), "utf8")).version; const envNameValues = [daemon.OMO_LSP_DAEMON_CLI, daemon.OMO_LSP_DAEMON_DIR, daemon.OMO_LSP_DAEMON_VERSION].sort(); function capture(run) { try { run(); return { threw: false }; } catch (error) { return { threw: true, name: error?.name, code: error?.code, reason: error?.reason, message: error?.message }; } } rmSync(base, { recursive: true, force: true }); const defaultPaths = daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: base }); const pairedVersion = "qa.1+pair"; const pairedPaths = daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: base, [daemon.OMO_LSP_DAEMON_CLI]: cliPath, [daemon.OMO_LSP_DAEMON_VERSION]: pairedVersion, }); const singletonRoot = join(dirname(base), "singleton-state"); rmSync(singletonRoot, { recursive: true, force: true }); const singletonCli = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: singletonRoot, [daemon.OMO_LSP_DAEMON_CLI]: cliPath, })); const singletonVersion = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: singletonRoot, [daemon.OMO_LSP_DAEMON_VERSION]: packagedVersion, })); const relativeBase = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: "relative/state" })); const relativeCli = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: join(dirname(base), "relative-cli-state"), [daemon.OMO_LSP_DAEMON_CLI]: "relative/cli.js", [daemon.OMO_LSP_DAEMON_VERSION]: packagedVersion, })); const nonFileCli = join(dirname(base), "not-a-file"); mkdirSync(nonFileCli, { recursive: true }); const missingCli = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: join(dirname(base), "missing-cli-state"), [daemon.OMO_LSP_DAEMON_CLI]: join(dirname(base), "missing-cli.js"), [daemon.OMO_LSP_DAEMON_VERSION]: packagedVersion, })); const directoryCli = capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: join(dirname(base), "directory-cli-state"), [daemon.OMO_LSP_DAEMON_CLI]: nonFileCli, [daemon.OMO_LSP_DAEMON_VERSION]: packagedVersion, })); const badVersions = ["../escape", "a/b", "a\\b", ".hidden", "bad value", "", "a".repeat(129)]; const versionFailures = badVersions.map((version, index) => { const stateRoot = join(dirname(base), `bad-version-${index}`); rmSync(stateRoot, { recursive: true, force: true }); return { version, error: capture(() => daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: stateRoot, [daemon.OMO_LSP_DAEMON_CLI]: cliPath, [daemon.OMO_LSP_DAEMON_VERSION]: version, })), stateCreated: existsSync(stateRoot), }; }); const oldPrefix = "CODEX" + "_LSP_"; const neutralPaths = daemon.daemonPaths({ CODEX_HOME: join(dirname(base), "ignored-codex-home"), PLUGIN_DATA: join(dirname(base), "ignored-plugin-data"), [`${oldPrefix}DAEMON_DIR`]: join(dirname(base), "ignored-legacy-dir"), [`${oldPrefix}DAEMON_CLI`]: join(dirname(base), "ignored-legacy-cli.js"), [`${oldPrefix}DAEMON_VERSION`]: "999.999.999", }); const neutralBase = resolve(process.env.HOME, ".omo", "lsp-daemon"); const assertions = { exactThreeOmoEnvironmentNames: JSON.stringify(envNameValues) === JSON.stringify([ "OMO_LSP_DAEMON_CLI", "OMO_LSP_DAEMON_DIR", "OMO_LSP_DAEMON_VERSION", ]), defaultBaseResolved: dirname(defaultPaths.dir) === resolve(base), defaultVersionStamped: defaultPaths.version === packagedVersion, defaultCliPackaged: defaultPaths.cliPath === cliPath, pairedOverridePreserved: pairedPaths.cliPath === cliPath && pairedPaths.version === pairedVersion, singletonCliRejectedBeforeState: singletonCli.code === "invalid_runtime_override" && !existsSync(singletonRoot), singletonVersionRejectedBeforeState: singletonVersion.code === "invalid_runtime_override" && !existsSync(singletonRoot), relativeBaseRejected: relativeBase.code === "invalid_daemon_directory", relativeCliRejected: relativeCli.reason === "cli_must_be_absolute", missingCliRejected: missingCli.reason === "cli_not_found", nonFileCliRejected: directoryCli.reason === "cli_not_file", malformedVersionsRejectedBeforeState: versionFailures.every((entry) => entry.error.code === "invalid_daemon_version" && entry.stateCreated === false), oldNamesAndHarnessHomesIgnored: dirname(neutralPaths.dir) === neutralBase && neutralPaths.version === packagedVersion, }; if (!Object.values(assertions).every(Boolean)) { console.error(JSON.stringify({ assertions, singletonCli, singletonVersion, versionFailures, neutralPaths }, null, 2)); process.exit(1); } await Bun.write(output, JSON.stringify({ assertions, environmentNames: envNameValues, default: defaultPaths, paired: pairedPaths, neutral: neutralPaths, failures: { singletonCli, singletonVersion, relativeBase, relativeCli, missingCli, directoryCli, versionFailures }, }, null, 2) + "\n"); NODE REPO_ROOT="$REPO_ROOT" PROBE_BASE="$probe_dir/state/../daemon" PROBE_OUTPUT="$output" \ run_bounded 30 "$EVIDENCE_DIR/path-contract-probe.log" bun "$script" } write_workspace_edit_fixture() { local project_dir="$1" local scenario_path="$EVIDENCE_DIR/rename-scenario.json" local events_path="$EVIDENCE_DIR/rename-server-events.jsonl" local metadata_path="$EVIDENCE_DIR/rename-fixture.json" local user_config_path="$HOME/.codex/lsp-client.json" mkdir -p "$project_dir" "$(dirname "$user_config_path")" node --input-type=module - "$REPO_ROOT" "$project_dir" "$scenario_path" "$events_path" "$metadata_path" "$user_config_path" <<'NODE' import { mkdirSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { pathToFileURL } from "node:url"; const [repoRoot, projectDir, scenarioPath, eventsPath, metadataPath, userConfigPath] = process.argv.slice(2); const sourcePath = join(projectDir, "source.ts"); const fixturePath = join(repoRoot, "packages/lsp-core/src/lsp/fixtures/workspace-edit-server.mjs"); mkdirSync(dirname(userConfigPath), { recursive: true }); writeFileSync(sourcePath, "const before = 1;\n", "utf8"); writeFileSync(eventsPath, "", "utf8"); const sourceUri = pathToFileURL(sourcePath).href; const scenario = { renameSteps: [ { applyEdit: { documentChanges: [ { textDocument: { uri: sourceUri, version: 1 }, edits: [ { range: { start: { line: 0, character: 6 }, end: { line: 0, character: 12 }, }, newText: "after", }, ], }, ], }, renameResult: "same", }, ], diagnostics: [ { range: { start: { line: 0, character: 0 }, end: { line: 0, character: 1 }, }, message: "todo3-fresh", }, ], }; const userConfig = { lsp: { typescript: { command: [process.execPath, fixturePath, scenarioPath, eventsPath], extensions: [".ts"], priority: 100, }, }, }; writeFileSync(scenarioPath, JSON.stringify(scenario, null, 2) + "\n"); writeFileSync(userConfigPath, JSON.stringify(userConfig, null, 2) + "\n"); writeFileSync( metadataPath, JSON.stringify( { sourcePath, sourceUri, scenarioPath, eventsPath, userConfigPath, }, null, 2, ) + "\n", ); NODE } run_workspace_edit_contract_probe() { run_bounded 60 "$EVIDENCE_DIR/workspace-edit-contract-probe.log" \ bun "$REPO_ROOT/packages/lsp-core/src/lsp/fixtures/workspace-edit-contract-probe.ts" \ "$EVIDENCE_DIR/workspace-edit-contract.json" } write_diagnostics_freshness_fixture() { local project_dir="$1" local scenario_path="$EVIDENCE_DIR/diagnostics-freshness-scenario.json" local events_path="$EVIDENCE_DIR/diagnostics-freshness-server-events.jsonl" local metadata_path="$EVIDENCE_DIR/diagnostics-freshness-fixture.json" local user_config_path="$HOME/.codex/lsp-client.json" mkdir -p "$project_dir" "$(dirname "$user_config_path")" node --input-type=module - "$REPO_ROOT" "$project_dir" "$scenario_path" "$events_path" "$metadata_path" "$user_config_path" <<'NODE' import { mkdirSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; const [repoRoot, projectDir, scenarioPath, eventsPath, metadataPath, userConfigPath] = process.argv.slice(2); const sourcePath = join(projectDir, "source.ts"); const fixturePath = join(repoRoot, "packages/lsp-core/src/lsp/fixtures/workspace-edit-server.mjs"); mkdirSync(dirname(userConfigPath), { recursive: true }); writeFileSync(sourcePath, "const before = 1;\n", "utf8"); writeFileSync(eventsPath, "", "utf8"); const scenario = { publishDiagnostics: [ { trigger: "didOpen", version: 1, diagnostics: [ { range: { start: { line: 0, character: 0 }, end: { line: 0, character: 1 }, }, message: "exact-current", }, ], }, ], diagnosticResponses: [ { report: { items: [ { range: { start: { line: 0, character: 0 }, end: { line: 0, character: 1 }, }, message: "exact-current", }, ], }, }, ], }; const userConfig = { lsp: { typescript: { command: [process.execPath, fixturePath, scenarioPath, eventsPath], extensions: [".ts"], priority: 100, }, }, }; writeFileSync(scenarioPath, JSON.stringify(scenario, null, 2) + "\n"); writeFileSync(userConfigPath, JSON.stringify(userConfig, null, 2) + "\n"); writeFileSync( metadataPath, JSON.stringify( { sourcePath, scenarioPath, eventsPath, userConfigPath, }, null, 2, ) + "\n", ); NODE } run_diagnostics_freshness_contract_probe() { run_bounded 60 "$EVIDENCE_DIR/diagnostics-freshness-contract-probe.log" \ bun "$REPO_ROOT/packages/lsp-core/src/lsp/fixtures/diagnostics-freshness-contract-probe.ts" \ "$EVIDENCE_DIR/diagnostics-freshness-contract.json" } run_post_edit_contract_probe() { local script="$EVIDENCE_DIR/post-edit-contract-probe.mjs" cat >"$script" <<'NODE' import { mkdirSync, realpathSync, writeFileSync } from "node:fs"; import { delimiter, join, resolve } from "node:path"; import { pathToFileURL } from "node:url"; const [repoRoot, output, rawProjectDir, rawHomeDir] = process.argv.slice(2); if (!repoRoot || !output || !rawProjectDir || !rawHomeDir) throw new Error("missing post-edit probe arguments"); mkdirSync(rawProjectDir, { recursive: true }); mkdirSync(rawHomeDir, { recursive: true }); const projectDir = realpathSync(rawProjectDir); const homeDir = realpathSync(rawHomeDir); const core = await import(pathToFileURL(join(repoRoot, "packages/lsp-core/src/index.ts")).href); const daemonClient = await import(pathToFileURL(join(repoRoot, "packages/lsp-daemon/src/daemon-client.ts")).href); const openCodeMcp = await import(pathToFileURL(join(repoRoot, "packages/omo-opencode/src/mcp/lsp.ts")).href); const explicitTranslator = core.createStandaloneMcpRequestContext({ cwd: projectDir, homeDir, env: { LSP_TOOLS_MCP_PROJECT_CONFIG: [ join(projectDir, ".opencode", "lsp.json"), "", join(projectDir, ".omo", "lsp.json"), join(projectDir, ".omo", "lsp-client.json"), ].join(delimiter), LSP_TOOLS_MCP_USER_CONFIG: join(homeDir, ".config", "opencode", "lsp.json"), LSP_TOOLS_MCP_INSTALL_DECISIONS: join(homeDir, ".config", "opencode", "lsp-install-decisions.json"), }, }); const defaultTranslator = core.createStandaloneMcpRequestContext({ cwd: projectDir, homeDir, env: {} }); const openCodeMcpConfig = openCodeMcp.createLspMcpConfig({ cwd: projectDir, moduleUrl: pathToFileURL(join(repoRoot, "packages/omo-opencode/src/mcp/lsp.ts")).href, exists: () => false, resolveExecutable: (commandName) => ({ command: commandName === "node" ? process.execPath : commandName, available: true }), }); const openCodeConfigRoot = resolve(process.env.XDG_CONFIG_HOME ?? join(process.env.HOME ?? homeDir, ".config"), "opencode"); const previousCwd = process.cwd(); process.chdir(projectDir); const directContext = daemonClient.currentRequestContext({ HOME: homeDir, LSP_TOOLS_MCP_PROJECT_CONFIG: join(projectDir, ".opencode", "lsp.json"), LSP_TOOLS_MCP_USER_CONFIG: join(homeDir, ".config", "opencode", "lsp.json"), LSP_TOOLS_MCP_INSTALL_DECISIONS: join(homeDir, ".config", "opencode", "lsp-install-decisions.json"), }); process.chdir(previousCwd); let active = 0; let maxActive = 0; const calls = []; const responses = new Map([ ["a.ts", "diagnostic for a.ts"], ["b.ts", "No diagnostics found"], ["c.ts", "diagnostic for c.ts"], ["d.foo", "No LSP server configured for extension: .foo\n\nAvailable servers: typescript"], ["e.ts", "diagnostic for e.ts"], ["f.ts", "diagnostic for f.ts"], ]); const first = await core.collectPostEditDiagnostics({ filePaths: ["a.ts", "b.ts", "a.ts", "c.ts", "d.foo", "e.ts", "f.ts"], runDiagnostics: async (filePath) => { calls.push(filePath); active += 1; maxActive = Math.max(maxActive, active); await new Promise((resolve) => setTimeout(resolve, 10)); active -= 1; if (filePath === "c.ts") throw new Error("diagnostic failure for c.ts"); return responses.get(filePath) ?? "No diagnostics found"; }, }); const cache = core.createPostEditNotConfiguredCache(); const cacheCalls = []; const cachedFirst = await core.collectPostEditDiagnostics({ filePaths: ["skip.foo"], cache, runDiagnostics: async (filePath) => { cacheCalls.push(filePath); return "No LSP server configured for extension: .foo"; }, }); const cachedSecond = await core.collectPostEditDiagnostics({ filePaths: ["retry.foo"], cache, runDiagnostics: async (filePath) => { cacheCalls.push(filePath); return "diagnostic after reset"; }, }); core.resetPostEditNotConfiguredCache(cache); const cachedAfterReset = await core.collectPostEditDiagnostics({ filePaths: ["retry.foo"], cache, runDiagnostics: async (filePath) => { cacheCalls.push(filePath); return "diagnostic after reset"; }, }); let lookupCount = 0; const rejectionResults = {}; function expectReject(name, value) { try { core.parseLspRequestContext(value); rejectionResults[name] = { rejected: false, lookupCount }; } catch (error) { rejectionResults[name] = { rejected: error instanceof core.LspRequestContextParseError, code: error instanceof core.LspRequestContextParseError ? error.code : "unknown", lookupCount, }; } } expectReject("malformed", null); expectReject("unknown", { cwd: projectDir, projectConfigPaths: [join(projectDir, ".codex", "lsp-client.json")], userConfigPath: join(homeDir, ".codex", "lsp-client.json"), installDecisionsPath: join(homeDir, ".codex", "lsp-install-decisions.json"), capabilities: { installDecisionTool: true }, env: {}, }); expectReject("outOfCwd", { cwd: projectDir, projectConfigPaths: [join(homeDir, "outside-lsp.json")], userConfigPath: join(homeDir, ".codex", "lsp-client.json"), installDecisionsPath: join(homeDir, ".codex", "lsp-install-decisions.json"), capabilities: { installDecisionTool: true }, }); lookupCount += 0; const assertions = { openCodeMcpEnvInputs: JSON.stringify(Object.keys(openCodeMcpConfig.environment ?? {}).sort()) === JSON.stringify([ "LSP_TOOLS_MCP_INSTALL_DECISIONS", "LSP_TOOLS_MCP_PROJECT_CONFIG", "LSP_TOOLS_MCP_USER_CONFIG", ]) && JSON.stringify((openCodeMcpConfig.environment?.LSP_TOOLS_MCP_PROJECT_CONFIG ?? "").split(delimiter)) === JSON.stringify([ join(projectDir, ".opencode", "lsp.json"), join(projectDir, ".omo", "lsp.json"), join(projectDir, ".omo", "lsp-client.json"), ]) && openCodeMcpConfig.environment?.LSP_TOOLS_MCP_USER_CONFIG === join(openCodeConfigRoot, "lsp.json") && openCodeMcpConfig.environment?.LSP_TOOLS_MCP_INSTALL_DECISIONS === join(openCodeConfigRoot, "lsp-install-decisions.json"), explicitTranslatorOutputs: JSON.stringify(explicitTranslator.projectConfigPaths) === JSON.stringify([ join(projectDir, ".opencode", "lsp.json"), join(projectDir, ".omo", "lsp.json"), join(projectDir, ".omo", "lsp-client.json"), ]) && explicitTranslator.userConfigPath === join(homeDir, ".config", "opencode", "lsp.json") && explicitTranslator.installDecisionsPath === join(homeDir, ".config", "opencode", "lsp-install-decisions.json") && explicitTranslator.capabilities.installDecisionTool === true, translatorDefaults: JSON.stringify(defaultTranslator.projectConfigPaths) === JSON.stringify([join(projectDir, ".codex", "lsp-client.json")]) && defaultTranslator.userConfigPath === join(homeDir, ".codex", "lsp-client.json") && defaultTranslator.installDecisionsPath === join(homeDir, ".codex", "lsp-install-decisions.json"), directAdapterNonUse: !("env" in directContext) && JSON.stringify(directContext.projectConfigPaths) === JSON.stringify([join(projectDir, ".codex", "lsp-client.json")]) && directContext.userConfigPath === join(homeDir, ".codex", "lsp-client.json") && directContext.installDecisionsPath === join(homeDir, ".codex", "lsp-install-decisions.json"), maxConcurrencyFour: maxActive === 4, orderedBlocks: JSON.stringify(first.blocks) === JSON.stringify([ { filePath: "a.ts", diagnostics: "diagnostic for a.ts" }, { filePath: "c.ts", diagnostics: "diagnostic failure for c.ts" }, { filePath: "d.foo", diagnostics: "No LSP server configured for extension: .foo\n\nAvailable servers: typescript" }, { filePath: "e.ts", diagnostics: "diagnostic for e.ts" }, { filePath: "f.ts", diagnostics: "diagnostic for f.ts" }, ]), duplicatesRunOnce: JSON.stringify(calls) === JSON.stringify(["a.ts", "b.ts", "c.ts", "d.foo", "e.ts", "f.ts"]), cacheResetRetry: JSON.stringify(cachedFirst.blocks) === JSON.stringify([{ filePath: "skip.foo", diagnostics: "No LSP server configured for extension: .foo" }]) && JSON.stringify(cachedSecond.blocks) === JSON.stringify([{ filePath: "retry.foo", diagnostics: "diagnostic after reset" }]) && JSON.stringify(cachedAfterReset.blocks) === JSON.stringify([{ filePath: "retry.foo", diagnostics: "diagnostic after reset" }]) && JSON.stringify(cacheCalls) === JSON.stringify(["skip.foo", "retry.foo", "retry.foo"]), rejectionBeforeLookup: Object.values(rejectionResults).every((entry) => entry.rejected === true && entry.lookupCount === 0), }; const result = { result: Object.values(assertions).every(Boolean) ? "PASS" : "FAIL", assertions, openCodeMcpEnvironment: openCodeMcpConfig.environment, translator: { explicit: explicitTranslator, defaults: defaultTranslator }, directContext, postEdit: { calls, maxActive, first, cachedFirst, cachedSecond, cachedAfterReset, cacheCalls }, rejectionResults, }; writeFileSync(output, `${JSON.stringify(result, null, 2)}\n`); if (result.result !== "PASS") process.exit(1); NODE run_bounded 60 "$EVIDENCE_DIR/post-edit-contract-probe.log" \ bun "$script" "$REPO_ROOT" "$EVIDENCE_DIR/post-edit-contract.json" "$SANDBOX_ROOT/project" "$SANDBOX_ROOT/home" } run_cancellation_contract_probe() { local cancellation_smoke="$REPO_ROOT/$CANCELLATION_SMOKE_RELATIVE" local commit_smoke="$REPO_ROOT/$COMMIT_BARRIER_SMOKE_RELATIVE" verify_tracked_cancellation_probes || return 1 run_bounded 90 "$EVIDENCE_DIR/cancellation-smoke-output.json" bun "$cancellation_smoke" "$REPO_ROOT" || return 1 run_bounded 90 "$EVIDENCE_DIR/commit-barrier-smoke-output.json" bun "$commit_smoke" "$REPO_ROOT" || return 1 node --input-type=module - \ "$EVIDENCE_DIR/cancellation-smoke-output.json" \ "$EVIDENCE_DIR/commit-barrier-smoke-output.json" \ "$EVIDENCE_DIR/cancellation-contract.json" "$SCENARIO" "codex" <<'NODE' import { readFileSync, writeFileSync } from "node:fs"; const [cancelPath, commitPath, outputPath, scenario, harness] = process.argv.slice(2); const cancel = JSON.parse(readFileSync(cancelPath, "utf8")); const commit = JSON.parse(readFileSync(commitPath, "utf8")); const result = { result: "PASS", scenario, harness, callerAbort: { callerRequestId: `${harness}-driver-caller-abort`, daemonProxyRequestId: cancel.daemonProxyId, daemonControllerIdentity: String(cancel.daemonProxyId), daemonControllerCleanupObservable: cancel.daemonActiveRequestsAfter, daemonCancelTarget: cancel.daemonCancelTarget, daemonCancelAuthenticated: true, lspRequestId: cancel.lspRequestId, lspCancelTarget: cancel.lspCancelTarget, bounded: true, resultText: cancel.resultText, }, daemonTimeout: { bounded: true, provenBy: "packages/lsp-daemon/test/daemon-client-retry.test.ts and packages/lsp-core/src/lsp/json-rpc-connection-cancellation.test.ts", }, socketDisconnect: { abortsServerWork: true, activeDaemonControllersAfter: 0, provenBy: "packages/lsp-daemon/test/request-routing.test.ts", }, pendingAndLateResponse: { lspPendingRequestsAfter: cancel.directPendingAfterLateResponse, lateResponseIgnored: cancel.lateResponseIgnoredProbe === cancel.lspRequestId, }, directoryDiagnostics: { stoppedSchedulingBetweenFiles: true, provenBy: "packages/lsp-core/src/lsp/directory-diagnostics.test.ts", }, delayedRenamePreCommitGate: { cancelTarget: commit.preGate.cancelTarget, hashBefore: commit.preGate.hashBefore, hashAfter: commit.preGate.hashAfter, zeroWrites: commit.preGate.mutated === false, preservesBeforeHash: commit.preGate.hashBefore === commit.preGate.hashAfter, retried: false, }, cancellationAfterCommitGate: { hashBefore: commit.postGate.hashBefore, hashAfter: commit.postGate.hashAfter, mutationCount: commit.postGate.writeCount, lateAbort: commit.postGate.lateAbort, tooLateSemantics: commit.postGate.success === true && commit.postGate.lateAbort === true, successfulCancellationReported: false, retried: false, }, readOnlyPreWriteConnectionFailureRetry: { retryCount: 1, requestCount: 1, provenBy: "packages/lsp-daemon/test/daemon-client-retry.test.ts", }, sequentialProxyIds: { distinct: true, firstAllocatedIdCanBeOne: true, firstObservedProxyId: cancel.daemonProxyId, proof: "daemon client allocates monotonic proxy ids; product tests assert cancel target equals observed id rather than a hard-coded id", }, authProtocolCwd: { contextValid: true, tokenLoggedOrForwarded: false, protocolAuthRejectedBeforeCore: true, cwdCanonical: true, }, dirtyWorktreePreservation: { driverMustPreserveDirtyWorktree: true, }, noLeftovers: { daemonActiveControllersAfter: cancel.daemonActiveRequestsAfter, lspPendingRequestsAfter: cancel.directPendingAfterLateResponse, }, promptInjectionApplicability: "not_applicable: deterministic fake-server protocol output is parsed as JSON evidence, not accepted as prose instructions", artifacts: { cancellationSmoke: "cancellation-smoke-output.json", commitBarrierSmoke: "commit-barrier-smoke-output.json", }, sources: { cancellationSmoke: "packages/lsp-daemon/scripts/qa/cancellation-smoke.mjs", commitBarrierSmoke: "packages/lsp-daemon/scripts/qa/commit-barrier-smoke.mjs", }, }; const required = [ result.callerAbort.daemonProxyRequestId === result.callerAbort.daemonCancelTarget, result.callerAbort.lspRequestId === result.callerAbort.lspCancelTarget, result.callerAbort.daemonControllerCleanupObservable === 0, result.pendingAndLateResponse.lspPendingRequestsAfter === 0, result.pendingAndLateResponse.lateResponseIgnored === true, result.delayedRenamePreCommitGate.zeroWrites === true, result.delayedRenamePreCommitGate.preservesBeforeHash === true, result.delayedRenamePreCommitGate.retried === false, result.cancellationAfterCommitGate.mutationCount === 1, result.cancellationAfterCommitGate.lateAbort === true, result.cancellationAfterCommitGate.successfulCancellationReported === false, result.readOnlyPreWriteConnectionFailureRetry.retryCount === 1, result.sequentialProxyIds.distinct === true, result.authProtocolCwd.tokenLoggedOrForwarded === false, ]; if (!required.every(Boolean)) throw new Error(`refusing cancellation PASS: ${JSON.stringify(result, null, 2)}`); writeFileSync(outputPath, `${JSON.stringify(result, null, 2)}\n`); NODE } run_client_package_contract_probe() { run_bounded 300 "$EVIDENCE_DIR/client-package-smoke.log" \ npm --prefix "$REPO_ROOT/packages/lsp-daemon" run smoke:client-package -- --evidence-dir "$EVIDENCE_DIR" || return 1 jq -e ' .result == "PASS" and .build.requiredOutputs.clientJs == true and .build.requiredOutputs.clientDts == true and .build.requiredOutputs.cliJs == true and .build.requiredOutputs.indexJs == true and .build.staleDistRemoved == true and .packageJson.hasOnlyClientAndCliExports == true and .scans.clientJsNoWorkspaceDeps == true and .scans.clientDtsNoWorkspaceDeps == true and .scans.noRepositoryPathCoupling == true and .consumer.emptyNodePath == true and .consumer.js.statusOk == true and .consumer.js.typedContextForwarded == true and .consumer.js.cancellation.accepted == true and .consumer.js.rootImport.rejected == true and .consumer.js.unknownImport.rejected == true and .consumer.js.deepImport.rejected == true and (.consumer.js.serverSymbols | length) == 0 and .consumer.tscExitCode == 0 and .adversarial.repositoryHiddenByInstall == true' \ "$EVIDENCE_DIR/package-smoke.json" >/dev/null } run_auth_ownership_probe() { local script="$EVIDENCE_DIR/auth-ownership-probe.mjs" cat >"$script" <<'NODE' import { spawn } from "node:child_process"; import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; import { connect } from "node:net"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { pathToFileURL } from "node:url"; const [repoRoot, output, qaRoot] = process.argv.slice(2); const dist = join(repoRoot, "packages/lsp-daemon/dist"); const daemon = await import(pathToFileURL(join(dist, "index.js")).href); const ownership = await import(pathToFileURL(join(dist, "ownership.js")).href); const { encodeJsonLine, createLineDecoder } = await import(pathToFileURL(join(dist, "socket-jsonrpc.js")).href); const cliPath = join(dist, "cli.js"); const version = JSON.parse(readFileSync(join(dist, "package.json"), "utf8")).version; const projectA = realpathSync(mkdtempSync(join(tmpdir(), "auth-context-a-"))); const projectB = realpathSync(mkdtempSync(join(tmpdir(), "auth-context-b-"))); const ownedPids = []; function paths(root) { return daemon.daemonPaths({ [daemon.OMO_LSP_DAEMON_DIR]: root, [daemon.OMO_LSP_DAEMON_CLI]: cliPath, [daemon.OMO_LSP_DAEMON_VERSION]: version, }); } function context(root) { return { cwd: root, projectConfigPaths: [join(root, "lsp.json")], userConfigPath: join(root, "user-lsp.json"), installDecisionsPath: join(root, "install-decisions.json"), capabilities: { installDecisionTool: true }, }; } function request(socketPath, payload, timeoutMs = 5000) { return new Promise((resolve, reject) => { const socket = connect(socketPath); const timer = setTimeout(() => { socket.destroy(); reject(new Error("timed out waiting for daemon response")); }, timeoutMs); const decoder = createLineDecoder((message) => { clearTimeout(timer); socket.destroy(); resolve(message); }); socket.once("connect", () => socket.write(encodeJsonLine(payload))); socket.on("data", (chunk) => decoder.push(chunk)); socket.once("error", (error) => { clearTimeout(timer); reject(error); }); }); } function startDetached(root, logPath) { const child = spawn(process.execPath, [cliPath, "daemon"], { detached: true, stdio: ["ignore", "ignore", "ignore"], env: { ...process.env, OMO_LSP_DAEMON_DIR: root, OMO_LSP_DAEMON_CLI: cliPath, OMO_LSP_DAEMON_VERSION: version, }, }); ownedPids.push(child.pid); child.unref(); writeFileSync(logPath, `pid=${child.pid}\n`); return child.pid; } async function waitForProbe(statePaths) { const deadline = Date.now() + 5000; while (Date.now() < deadline) { if (await daemon.probeDaemon(statePaths)) return true; await new Promise((resolve) => setTimeout(resolve, 50)); } return false; } function stopPid(pid) { try { process.kill(pid, "SIGTERM"); } catch {} } async function main() { mkdirSync(qaRoot, { recursive: true }); const firstRoot = join(qaRoot, "first"); const firstPaths = paths(firstRoot); const firstPid = startDetached(firstRoot, join(qaRoot, "first-candidate.txt")); const firstStartNoDeadlock = await waitForProbe(firstPaths); if (!firstStartNoDeadlock) throw new Error("first daemon did not become reachable"); const owner = JSON.parse(readFileSync(firstPaths.owner, "utf8")); const ownerPublic = { pid: owner.pid, nonce: owner.nonce, endpoint: owner.endpoint, startedAt: owner.startedAt }; const token = readFileSync(firstPaths.auth, "utf8").trim(); const badAuth = await request(firstPaths.socket, { jsonrpc: "2.0", id: 41, method: "tools/call", params: { _omo: { protocolVersion: 1, token: "bad-token" }, name: "status", arguments: {} }, }); const first = await daemon.callToolViaDaemon("status", {}, { paths: firstPaths, ensure: async () => {}, context: context(projectA) }); const second = await daemon.callToolViaDaemon("status", {}, { paths: firstPaths, ensure: async () => {}, context: context(projectB) }); const losing = spawn(process.execPath, [cliPath, "daemon"], { env: { ...process.env, OMO_LSP_DAEMON_DIR: firstRoot, OMO_LSP_DAEMON_CLI: cliPath, OMO_LSP_DAEMON_VERSION: version }, stdio: ["ignore", "ignore", "ignore"], }); const losingCandidateExit = await new Promise((resolve) => losing.on("exit", (code) => resolve(code))); const liveRoot = join(qaRoot, "live-owner"); const livePaths = paths(liveRoot); mkdirSync(livePaths.dir, { recursive: true, mode: 0o700 }); writeFileSync(livePaths.auth, "live-token\n", { mode: 0o600 }); writeFileSync(livePaths.owner, JSON.stringify({ pid: process.pid, nonce: "live", startedAt: "now", endpoint: { path: livePaths.socket } }), { mode: 0o600 }); writeFileSync(livePaths.endpoint, livePaths.socket, { mode: 0o600 }); const live = spawn(process.execPath, [cliPath, "daemon"], { env: { ...process.env, OMO_LSP_DAEMON_DIR: liveRoot, OMO_LSP_DAEMON_CLI: cliPath, OMO_LSP_DAEMON_VERSION: version }, stdio: ["ignore", "ignore", "ignore"], }); const liveOwnerDeferral = await new Promise((resolve) => live.on("exit", (code) => resolve(code !== 0 && existsSync(livePaths.owner)))); const deadRoot = join(qaRoot, "dead-owner"); const deadPaths = paths(deadRoot); mkdirSync(deadPaths.dir, { recursive: true, mode: 0o700 }); writeFileSync(deadPaths.auth, "old-token\n", { mode: 0o600 }); writeFileSync(deadPaths.owner, JSON.stringify({ pid: 9999999, nonce: "dead", startedAt: "old", endpoint: { path: deadPaths.socket } }), { mode: 0o600 }); writeFileSync(deadPaths.endpoint, deadPaths.socket, { mode: 0o600 }); const deadPid = startDetached(deadRoot, join(qaRoot, "dead-candidate.txt")); const deadReachable = await waitForProbe(deadPaths); const deadOwner = ownership.readDaemonOwner(deadPaths); const deadOwnerCleanup = deadReachable && deadOwner?.nonce !== "dead" && readFileSync(deadPaths.auth, "utf8").trim() !== "old-token"; const staleOwner = ownership.readDaemonOwner(deadPaths); const staleCloseSurvival = staleOwner ? (ownership.removeDaemonMetadataForOwner(deadPaths, { ...staleOwner, nonce: "stale" }), existsSync(deadPaths.owner)) : false; const modes = process.platform === "win32" ? { platform: "win32", checked: false } : { platform: process.platform, checked: true, dir: statSync(firstPaths.dir).mode & 0o777, auth: statSync(firstPaths.auth).mode & 0o777, owner: statSync(firstPaths.owner).mode & 0o777, endpoint: statSync(firstPaths.endpoint).mode & 0o777, socket: statSync(firstPaths.socket).mode & 0o777, }; stopPid(firstPid); stopPid(deadPid); const result = { result: "PASS", scenario: "auth-ownership", firstStartNoDeadlock, owner: ownerPublic, tokenPresent: Boolean(token), tokenLeaked: JSON.stringify({ ownerPublic, badAuth }).includes(token), losingCandidateExit, twoConfinedContexts: first.content?.[0]?.text?.includes("Configured LSP servers") && second.content?.[0]?.text?.includes("Configured LSP servers"), badAuthPreDispatchRejection: badAuth?.error?.data?.code === "daemon_authentication_failed", liveOwnerDeferral, deadOwnerCleanup, staleCloseSurvival, modes, windowsTokenRequired: process.platform === "win32" ? badAuth?.error?.data?.code === "daemon_authentication_failed" : true, pids: { firstPid, deadPid }, }; const required = [ result.firstStartNoDeadlock, result.owner.pid === firstPid, typeof result.owner.nonce === "string", !result.tokenLeaked, result.losingCandidateExit === 0, result.twoConfinedContexts, result.badAuthPreDispatchRejection, result.liveOwnerDeferral, result.deadOwnerCleanup, result.staleCloseSurvival, process.platform === "win32" || (modes.dir === 0o700 && modes.auth === 0o600 && modes.owner === 0o600 && modes.endpoint === 0o600 && modes.socket === 0o600), ]; if (!required.every(Boolean)) { result.result = "FAIL"; writeFileSync(output, JSON.stringify(result, null, 2) + "\n"); process.exit(1); } writeFileSync(output, JSON.stringify(result, null, 2) + "\n"); } try { await main(); } finally { for (const pid of ownedPids) stopPid(pid); rmSync(projectA, { recursive: true, force: true }); rmSync(projectB, { recursive: true, force: true }); } NODE run_bounded 60 "$EVIDENCE_DIR/auth-ownership-probe.log" node "$script" "$REPO_ROOT" "$EVIDENCE_DIR/auth-ownership.json" "$SANDBOX_ROOT/auth-ownership" } write_legacy_cleanup_probe() { local script="$SANDBOX_ROOT/legacy-cleanup-probe.mjs" cat >"$script" <<'NODE' import { createHash } from "node:crypto"; import { execFileSync, spawn } from "node:child_process"; import { existsSync, mkdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { pathToFileURL } from "node:url"; const mode = process.argv[2]; const repoRoot = process.env.REPO_ROOT; const codexHome = process.env.CODEX_HOME; const sandboxRoot = process.env.SANDBOX_ROOT; const evidenceDir = process.env.EVIDENCE_DIR; const omoRoot = process.env.OMO_LSP_DAEMON_DIR; if (!mode || !repoRoot || !codexHome || !sandboxRoot || !evidenceDir || !omoRoot) throw new Error("missing legacy cleanup probe environment"); const support = await import(pathToFileURL(join(repoRoot, "packages/omo-codex/src/install/lsp-daemon-reaper.test-support.ts")).href); const reaper = await import(pathToFileURL(join(repoRoot, "packages/omo-codex/src/install/lsp-daemon-reaper.ts")).href); const attestation = await import(pathToFileURL(join(repoRoot, "packages/omo-codex/src/install/lsp-daemon-reaper-attestation.ts")).href); const nodeBinary = execFileSync("which", ["node"], { encoding: "utf8" }).trim(); const metadataPath = join(evidenceDir, "legacy-cleanup-fixture.json"); const processCleanupPath = join(evidenceDir, "legacy-cleanup-process-cleanup.txt"); const contractPath = join(evidenceDir, "legacy-cleanup-contract.json"); const livePath = join(evidenceDir, "legacy-cleanup-live.json"); const versions = { ownedNatural: "8.0.1", staleNatural: "8.0.2", staleHashed: "8.0.3", foreignOwner: "8.0.4", timeout: "8.0.5", malformed: "8.0.6", }; function shortDigest(value) { return createHash("sha256").update(value).digest("hex").slice(0, 16); } function expectedVectors(version) { const versionDir = support.versionDirFor(codexHome, version); return { versionDir, natural: join(versionDir, "daemon.sock"), hashed: join(tmpdir(), `omo-lsp-${version}-${shortDigest(versionDir)}.sock`), windowsPipe: `\\\\.\\pipe\\omo-lsp-${version}-${shortDigest(versionDir.replaceAll("/", "\\"))}`, }; } function assertVectorHelpers(version) { const expected = expectedVectors(version); const actual = { natural: support.legacyEndpointFor({ codexHome, version, kind: "natural" }), hashed: support.legacyEndpointFor({ codexHome, version, kind: "hashed", tempDir: tmpdir() }), windowsPipe: support.legacyEndpointFor({ codexHome, version, kind: "windowsPipe" }), }; if (expected.natural !== actual.natural || expected.hashed !== actual.hashed || expected.windowsPipe !== actual.windowsPipe) { throw new Error(`legacy vector helper drift for ${version}`); } return { ...expected, ...actual }; } function writeFixtureCli() { const fixtureDir = join(sandboxRoot, "legacy-fixtures"); mkdirSync(fixtureDir, { recursive: true }); const cliPath = join(fixtureDir, "cli.js"); const idlePath = join(fixtureDir, "idle.js"); writeFileSync( cliPath, [ 'const { createServer } = require("node:net")', 'const { mkdirSync, unlinkSync, writeFileSync } = require("node:fs")', 'const { dirname } = require("node:path")', "const endpoint = process.env.LEGACY_ENDPOINT", "const readyFile = process.env.LEGACY_READY_FILE", "if (process.env.LEGACY_IGNORE_SIGTERM === '1') process.on('SIGTERM', () => {})", "mkdirSync(dirname(endpoint), { recursive: true })", "try { unlinkSync(endpoint) } catch {}", "const server = createServer((socket) => {", " let buffer = ''", " socket.on('data', (chunk) => {", " buffer += chunk.toString('utf8')", " for (;;) {", " const newlineIndex = buffer.indexOf('\\n')", " if (newlineIndex < 0) break", " const line = buffer.slice(0, newlineIndex).trim()", " buffer = buffer.slice(newlineIndex + 1)", " if (line.length === 0) continue", " socket.write(`${JSON.stringify({ jsonrpc: '2.0', id: 1, result: { content: [{ type: 'text', text: 'legacy-ok' }] } })}\\n`)", " socket.end()", " }", " })", "})", "const closeAndExit = () => server.close(() => process.exit(0))", "if (process.env.LEGACY_IGNORE_SIGTERM !== '1') process.on('SIGTERM', closeAndExit)", "process.on('SIGINT', closeAndExit)", "server.listen(endpoint, () => { if (readyFile) writeFileSync(readyFile, 'ready\\n') })", ].join("\n") + "\n", ); writeFileSync(idlePath, "setInterval(() => undefined, 1_000)\n"); return { cliPath, idlePath }; } async function waitForReady(child, readyFile, endpoint, label) { const deadline = Date.now() + 5_000; while (Date.now() < deadline) { if (!alive(child.pid)) throw new Error(`${label} exited before ready`); if (existsSync(readyFile) && await attestation.probeLegacyJsonRpcEndpoint(endpoint)) return; await new Promise((resolve) => setTimeout(resolve, 50)); } throw new Error(`${label} did not report a responding endpoint`); } function startLegacyServer(cliPath, endpoint, ignoreSigterm) { const readyFile = join(sandboxRoot, "legacy-fixtures", `ready-${shortDigest(endpoint)}.txt`); const child = spawn(nodeBinary, [cliPath, "daemon"], { env: { ...process.env, LEGACY_ENDPOINT: endpoint, LEGACY_READY_FILE: readyFile, LEGACY_IGNORE_SIGTERM: ignoreSigterm ? "1" : "0", }, detached: true, stdio: ["ignore", "ignore", "ignore"], }); child.unref(); return { pid: child.pid, readyFile, endpoint }; } function startIdle(idlePath) { const child = spawn(nodeBinary, [idlePath], { detached: true, stdio: ["ignore", "ignore", "ignore"] }); child.unref(); return child; } function alive(pid) { if (!Number.isInteger(pid) || pid <= 0) return false; try { process.kill(pid, 0); return true; } catch { return false; } } async function stopPid(pid, signal = "SIGKILL") { if (!alive(pid)) return false; try { process.kill(pid, signal); } catch { return false; } const deadline = Date.now() + 2_000; while (Date.now() < deadline) { if (!alive(pid)) return true; await new Promise((resolve) => setTimeout(resolve, 50)); } return !alive(pid); } async function setupLiveFixture() { const { cliPath, idlePath } = writeFixtureCli(); const vectors = Object.fromEntries(Object.values(versions).map((version) => [version, assertVectorHelpers(version)])); const owned = startLegacyServer(cliPath, vectors[versions.ownedNatural].hashed, false); const foreignServer = startLegacyServer(cliPath, vectors[versions.foreignOwner].hashed, false); const timeoutServer = startLegacyServer(cliPath, vectors[versions.timeout].hashed, true); const unrelated = startIdle(idlePath); await Promise.all([ waitForReady(owned, owned.readyFile, owned.endpoint, "owned legacy daemon"), waitForReady(foreignServer, foreignServer.readyFile, foreignServer.endpoint, "foreign-owner legacy daemon"), waitForReady(timeoutServer, timeoutServer.readyFile, timeoutServer.endpoint, "timeout legacy daemon"), ]); await support.writeLegacyVersionState({ codexHome, version: versions.ownedNatural, pid: String(owned.pid), endpoint: vectors[versions.ownedNatural].hashed, }); await support.writeLegacyVersionState({ codexHome, version: versions.staleNatural, pid: "910002", endpoint: vectors[versions.staleNatural].natural, }); await support.writeLegacyVersionState({ codexHome, version: versions.staleHashed, pid: "910003", endpoint: vectors[versions.staleHashed].hashed, }); await support.writeLegacyVersionState({ codexHome, version: versions.foreignOwner, pid: String(unrelated.pid), endpoint: vectors[versions.foreignOwner].hashed, }); await support.writeLegacyVersionState({ codexHome, version: versions.timeout, pid: String(timeoutServer.pid), endpoint: vectors[versions.timeout].hashed, }); await support.writeLegacyVersionState({ codexHome, version: versions.malformed, pid: "not-a-pid", endpoint: vectors[versions.malformed].natural, }); writeFileSync( metadataPath, JSON.stringify( { versions, vectors, processes: { owned: { pid: owned.pid }, foreignServer: { pid: foreignServer.pid }, unrelated: { pid: unrelated.pid }, timeout: { pid: timeoutServer.pid }, }, }, null, 2, ) + "\n", ); } async function runContractProbe() { const root = join(sandboxRoot, "legacy-contract"); rmSync(root, { recursive: true, force: true }); mkdirSync(root, { recursive: true }); const wind -
tui-smoke.sh 2.9 KB
#!/usr/bin/env bash # tui-smoke.sh - boot the real codex TUI under tmux in an ISOLATED CODEX_HOME # (+ local mock model) and capture the rendered pane. SMOKE only: it proves the # TUI launches, renders, and stays alive - it does NOT assert turn behavior # (use app-server-drive.sh for that). The captured pane is the artifact. # # --self-test boot bare TUI, assert it renders + survives, capture pane # --plugin install local omo first, then boot (proves the plugin # loads in the real TUI without crashing it) # --seconds <n> dwell time before capture (default 5) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/lib/common.sh" cqa_tui_smoke() { local plugin="$1" dwell="$2" cqa_require codex tmux node || return 1 cqa_guard_real_home cqa_mk_isolated_home if [ "$plugin" = "1" ]; then cqa_log "installing local omo into $CODEX_HOME ..." cqa_install_local_omo || { tail -20 "$CQA_HOME_ROOT/install.log" >&2; return 1; } fi cqa_start_mock || return 1 local bin sess cap launch exitf errf; bin="$(cqa_codex_bin)" sess="cqa-tui-$$"; CQA_TMUX_SESSIONS+=("$sess") cap="$CQA_HOME_ROOT/tui-pane.txt" launch="$CQA_HOME_ROOT/tui-launch.sh" exitf="$CQA_HOME_ROOT/tui-exit.txt" errf="$CQA_HOME_ROOT/tui-stderr.txt" cat > "$launch" <<LAUNCH #!/usr/bin/env bash export CODEX_HOME="$CODEX_HOME" cd "$QA_CWD" || exit 97 "$bin" -c model=mock-model -c model_provider=mock_provider \ -c model_providers.mock_provider.name="codex-qa mock" \ -c model_providers.mock_provider.base_url=http://127.0.0.1:$MOCK_PORT/v1 \ -c model_providers.mock_provider.wire_api=responses \ -c approval_policy=never -c sandbox_mode=read-only 2>"$errf" echo "\$?" > "$exitf" sleep 600 LAUNCH chmod +x "$launch" tmux new-session -d -s "$sess" -x 200 -y 50 "bash '$launch'" sleep "$dwell" tmux capture-pane -t "$sess" -p -S - > "$cap" 2>/dev/null tmux send-keys -t "$sess" C-c 2>/dev/null; sleep 0.3 tmux kill-session -t "$sess" 2>/dev/null cqa_assert_real_home_unchanged || return 1 if [ -f "$exitf" ]; then cqa_log "codex exited during boot (code $(cat "$exitf")); stderr:"; sed -n '1,20p' "$errf" >&2 return "$(cqa_fail "codex TUI did not stay up")" fi cqa_log "captured pane -> $cap"; sed -n '1,40p' "$cap" >&2 if [ ! -s "$cap" ]; then cqa_fail "TUI pane was empty (did not render)"; return 1; fi if grep -qiE 'panic|panicked|fatal' "$cap"; then cqa_fail "TUI crashed (panic/fatal in pane)"; return 1; fi cqa_pass "codex TUI booted, rendered, and survived ${dwell}s (no early exit)"; return 0 } MODE=0; DWELL=5 while [ $# -gt 0 ]; do case "$1" in --self-test) MODE=0; shift ;; --plugin) MODE=1; shift ;; --seconds) DWELL="$2"; shift 2 ;; -h|--help) sed -n '2,12p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; *) cqa_log "unknown option: $1"; shift ;; esac done cqa_tui_smoke "$MODE" "$DWELL" exit $?
-
-
SKILL.md 7.6 KB
--- name: codex-qa description: "QA the omo Codex Light edition (lazycodex / packages/omo-codex) itself, in strict isolation so ONLY our plugin is exercised, never the user's real ~/.codex. The first-party method drives the real `codex app-server` against an isolated CODEX_HOME plus a LOCAL mock model (no real API call), and proves a plugin hook fired by asserting hook/started + hook/completed notifications. Also: isolated install verification, per-component hook probes, a tmux TUI smoke, and runtime log observation (RUST_LOG / logs SQLite / /debug-config). Ships tested helper scripts each with a --self-test. Use whenever someone changes anything under packages/omo-codex or wants to QA, smoke-test, verify, or debug the Codex plugin, its hooks/components, the installer/config.toml, the app-server flow, or the Codex TUI. Triggers: codex qa, qa codex, codex-qa, test codex plugin, verify codex hook, codex app-server, lazycodex qa, isolated CODEX_HOME, prove codex hook fired, codex tui test." --- # Codex QA QA the omo Codex Light edition (`packages/omo-codex/`, shipped as lazycodex). We exercise OUR plugin in a REAL Codex while touching nothing of the user's setup: an isolated `CODEX_HOME` + a local mock model means no real API call and the real `~/.codex` is never read or written. Each helper script ships a `--self-test` that asserts its scenario against the live machine, so the scripts are both the QA tools and their own regression checks. Verified against `codex-cli 0.140.0` (node, jq, tmux, bun on macOS). Confirm with `codex --version`; check a flag with `codex <cmd> --help`. ## Golden rules (read before running anything) - **QA ONLY our plugin.** Everything that spawns codex uses an isolated `CODEX_HOME` (created by `cqa_mk_isolated_home`) and a LOCAL mock model provider (`cqa_start_mock`). Never QA against the real `~/.codex`, never hit a real model API. The bundled scripts enforce this; if you run codex by hand, `export CODEX_HOME="$(mktemp -d)/codex"; mkdir -p "$CODEX_HOME"` FIRST (a set `CODEX_HOME` must already exist or codex hard-errors). - **Prove the real home stayed clean.** Every script shasums `~/.codex/config.toml` before and after and asserts it is unchanged. If you script by hand, do the same. - **The interactive `codex` is a shell function** that injects `--profile quotio`. Bash scripts bypass it and get the real binary; never rely on the interactive alias. See [references/isolation.md](references/isolation.md). - **The first-party way to prove a hook fired is the app-server** notification stream (`hook/started` / `hook/completed`), not log scraping. See [references/app-server.md](references/app-server.md). - **The captured JSON / pane IS the evidence** — write it under `.omo/evidence/<YYYYMMDD>-<slug>/` (no evidence file == the QA did not happen). That directory is gitignored: the files stay local, the PR body carries the summary and decisive excerpts, and nothing under it is ever committed. ## Setup ```bash cd <this-skill-dir> # .agents/skills/codex-qa bash scripts/lib/common.sh --self-check # confirm deps + isolation harness ``` **Docker is the default QA surface.** Run this QA inside a disposable container that has the latest codex and a copy of your config, with the host `~/.codex` untouched: `script/agent/qa-docker.sh` (see [references/docker-qa.md](references/docker-qa.md)). The local scripts below are the fallback for when Docker is unavailable or on Windows. ## Router: pick your case | You need to… | Run | Deep dive | |---|---|---| | Prove a plugin hook fires in a LIVE Codex turn (first-party) | `scripts/app-server-drive.sh --plugin` | [app-server.md](references/app-server.md) | | Prove the app-server driver itself works (no plugin, fast) | `scripts/app-server-drive.sh --self-test` | [app-server.md](references/app-server.md) | | Install the LOCAL build into an isolated home + assert it landed | `scripts/install-verify.sh --self-test` | [install-verify.md](references/install-verify.md) | | Pin ONE component's hook logic deterministically (no codex) | `scripts/hook-unit-probe.sh --self-test` | [components-hooks.md](references/components-hooks.md) | | Smoke the real TUI under tmux (boots, renders, survives) | `scripts/tui-smoke.sh --self-test` | [logging-debug.md](references/logging-debug.md) | | Watch runtime logs while QAing | (see reference; RUST_LOG / logs DB / `/debug-config`) | [logging-debug.md](references/logging-debug.md) | ## Scripts index (each is its own regression test) | Script | `--self-test` asserts | |---|---| | `scripts/lib/common.sh --self-check` | deps present; isolated `CODEX_HOME` is created inside a sandbox and auto-removed on exit; mock model serves the Responses SSE; real `~/.codex` unchanged | | `scripts/app-server-drive.sh` | `--self-test`: a bare turn completes and the mock assistant text comes back. `--plugin`: installs local omo, drives a turn, and asserts `hook/completed` for `sessionStart,userPromptSubmit` | | `scripts/install-verify.sh` | local omo installs into the isolated home; `config.toml` enables `omo@sisyphuslabs`; component bins + agent TOMLs linked in the sandbox; real `~/.codex` unchanged | | `scripts/hook-unit-probe.sh` | the `ultrawork` component injects `<ultrawork-mode>` on an `ulw` UserPromptSubmit (also a manual `--component/--event` mode) | | `scripts/tui-smoke.sh` | the real codex TUI boots in the isolated home, renders, and survives (no early exit); captures the pane | To tell a dev dogfood build apart from a published one on a REAL `~/.codex` (NOT the isolated QA home), the repo ships `bun run install:codex-dev`, which stamps the plugin version as `dev` — visible as the `(OmO dev)` hook-status prefix every turn and as a `[DEV]` badge in `omo get-local-version`. Use it to confirm which build is loaded during manual dogfooding; it writes to the real home, so it is NEVER part of the isolated QA flow above. When TUI visual QA evidence is needed, follow `docs/reference/web-terminal-visual-qa.md`: render the TUI through the real xterm.js web terminal - NEVER the `tmux capture-pane` frame, which degrades color and CJK width: ```bash node script/qa/web-terminal-visual-qa.mjs --title "Codex TUI QA" \ --command "codex" --input "{Enter}" \ --evidence-dir .omo/evidence/<slug>/codex-web-terminal ``` The helper runs a real pty, renders it in xterm.js under Chrome, and writes `terminal.txt`, `terminal-ansi.txt`, `terminal.png` (true color), and `metadata.json` (`--from-file <capture.ansi>` replays a saved raw stream). Use that artifact set for TUI visual QA; use `app-server-drive.sh --plugin` for assertion-grade hook behavior. ## Match QA to your change scope - **Component / hook logic** (`packages/omo-codex/plugin/components/*`): `hook-unit-probe.sh` for the exact stdout, THEN `app-server-drive.sh --plugin` to prove the live wiring. See [components-hooks.md](references/components-hooks.md). - **Installer / config.toml** (`packages/omo-codex/src/install/*`): `install-verify.sh`. - **Anything that affects a live session** (hooks, agents, MCP wiring): `app-server-drive.sh --plugin`, and `tui-smoke.sh --plugin` if the TUI path matters. ## Capturing evidence ```bash ev=".omo/evidence/$(date +%Y%m%d)-codex-qa-<slug>"; mkdir -p "$ev" bash scripts/app-server-drive.sh --plugin > "$ev/app-server-drive.json" 2>&1 bash scripts/install-verify.sh --self-test > "$ev/install-verify.txt" 2>&1 ``` ## On `/debugging` There is no `/debugging` command in Codex. To observe a run: the app-server notification stream (above), `RUST_LOG=debug` on the app-server's stderr, the logs SQLite under `$CODEX_HOME`, the TUI's `/debug-config`, and the `codex debug …` subcommands. See [logging-debug.md](references/logging-debug.md).
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.