Claude Cursor opencode Skill

isolated-agent-eval

Use when an unattended agent session in a test could act on Ryugi Cloud — publish, auto-publish, or use a Cloud sign-in — including writing-skills baseline and with-skill runs and pressure scenarios that use artifacts with `--auto-approve`.

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Download innei-ryugi-.claude_skills_isolated-agent-eval-fe17c7b.zip · 6 KB

Install

skills CLI npx skills add https://github.com/Innei/ryugi/tree/main/.claude/skills/isolated-agent-eval
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install innei-ryugi@llmmart
Git git clone https://github.com/Innei/ryugi.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole innei/ryugi collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Isolated Agent Eval

An agent with auto-approve runs whatever it decides to. When the run can see a Ryugi Cloud sign-in, that includes publishing to your account: a host-side baseline run once published its artboard publicly. Such runs happen in a throwaway container with no sign-in, writable only to its own output directory.

A run whose Cloud sign-in is already hidden, as tests/skill-bootstrap-eval does with a private XDG_CONFIG_HOME per session, may stay on the host. The recipe below runs the OMP CLI; there is no container recipe for Claude Code, Codex or other harnesses, and none is planned.

Setup, once per session

S=.claude/skills/isolated-agent-eval
docker build --build-arg OMP_VERSION="$(omp --version | cut -d/ -f2)" -t ryugi-agent-eval:local "$S/docker"
python3 -I "$S/provider-config.py" deepseek "$SCRATCH/secrets/models.yml"

$SCRATCH is your session scratchpad, outside the repository. The container gets that one provider and its key, never ~/.omp.

Then smoke-test before the first real run, and read the output:

mkdir -p "$SCRATCH/empty"
bash "$S/run.sh" --fixture "$SCRATCH/empty" --out "$SCRATCH/runs/smoke" --models-yml "$SCRATCH/secrets/models.yml" \
  --prompt "Start the ryugi artifacts server for artboard/smoke and reply with the Open link it printed."

Pass when status ends done, env.txt names the model, and the last message of turn1.jsonl carries an Open: http://localhost:… link. Anything else is an infrastructure failure; fix it before running scenarios.

A run

bash "$S/run.sh" --fixture "$SCRATCH/runs/<arm>-<case>-<rep>" --models-yml "$SCRATCH/secrets/models.yml" \
  --prompt "<task>" --turns 3 [--next "按推荐,继续"] [--plugin < dir > --out < checkout > ]
  • Each run gets its own --out. Run several in parallel as background jobs, at most Docker's memory divided by --memory (3g).
  • --out holds status, env.txt, turnN.jsonl, turnN.err, after-turnN.txt (git status after each turn), repo/ (the fixture after the run, with .ryugi/), sessions/ and container.log.
  • --resume <run dir> instead of --fixture continues that run's session and repo in a new --out; --prompt is the next turn's message. To test one turn many times, run the turns before it once with --turns 1 and replay from that run. The source run is mounted read-only and never changed.
  • --plugin is the checkout whose skills are under test; default this repository. It is mounted read-only but live: edits to it change what running agents read. Give each arm its own worktree (git worktree add .worktrees/eval-baseline <rev>) so baseline and candidate can run at once. A worktree has no gitignored local artifacts build, so its runs use the npm release.
  • Symlinks in the checkout pointing outside it (a local ryugi-cloud build) are mounted read-only at their own paths automatically.
  • The container has no Ryugi Cloud sign-in, so nothing is published.

Rules

  • A session that could act on Ryugi Cloud does not run on the host with your sign-in visible. Not for "just one quick check".
  • Never mount a home directory, ~/.omp, ~/.claude, ~/.ryugi or a credentials file other than the provider config.
  • run.sh runs from a private copy, so editing it mid-run is safe. Anything else a run executes from the host (your own batch script, a scorer it calls) is not: never edit or format it while runs are in flight. The Dockerfile and entrypoint.sh are baked into the image at build; rebuild before the next run, not during one.
  • Never export variables in a persistent shell to isolate something; they leak into every later command.
  • When done: delete $SCRATCH/secrets, and check docker ps --filter name=agent-eval is empty.
  • In the eval record, name the image (env.txt) and say the runs were containerized.

Red Flags

Thought Reality
"I'll give each run its own server home and port on the host" One host run still published its artboard publicly to Cloud; another deleted ~/.ryugi/server/start.lock.
"I'll fix the runner script now; the runs already started" The running scripts read the new file and broke mid-run. Wait, or copy to a new name.
"Formatting the skill folder doesn't change behaviour" prettier --write rewrote a running script and five runs hit unexpected EOF. A formatter is an edit.
"Mounting ~/.omp is simpler than extracting the provider" It hands every key and session on the machine to an agent with auto-approve.
"The checkout is read-only in the container, so I can edit it" Read-only to the container, live to it too. Agents read your edits mid-run.
Files (ryugi)
  • docker
    • Dockerfile 895 B · in bundle
    • entrypoint.sh 2 KB
      Loading…
  • provider-config.py 1 KB
    Loading…
  • run.sh 3.7 KB
    Loading…
  • SKILL.md 5.3 KB
    ---
    name: isolated-agent-eval
    description: Use when an unattended agent session in a test could act on Ryugi Cloud — publish, auto-publish, or use a Cloud sign-in — including writing-skills baseline and with-skill runs and pressure scenarios that use artifacts with `--auto-approve`.
    ---
    
    # Isolated Agent Eval
    
    An agent with auto-approve runs whatever it decides to. When the run can see a Ryugi Cloud sign-in, that includes publishing to your account: a host-side baseline run once published its artboard publicly. Such runs happen in a throwaway container with no sign-in, writable only to its own output directory.
    
    A run whose Cloud sign-in is already hidden, as `tests/skill-bootstrap-eval` does with a private `XDG_CONFIG_HOME` per session, may stay on the host. The recipe below runs the OMP CLI; there is no container recipe for Claude Code, Codex or other harnesses, and none is planned.
    
    ## Setup, once per session
    
    ```bash
    S=.claude/skills/isolated-agent-eval
    docker build --build-arg OMP_VERSION="$(omp --version | cut -d/ -f2)" -t ryugi-agent-eval:local "$S/docker"
    python3 -I "$S/provider-config.py" deepseek "$SCRATCH/secrets/models.yml"
    ```
    
    `$SCRATCH` is your session scratchpad, outside the repository. The container gets that one provider and its key, never `~/.omp`.
    
    Then smoke-test before the first real run, and read the output:
    
    ```bash
    mkdir -p "$SCRATCH/empty"
    bash "$S/run.sh" --fixture "$SCRATCH/empty" --out "$SCRATCH/runs/smoke" --models-yml "$SCRATCH/secrets/models.yml" \
      --prompt "Start the ryugi artifacts server for artboard/smoke and reply with the Open link it printed."
    ```
    
    Pass when `status` ends `done`, `env.txt` names the model, and the last message of `turn1.jsonl` carries an `Open: http://localhost:…` link. Anything else is an infrastructure failure; fix it before running scenarios.
    
    ## A run
    
    ```bash
    bash "$S/run.sh" --fixture "$SCRATCH/runs/<arm>-<case>-<rep>" --models-yml "$SCRATCH/secrets/models.yml" \
      --prompt "<task>" --turns 3 [--next "按推荐,继续"] [--plugin < dir > --out < checkout > ]
    ```
    
    - Each run gets its own `--out`. Run several in parallel as background jobs, at most Docker's memory divided by `--memory` (3g).
    - `--out` holds `status`, `env.txt`, `turnN.jsonl`, `turnN.err`, `after-turnN.txt` (`git status` after each turn), `repo/` (the fixture after the run, with `.ryugi/`), `sessions/` and `container.log`.
    - `--resume <run dir>` instead of `--fixture` continues that run's session and repo in a new `--out`; `--prompt` is the next turn's message. To test one turn many times, run the turns before it once with `--turns 1` and replay from that run. The source run is mounted read-only and never changed.
    - `--plugin` is the checkout whose skills are under test; default this repository. It is mounted read-only but live: edits to it change what running agents read. Give each arm its own worktree (`git worktree add .worktrees/eval-baseline <rev>`) so baseline and candidate can run at once. A worktree has no gitignored local artifacts build, so its runs use the npm release.
    - Symlinks in the checkout pointing outside it (a local `ryugi-cloud` build) are mounted read-only at their own paths automatically.
    - The container has no Ryugi Cloud sign-in, so nothing is published.
    
    ## Rules
    
    - A session that could act on Ryugi Cloud does not run on the host with your sign-in visible. Not for "just one quick check".
    - Never mount a home directory, `~/.omp`, `~/.claude`, `~/.ryugi` or a credentials file other than the provider config.
    - `run.sh` runs from a private copy, so editing it mid-run is safe. Anything else a run executes from the host (your own batch script, a scorer it calls) is not: never edit or format it while runs are in flight. The Dockerfile and `entrypoint.sh` are baked into the image at build; rebuild before the next run, not during one.
    - Never `export` variables in a persistent shell to isolate something; they leak into every later command.
    - When done: delete `$SCRATCH/secrets`, and check `docker ps --filter name=agent-eval` is empty.
    - In the eval record, name the image (`env.txt`) and say the runs were containerized.
    
    ## Red Flags
    
    | Thought                                                        | Reality                                                                                                    |
    | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
    | "I'll give each run its own server home and port on the host"  | One host run still published its artboard publicly to Cloud; another deleted `~/.ryugi/server/start.lock`. |
    | "I'll fix the runner script now; the runs already started"     | The running scripts read the new file and broke mid-run. Wait, or copy to a new name.                      |
    | "Formatting the skill folder doesn't change behaviour"         | `prettier --write` rewrote a running script and five runs hit `unexpected EOF`. A formatter is an edit.    |
    | "Mounting `~/.omp` is simpler than extracting the provider"    | It hands every key and session on the machine to an agent with auto-approve.                               |
    | "The checkout is read-only in the container, so I can edit it" | Read-only to the container, live to it too. Agents read your edits mid-run.                                |
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.