isolated-agent-eval
Use when an unattended agent session in a test could act on Ryugi Cloud — publish, auto-publish, or use a Cloud sign-in — including writing-skills baseline and with-skill runs and pressure scenarios that use artifacts with `--auto-approve`.
Install
npx skills add https://github.com/Innei/ryugi/tree/main/.claude/skills/isolated-agent-eval
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install innei-ryugi@llmmart
git clone https://github.com/Innei/ryugi.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole innei/ryugi collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Isolated Agent Eval
An agent with auto-approve runs whatever it decides to. When the run can see a Ryugi Cloud sign-in, that includes publishing to your account: a host-side baseline run once published its artboard publicly. Such runs happen in a throwaway container with no sign-in, writable only to its own output directory.
A run whose Cloud sign-in is already hidden, as tests/skill-bootstrap-eval does with a private XDG_CONFIG_HOME per session, may stay on the host. The recipe below runs the OMP CLI; there is no container recipe for Claude Code, Codex or other harnesses, and none is planned.
Setup, once per session
S=.claude/skills/isolated-agent-eval
docker build --build-arg OMP_VERSION="$(omp --version | cut -d/ -f2)" -t ryugi-agent-eval:local "$S/docker"
python3 -I "$S/provider-config.py" deepseek "$SCRATCH/secrets/models.yml"
$SCRATCH is your session scratchpad, outside the repository. The container gets that one provider and its key, never ~/.omp.
Then smoke-test before the first real run, and read the output:
mkdir -p "$SCRATCH/empty"
bash "$S/run.sh" --fixture "$SCRATCH/empty" --out "$SCRATCH/runs/smoke" --models-yml "$SCRATCH/secrets/models.yml" \
--prompt "Start the ryugi artifacts server for artboard/smoke and reply with the Open link it printed."
Pass when status ends done, env.txt names the model, and the last message of turn1.jsonl carries an Open: http://localhost:… link. Anything else is an infrastructure failure; fix it before running scenarios.
A run
bash "$S/run.sh" --fixture "$SCRATCH/runs/<arm>-<case>-<rep>" --models-yml "$SCRATCH/secrets/models.yml" \
--prompt "<task>" --turns 3 [--next "按推荐,继续"] [--plugin < dir > --out < checkout > ]
- Each run gets its own
--out. Run several in parallel as background jobs, at most Docker's memory divided by--memory(3g). --outholdsstatus,env.txt,turnN.jsonl,turnN.err,after-turnN.txt(git statusafter each turn),repo/(the fixture after the run, with.ryugi/),sessions/andcontainer.log.--resume <run dir>instead of--fixturecontinues that run's session and repo in a new--out;--promptis the next turn's message. To test one turn many times, run the turns before it once with--turns 1and replay from that run. The source run is mounted read-only and never changed.--pluginis the checkout whose skills are under test; default this repository. It is mounted read-only but live: edits to it change what running agents read. Give each arm its own worktree (git worktree add .worktrees/eval-baseline <rev>) so baseline and candidate can run at once. A worktree has no gitignored local artifacts build, so its runs use the npm release.- Symlinks in the checkout pointing outside it (a local
ryugi-cloudbuild) are mounted read-only at their own paths automatically. - The container has no Ryugi Cloud sign-in, so nothing is published.
Rules
- A session that could act on Ryugi Cloud does not run on the host with your sign-in visible. Not for "just one quick check".
- Never mount a home directory,
~/.omp,~/.claude,~/.ryugior a credentials file other than the provider config. run.shruns from a private copy, so editing it mid-run is safe. Anything else a run executes from the host (your own batch script, a scorer it calls) is not: never edit or format it while runs are in flight. The Dockerfile andentrypoint.share baked into the image at build; rebuild before the next run, not during one.- Never
exportvariables in a persistent shell to isolate something; they leak into every later command. - When done: delete
$SCRATCH/secrets, and checkdocker ps --filter name=agent-evalis empty. - In the eval record, name the image (
env.txt) and say the runs were containerized.
Red Flags
| Thought | Reality |
|---|---|
| "I'll give each run its own server home and port on the host" | One host run still published its artboard publicly to Cloud; another deleted ~/.ryugi/server/start.lock. |
| "I'll fix the runner script now; the runs already started" | The running scripts read the new file and broke mid-run. Wait, or copy to a new name. |
| "Formatting the skill folder doesn't change behaviour" | prettier --write rewrote a running script and five runs hit unexpected EOF. A formatter is an edit. |
"Mounting ~/.omp is simpler than extracting the provider" |
It hands every key and session on the machine to an agent with auto-approve. |
| "The checkout is read-only in the container, so I can edit it" | Read-only to the container, live to it too. Agents read your edits mid-run. |
Files (ryugi)
-
docker
-
Dockerfile 895 B · in bundle
-
entrypoint.sh 2 KB
Loading…
-
-
provider-config.py 1 KB
Loading… -
run.sh 3.7 KB
Loading… -
SKILL.md 5.3 KB
--- name: isolated-agent-eval description: Use when an unattended agent session in a test could act on Ryugi Cloud — publish, auto-publish, or use a Cloud sign-in — including writing-skills baseline and with-skill runs and pressure scenarios that use artifacts with `--auto-approve`. --- # Isolated Agent Eval An agent with auto-approve runs whatever it decides to. When the run can see a Ryugi Cloud sign-in, that includes publishing to your account: a host-side baseline run once published its artboard publicly. Such runs happen in a throwaway container with no sign-in, writable only to its own output directory. A run whose Cloud sign-in is already hidden, as `tests/skill-bootstrap-eval` does with a private `XDG_CONFIG_HOME` per session, may stay on the host. The recipe below runs the OMP CLI; there is no container recipe for Claude Code, Codex or other harnesses, and none is planned. ## Setup, once per session ```bash S=.claude/skills/isolated-agent-eval docker build --build-arg OMP_VERSION="$(omp --version | cut -d/ -f2)" -t ryugi-agent-eval:local "$S/docker" python3 -I "$S/provider-config.py" deepseek "$SCRATCH/secrets/models.yml" ``` `$SCRATCH` is your session scratchpad, outside the repository. The container gets that one provider and its key, never `~/.omp`. Then smoke-test before the first real run, and read the output: ```bash mkdir -p "$SCRATCH/empty" bash "$S/run.sh" --fixture "$SCRATCH/empty" --out "$SCRATCH/runs/smoke" --models-yml "$SCRATCH/secrets/models.yml" \ --prompt "Start the ryugi artifacts server for artboard/smoke and reply with the Open link it printed." ``` Pass when `status` ends `done`, `env.txt` names the model, and the last message of `turn1.jsonl` carries an `Open: http://localhost:…` link. Anything else is an infrastructure failure; fix it before running scenarios. ## A run ```bash bash "$S/run.sh" --fixture "$SCRATCH/runs/<arm>-<case>-<rep>" --models-yml "$SCRATCH/secrets/models.yml" \ --prompt "<task>" --turns 3 [--next "按推荐,继续"] [--plugin < dir > --out < checkout > ] ``` - Each run gets its own `--out`. Run several in parallel as background jobs, at most Docker's memory divided by `--memory` (3g). - `--out` holds `status`, `env.txt`, `turnN.jsonl`, `turnN.err`, `after-turnN.txt` (`git status` after each turn), `repo/` (the fixture after the run, with `.ryugi/`), `sessions/` and `container.log`. - `--resume <run dir>` instead of `--fixture` continues that run's session and repo in a new `--out`; `--prompt` is the next turn's message. To test one turn many times, run the turns before it once with `--turns 1` and replay from that run. The source run is mounted read-only and never changed. - `--plugin` is the checkout whose skills are under test; default this repository. It is mounted read-only but live: edits to it change what running agents read. Give each arm its own worktree (`git worktree add .worktrees/eval-baseline <rev>`) so baseline and candidate can run at once. A worktree has no gitignored local artifacts build, so its runs use the npm release. - Symlinks in the checkout pointing outside it (a local `ryugi-cloud` build) are mounted read-only at their own paths automatically. - The container has no Ryugi Cloud sign-in, so nothing is published. ## Rules - A session that could act on Ryugi Cloud does not run on the host with your sign-in visible. Not for "just one quick check". - Never mount a home directory, `~/.omp`, `~/.claude`, `~/.ryugi` or a credentials file other than the provider config. - `run.sh` runs from a private copy, so editing it mid-run is safe. Anything else a run executes from the host (your own batch script, a scorer it calls) is not: never edit or format it while runs are in flight. The Dockerfile and `entrypoint.sh` are baked into the image at build; rebuild before the next run, not during one. - Never `export` variables in a persistent shell to isolate something; they leak into every later command. - When done: delete `$SCRATCH/secrets`, and check `docker ps --filter name=agent-eval` is empty. - In the eval record, name the image (`env.txt`) and say the runs were containerized. ## Red Flags | Thought | Reality | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | "I'll give each run its own server home and port on the host" | One host run still published its artboard publicly to Cloud; another deleted `~/.ryugi/server/start.lock`. | | "I'll fix the runner script now; the runs already started" | The running scripts read the new file and broke mid-run. Wait, or copy to a new name. | | "Formatting the skill folder doesn't change behaviour" | `prettier --write` rewrote a running script and five runs hit `unexpected EOF`. A formatter is an edit. | | "Mounting `~/.omp` is simpler than extracting the provider" | It hands every key and session on the machine to an agent with auto-approve. | | "The checkout is read-only in the container, so I can edit it" | Read-only to the container, live to it too. Agents read your edits mid-run. |
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.