start
Orient users on RELAI's capabilities, current eval status and next steps, or initialize RELAI records when a workflow needs them. Route clear requests directly to the relevant skill and reuse existing project infrastructure.
Install
npx skills add https://github.com/relai-ai/agent-eval-loop/tree/main/skills/start
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install relai-ai-agent-eval-loop@llmmart
git clone https://github.com/relai-ai/agent-eval-loop.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole relai-ai/agent-eval-loop collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Start here
RELAI helps teams understand and improve agent behavior. Choose the smallest useful workflow:
- discover: requirements, risks and open questions from code, docs or traces.
- trace-review: build or adapt a UI for traces and source-linked feedback, including findings, questions and eval review when needed.
- eval: build missing evaluations, reuse a benchmark, or measure current behavior; in agent-neutral mode, shared tasks with services adapted to the agreed agents and boundary.
- improve: a direct fix, regression check, or measured behavioral, cost or latency improvement.
- loop: coordinate ongoing learning and experiments under one agreement.
- coding-evals: agent-neutral coding tasks from sessions, pull requests and history, including for a custom coding agent developed here.
Route before setting up
Use the shared purpose guidance when the intended outcome or priorities would change routing or scope. Reuse clear intent; leave workflow-specific clarifications to the selected skill rather than front-loading a questionnaire.
Inspect the request and existing project tests, runners and results. A deterministic fix or a
regression check using project tests goes directly to improve; that handoff completes this skill
without requiring evaluation setup or a board. Maintain durable knowledge when useful as the
knowledge policy describes; a small task may yield
a valuable constraint or lesson. Do not ask the user to choose a step they already requested.
Other workflows reuse working infrastructure and initialize only the RELAI artifacts they need.
For shared terms, knowledge maintenance and user interaction, read the relevant sections of the
project's relai/README.md, otherwise the bundled terms,
knowledge policy and
user interaction guidance. Plugin methods are linked from each skill; relai/
paths refer to project artifacts, not files inside the plugin.
Initialize or recover managed state when needed
- A README beginning
<!-- relaiidentifies RELAI state. If an unrelated folder occupiesrelai/, explain the collision and ask how to resolve it. Do not move or overwrite it; another state root is not currently supported by these workflows. - When initialization is needed and authorized, copy only the parts of template/
needed for the selected workflow into the repository's
relai/, including supporting contracts and checker dependencies. Keep needed local references usable; references to optional stages do not require initializing those stages. Skip caches and renamegitignoreto.gitignorewhen copied. Explain material additions; ask only when existing instructions do not authorize them. Add other template assets when a concrete workflow needs them, rather than copying unused harness or reporting infrastructure in advance. - Record the source of copied files in
relai/UPSTREAM.mdor existing project provenance, following the project-copy upgrade policy. Apply that policy when upgrading existing copies; an installed plugin update does not replace project files. - If setup was interrupted, restore missing template files needed for the selected work. Preserve project modifications. If an existing file is incompatible with the needed operation, explain the difference and resolve that file rather than silently replacing the project copy.
- This is orientation and setup: do not run a model or container here. Name a credential only when a later step needs it, using the project's existing credential mechanism where suitable.
Orient and hand off
When using managed state, read its contract and update relai/STATUS.md using the
board contract. Show the current request, relevant progress, pending
decisions and the useful next action. Missing optional stages are not prerequisites.
Start the already requested skill. If the goal remains unclear, offer the relevant choices with a recommendation. This skill is complete when the request has been routed, or the user understands the project's relevant state and next action; setup and a board are required only on the managed-state path.
Files (agent-eval-loop)
-
template
-
bin
-
benchmark.py 22.1 KB
Loading… -
check_task.py 66.9 KB
Loading… -
check_trajectory.py 7.7 KB
Loading… -
check_wiki.py 6.9 KB
Loading… -
freeze.py 5 KB
Loading… -
summarize.py 44 KB
Loading… -
_common.py 11.3 KB
Loading…
-
-
docs
-
board.md 5.1 KB
Loading… -
files.md 28.6 KB
Loading… -
harbor.md 5.5 KB
Loading… -
native.md 3.5 KB
Loading… -
upgrades.md 1.7 KB
Loading…
-
-
gitignore 24 B · in bundle
-
README.md 35.9 KB
Loading…
-
-
SKILL.md 4.7 KB
--- name: start description: Orient users on RELAI's capabilities, current eval status and next steps, or initialize RELAI records when a workflow needs them. Route clear requests directly to the relevant skill and reuse existing project infrastructure. --- # Start here RELAI helps teams understand and improve agent behavior. Choose the smallest useful workflow: - [discover](../discover/SKILL.md): requirements, risks and open questions from code, docs or traces. - [trace-review](../trace-review/SKILL.md): build or adapt a UI for traces and source-linked feedback, including findings, questions and eval review when needed. - [eval](../eval/SKILL.md): build missing evaluations, reuse a benchmark, or measure current behavior; in agent-neutral mode, shared tasks with services adapted to the agreed agents and boundary. - [improve](../improve/SKILL.md): a direct fix, regression check, or measured behavioral, cost or latency improvement. - [loop](../loop/SKILL.md): coordinate ongoing learning and experiments under one agreement. - [coding-evals](../coding-evals/SKILL.md): agent-neutral coding tasks from sessions, pull requests and history, including for a custom coding agent developed here. ## Route before setting up Use the shared [purpose guidance](template/README.md#establish-the-purpose) when the intended outcome or priorities would change routing or scope. Reuse clear intent; leave workflow-specific clarifications to the selected skill rather than front-loading a questionnaire. Inspect the request and existing project tests, runners and results. A deterministic fix or a regression check using project tests goes directly to `improve`; that handoff completes this skill without requiring evaluation setup or a board. Maintain durable knowledge when useful as the [knowledge policy](template/README.md#maintain-useful-knowledge) describes; a small task may yield a valuable constraint or lesson. Do not ask the user to choose a step they already requested. Other workflows reuse working infrastructure and initialize only the RELAI artifacts they need. For shared terms, knowledge maintenance and user interaction, read the relevant sections of the project's `relai/README.md`, otherwise the bundled [terms](template/README.md#terms), [knowledge policy](template/README.md#maintain-useful-knowledge) and [user interaction guidance](template/README.md#working-with-the-user). Plugin methods are linked from each skill; `relai/` paths refer to project artifacts, not files inside the plugin. ## Initialize or recover managed state when needed - A README beginning `<!-- relai` identifies RELAI state. If an unrelated folder occupies `relai/`, explain the collision and ask how to resolve it. Do not move or overwrite it; another state root is not currently supported by these workflows. - When initialization is needed and authorized, copy only the parts of [template/](template/) needed for the selected workflow into the repository's `relai/`, including supporting contracts and checker dependencies. Keep needed local references usable; references to optional stages do not require initializing those stages. Skip caches and rename `gitignore` to `.gitignore` when copied. Explain material additions; ask only when existing instructions do not authorize them. Add other template assets when a concrete workflow needs them, rather than copying unused harness or reporting infrastructure in advance. - Record the source of copied files in `relai/UPSTREAM.md` or existing project provenance, following the [project-copy upgrade policy](template/docs/upgrades.md). Apply that policy when upgrading existing copies; an installed plugin update does not replace project files. - If setup was interrupted, restore missing template files needed for the selected work. Preserve project modifications. If an existing file is incompatible with the needed operation, explain the difference and resolve that file rather than silently replacing the project copy. - This is orientation and setup: do not run a model or container here. Name a credential only when a later step needs it, using the project's existing credential mechanism where suitable. ## Orient and hand off When using managed state, read its contract and update `relai/STATUS.md` using the [board contract](template/docs/board.md). Show the current request, relevant progress, pending decisions and the useful next action. Missing optional stages are not prerequisites. Start the already requested skill. If the goal remains unclear, offer the relevant choices with a recommendation. This skill is complete when the request has been routed, or the user understands the project's relevant state and next action; setup and a board are required only on the managed-state path.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.