GitHub collection

uipath/coder_eval

Imported from GitHub — is this yours?

6 skills imported from this repository.

View on GitHub
Stars
160
Forks
9
License
Apache-2.0

Skills (6)

analyze

Analyze a finished coder-eval run and write analysis.md — cluster failures into systemic patterns, diagnose prompts, criteria, config, environment and cost, and recommend concrete fixes. Use when the user wants to know why a run failed, what to fix, or what a run says about their

Claude Skill coder-eval

check-skill

Generate and run a coder-eval activation suite for a Claude Code skill — does the agent actually engage it when it should, and leave it alone when it shouldn't? Use when the user asks whether a skill triggers, wants to test skill activation, or worries a skill has silently stoppe

Claude Skill coder-eval

ci

Generate a GitHub Actions workflow that runs a coder-eval suite as a CI gate or on a schedule, using the published composite action — with the agent runtime, credentials, JUnit output and a score floor wired correctly.

Claude Skill coder-eval

init

Set up coder-eval in this repository — scan for what is worth evaluating (Claude Code skills, an MCP server, a CLI), then scaffold a task directory with one real, passing-or-failing task and the exact command to run it.

Claude Skill coder-eval

lint-tasks

Review coder-eval task YAML that already exists — find criteria that cannot fail, prompts that give away the answer, fixtures with no cleanup, and near-duplicate tasks, each with a severity and a concrete fix. Read-only. Use when the user wants existing tasks reviewed, linted, au

Claude Skill coder-eval

task

Turn a natural-language description into one or more coder-eval task YAML files — minimal prompts, weighted success criteria that check output content, validated with `coder-eval plan`. Use when the user wants to write, add, or generate an evaluation task.

Claude Skill coder-eval

Agents (4)

ANTIGRAVITY

Run Google Antigravity (Gemini) as the agent under evaluation in Coder Eval — installation, authentication, model and skill configuration, and how its telemetry maps to sandboxed, weighted scoring.

Claude Agent

CLAUDE_CODE

Configure and run the default Claude Code agent in Coder Eval — the full agent-config surface, direct vs. Bedrock authentication, permission modes, sandbox isolation, skills/plugins, early stop, and token telemetry.

Claude Agent

CODEX

Run OpenAI Codex as the agent under evaluation in Coder Eval — installation, authentication, task configuration, and how Codex telemetry maps to sandboxed, weighted scoring.

Claude Agent

HARNESS_PARITY

Imported from uipath/coder_eval/docs/agents/HARNESS_PARITY.md.

Claude Agent

Commands (6)

  • /coder-eval-code-review — Run a multi-model code review on uncommitted changes or a described set of files
    source
  • /coder-eval-code-review-full — Review the codebase across critical quality axes
    source
  • /coder-eval-code-review-wf — Workflow-based 8-axis codebase review — per-axis sub-workflows, adversarial verify, deterministic scoring + rendering
    source
  • /coder-eval-create-plan — Create a structured, phased implementation plan for a feature or change in the coder_eval codebase, executable from a fresh session by /coder-eval-implement-plan
    source
  • /coder-eval-implement-plan — Implement an approved coder_eval plan phase by phase with risk-scaled per-phase review, then a final code review
    source
  • /coder-eval-review — Generate per-task review.json (summary + tags) for a completed run
    source