GitHub collection

uipath/coder_eval

Imported from GitHub — is this yours?

6 skills imported from this repository.

View on GitHub
141 3 Apache-2.0
Claude Skill coder-eval

analyze

Analyze a finished coder-eval run and write analysis.md — cluster failures into systemic patterns, diagnose prompts, criteria, config, environment and cost, and recommend concrete fixes. Use when the user wants to know why a run failed, what to fix, or what a run says about their

LLM Mart

Claude Skill coder-eval

check-skill

Generate and run a coder-eval activation suite for a Claude Code skill — does the agent actually engage it when it should, and leave it alone when it shouldn't? Use when the user asks whether a skill triggers, wants to test skill activation, or worries a skill has silently stoppe

LLM Mart

Claude Skill coder-eval

ci

Generate a GitHub Actions workflow that runs a coder-eval suite as a CI gate or on a schedule, using the published composite action — with the agent runtime, credentials, JUnit output and a score floor wired correctly.

LLM Mart

Claude Skill coder-eval

init

Set up coder-eval in this repository — scan for what is worth evaluating (Claude Code skills, an MCP server, a CLI), then scaffold a task directory with one real, passing-or-failing task and the exact command to run it.

LLM Mart

Claude Skill coder-eval

lint-tasks

Review coder-eval task YAML that already exists — find criteria that cannot fail, prompts that give away the answer, fixtures with no cleanup, and near-duplicate tasks, each with a severity and a concrete fix. Read-only. Use when the user wants existing tasks reviewed, linted, au

LLM Mart

Claude Skill coder-eval

task

Turn a natural-language description into one or more coder-eval task YAML files — minimal prompts, weighted success criteria that check output content, validated with `coder-eval plan`. Use when the user wants to write, add, or generate an evaluation task.

LLM Mart

Agents (4)

Claude Agent

ANTIGRAVITY

Run Google Antigravity (Gemini) as the agent under evaluation in Coder Eval — installation, authentication, model and skill configuration, and how its telemetry maps to sandboxed, weighted scoring.

Claude Agent

CLAUDE_CODE

Configure and run the default Claude Code agent in Coder Eval — the full agent-config surface, direct vs. Bedrock authentication, permission modes, sandbox isolation, skills/plugins, early stop, and token telemetry.

Claude Agent

CODEX

Run OpenAI Codex as the agent under evaluation in Coder Eval — installation, authentication, task configuration, and how Codex telemetry maps to sandboxed, weighted scoring.

Claude Agent

HARNESS_PARITY

Imported from uipath/coder_eval/docs/agents/HARNESS_PARITY.md.

Commands (6)

  • /coder-eval-code-review — Run a multi-model code review on uncommitted changes or a described set of files source
  • /coder-eval-code-review-full — Review the codebase across critical quality axes source
  • /coder-eval-code-review-wf — Workflow-based 8-axis codebase review — per-axis sub-workflows, adversarial verify, deterministic scoring + rendering source
  • /coder-eval-create-plan — Create a structured, phased implementation plan for a feature or change in the coder_eval codebase, executable from a fresh session by /coder-eval-implement-plan source
  • /coder-eval-implement-plan — Implement an approved coder_eval plan phase by phase with risk-scaled per-phase review, then a final code review source
  • /coder-eval-review — Generate per-task review.json (summary + tags) for a completed run source