Claude Skill

jevaluate-eval

Use when testing whether the jevaluate rubric and instructions lead graders to the right answers: after a rubric or instruction change, before freezing a rubric, before publishing ratings, or when a grader's calls look wrong.

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download tiffygk-jev-mode-jevaluate-eval-30a0a86.zip · 2106 KB

Install

skills CLI npx skills add https://github.com/tiffygk/jev-mode/tree/main/jevaluate-eval
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tiffygk-jev-mode@llmmart
Git git clone https://github.com/tiffygk/jev-mode.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tiffygk/jev-mode collection as a plugin from our marketplace. Git is the plain clone.

README

Jevaluate Eval

Tests whether graders apply the Jevaluate rubric correctly, before a rubric change reaches any rating. Graders are LLMs applying the rubric like a codebook.

Situation Use
You changed the rubric or a rater instruction Materials check, quiz and judgment eval, in order
You're about to freeze or publish a rubric version The judgment eval on real projects
A second rater rated the same projects Rater agreement

It never edits the rubric or a rating.

Who it's for

Maintainers of a Jevaluate fork who change its rubric.

Sample

Jevaluate Eval · rubric 2026-09-29 · 2026-09-30

Pass: graders route 19 real projects the way the owner does

  • Judgment eval: 19 of 19 cases right in 3 of 3 runs on type, calls Jev, verdict-1 code and stakes.
  • Quiz: the vocabulary-only control scored 73%, under the 80% give-away line; the real instructions got 33 of 33.

Next: re-rate the published projects on the frozen rubric.

Results

Result per rule What it means What sets it
Pass Graders match the owner Routing right in 3 of 3 runs for tuning cases and 2 of 3 for held-out; stakes 2 of 3
Fail Graders misread one rule A case with that rule falls short; one fix is proposed

How it works

  1. Materials check: lint what a grader reads, then show the owner exactly that.
  2. Quiz: a model that never saw the rubric writes most invented projects. The owner answers the key first, and the held-out set is drawn only after the key is final. A vocabulary-only grader runs first and should fail, then the real instructions, three runs each.
  3. Judgment eval: build each real project's evidence by a fixed rule. A blind model and the owner each label new projects. Every answer must be provable from the grader's evidence.
  4. Rater agreement: compare a second rater with the first outside the tuning sample, then score both against a key the owner labels blind (rater-agreement.md).

Why code runs the steps

In earlier rounds:

  • Graders got the whole 43,000-character rubric, and its examples named seven eval cases. They now get only the routing section, and the lint refuses eval-case names.
  • After a type rename, the key and rubric disagreed, and a labeler counted a design-doc snippet as a call. Code now refuses both.

Who judges what

  • Claude Sonnet, headless, and GPT-6 Sol in Codex are the graders under test. Each cites a file and line for answers read off the code, and judges type and stakes.
  • No LLM judges their answers: code scores each against the owner's key.
  • Code holds the eval to its process: it refuses a run on uncommitted rubric changes, a score after the key changed, and contradictory key answers.
  • The Claude session running the eval drafts one fix per miss; the owner approves it.

The owner sets the answer key here, before any model runs.

Where the rules come from

The rubric in jevaluate/rubric.md, with each rule's source in shared/jev-rules.md: TypeSafe's public docs and cookbooks, or a finding from rating projects. Option-order handling, for example, comes from TypeSafe's consistency cookbook.

Install and use

Needs Claude Code with the claude CLI logged in, Python 3.8+ and Jevaluate Publish. The Sol grader also needs Codex.

In Claude Code:

claude plugin marketplace add tiffygk/jev-mode
claude plugin install jevaluate-eval@jev-mode
claude plugin install jevaluate@jev-mode

In Codex:

codex plugin marketplace add tiffygk/jev-mode
codex plugin add jevaluate-eval@jev-mode
codex plugin add jevaluate@jev-mode

Every command runs from a clone; all are in commands.md:

git clone https://github.com/tiffygk/jev-mode
cd jev-mode
python3 jevaluate-eval/lint_materials.py

Then ask Claude Code: test the jevaluate rubric after my change.

To install every jev-mode skill at once, or to copy skills by hand, see the repo README.

Limits

It runs on Claude or Codex and needs no TypeSafe key. Jevaluate rates only projects that call Jev. An owner who misreads a rule tunes graders to that misreading. It doesn't test scores or the final verdict. A three-run judgment eval costs about 380k tokens. Not affiliated with TypeSafe.

License

PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: open an issue.

Skill manifest

Jevaluate Eval

Tests whether graders apply the jevaluate rubric correctly, and proposes wording fixes. It never edits rubric.md, read.md or the library. If $JEVALUATE_OVERLAY is set, read that file first; its rules override the defaults here.

The scripts are in jevaluate-eval/ (the quiz in jevaluate-eval/quiz/); every command is in commands.md. The runners refuse while jevaluate/ or jevaluate-harness/ has uncommitted changes, or main is ahead on them: commit or merge first.

Four instruments, in order

Before each, tell the owner in one line which runs next and what it tests.

  1. Materials check: does what a grader reads say what the rules mean?

    • lint_materials.py must print "lint clean": allowed values, eval-case names, em-dashes, named files, a jevaluate/enforcement.md row for every rule, and no sentence over 35 words added since the newest freeze.
    • A fresh reviewer, briefed as a grader, reads them for contradictions.
    • build_grader_view.py <out.html> builds the step-by-step view: the rater's start files, what step.py serves at each phase, every stakes row, and the eval grader's prompt. The owner reviews it. Schedule that review after any planned task that moves instructions into code, never before.
  2. Quiz: does a grader understand the rules on invented projects?

    • Ask the owner how many scenarios they'll answer. Default 12: 8 tuning, 4 held out.
    • A writer that never sees the rules writes them (a headless call with no tools). Rewrite any that share 8 words with a rubric example or name an eval case.
    • The owner answers first, on build_answer_form.py --count N: type, calls Jev, verdict-1 code and stakes as read, with the type definitions beside each row. Save copies to the clipboard. build_answer_form.py --key <answers.json> derives kind and every n.a., and refuses contradictory answers: take each refusal line to the owner.
    • Show each disagreement with your reading, one scenario at a time, with the rule behind it. The owner rules; that is the key.
    • Draw the held-out set at random (record the seed), only after the key and any wording it triggered are final. In the results, name held-out scenarios whose discussion fed wording.
    • Red control first: run_quiz.py --system vocab gives only the vocabulary and should miss most answers; at 80% or more, the scenarios give their answers away. Report which rule tags the control passes: those scenarios check that the rules don't confuse a grader, not that it learned them. Then run_quiz.py --system routing --reps 3 and score_quiz.py.
  3. Judgment eval: does a grader route real projects correctly? run_eval.py --phase baseline|after --out <runs>/<run> (runs folder: $JEVALUATE_RUNS when set, else .work/), then score.py <run>. Every gold answer must be provable from its packet; held-out labels come from blind labelers, and the owner answers before seeing them.

    • New cases: packets follow pick_files.py. Each blind label goes through check_labels.py run, which applies the same call-line rule as library.py check and reruns a refused label once, keeping both. The owner answers on the same form as the quiz, built with build_answer_form.py --cards: the same fields, the type definitions beside each row, answers kept in the browser and saved to the clipboard. Each row is an evidence card from build_cards.py (call lines, README claims, acting lines, one plain sentence each), never a verdict; the card script refuses type, code and stakes words. Acting lines follow the answer to whoever it reaches: quote the line that acts and a line that shows who sees or receives the result (a README usage example counts), so the owner can read stakes and type without the code.
  4. Rater agreement: do two raters agree, and is either right? For a second rater, such as another model, on real projects. Follow rater-agreement.md: name the tuning sample first, lead with held-out agreement and kappa (rater_agreement/rater_agreement.py agreement), then score both raters against an answer key the owner labels blind (facts, draw, score). Agreement claims consistency only.

Rules

  • Fix wording only against tuning misses, never naming a held-out scenario or an eval case. Each fix goes to the owner as a before-and-after before the rerun.
  • Report per rule tag, propose one wording fix per missed rule, and stop.
Files (jev-mode)
  • cases
    • email-tagger.md 912 B
      Loading…
    • refund-bot.md 882 B
      Loading…
  • images
    • answer-form-dark.png 247 KB · in bundle
    • answer-form-light.png 241.3 KB · in bundle
    • eval-pipeline-dark.png 908.2 KB · in bundle
    • eval-pipeline-light.png 785.1 KB · in bundle
  • new-cases
    • cards.json 17.3 KB
      Loading…
    • case-answers.html 50 KB · in bundle
    • labeler-prompt.md 2.4 KB
      Loading…
    • notes.json 8.1 KB
      Loading…
  • quiz
    • build_answer_form.py 13.7 KB
      Loading…
    • expected.json 1.7 KB
      Loading…
    • quiz-answers.html 96.6 KB · in bundle
    • run_quiz.py 2.6 KB
      Loading…
    • scenarios-raw.json 9.8 KB
      Loading…
    • scenarios.json 13.6 KB
      Loading…
    • score_quiz.py 1.7 KB
      Loading…
    • test_answer_form.py 10.3 KB
      Loading…
    • test_score_quiz.py 929 B
      Loading…
    • writer-prompt.md 1.1 KB
      Loading…
  • rater_agreement
    • rater_agreement.py 9 KB
      Loading…
  • results
    • 2026-09-28-guide-run1.md 1.2 KB
      Loading…
    • 2026-09-30-baseline-run1-failed.md 1.8 KB
      Loading…
    • 2026-09-30-baseline.md 1.9 KB
      Loading…
    • 2026-09-30-quiz.md 3.5 KB
      Loading…
    • 2026-10-02-codex-gpt-6-sol.md 4.4 KB
      Loading…
    • 2026-10-05-rubric-2026-09-29.1-targeted.md 2.4 KB
      Loading…
    • 2026-10-06-rubric-2026-09-29.2-targeted.md 3.2 KB
      Loading…
  • build_cards.py 5.6 KB
    Loading…
  • build_grader_view.py 9.7 KB
    Loading…
  • build_packets.py 4.3 KB
    Loading…
  • check_labels.py 5.4 KB
    Loading…
  • codex-notes.md 0 B
    Loading…
  • commands.md 1.8 KB
    Loading…
  • compare_runs.py 5.5 KB
    Loading…
  • conftest.py 653 B
    Loading…
  • gold.json 3.3 KB
    Loading…
  • guide_gold.json 734 B
    Loading…
  • guide_task.md 322 B
    Loading…
  • guide_test.py 4.7 KB
    Loading…
  • lint_materials.py 7.7 KB
    Loading…
  • pick_files.py 2.2 KB
    Loading…
  • rater-agreement.md 4.5 KB
    Loading…
  • README.md 4.8 KB
    # Jevaluate Eval
    
    Tests whether graders apply the Jevaluate rubric correctly, before a rubric change reaches any rating. Graders are LLMs applying the rubric like a codebook.
    
    | Situation | Use |
    |---|---|
    | You changed the rubric or a rater instruction | Materials check, quiz and judgment eval, in order |
    | You're about to freeze or publish a rubric version | The judgment eval on real projects |
    | A second rater rated the same projects | Rater agreement |
    
    It never edits the rubric or a rating.
    
    ## Who it's for
    
    Maintainers of a Jevaluate fork who change its rubric.
    
    ## Sample
    
    > **Jevaluate Eval · rubric 2026-09-29 · 2026-09-30**
    >
    > ### Pass: graders route 19 real projects the way the owner does
    > - Judgment eval: 19 of 19 cases right in 3 of 3 runs on type, calls Jev, verdict-1 code and stakes.
    > - Quiz: the vocabulary-only control scored 73%, under the 80% give-away line; the real instructions got 33 of 33.
    >
    > **Next: re-rate the published projects on the frozen rubric.**
    
    ## Results
    
    | Result per rule | What it means | What sets it |
    |---|---|---|
    | **Pass** | Graders match the owner | Routing right in 3 of 3 runs for tuning cases and 2 of 3 for held-out; stakes 2 of 3 |
    | **Fail** | Graders misread one rule | A case with that rule falls short; one fix is proposed |
    
    ## How it works
    
    1. Materials check: lint what a grader reads, then show the owner exactly that.
    2. Quiz: a model that never saw the rubric writes most invented projects. The owner answers the key first, and the held-out set is drawn only after the key is final. A vocabulary-only grader runs first and should fail, then the real instructions, three runs each.
    3. Judgment eval: build each real project's evidence by a fixed rule. A blind model and the owner each label new projects. Every answer must be provable from the grader's evidence.
    4. Rater agreement: compare a second rater with the first outside the tuning sample, then score both against a key the owner labels blind ([rater-agreement.md](rater-agreement.md)).
    
    <a href="https://tiffygk.github.io/jev-mode/system/#d3-h"><picture><source media="(prefers-color-scheme: dark)" srcset="images/eval-pipeline-dark.png"><img alt="Eval pipeline" src="images/eval-pipeline-light.png"></picture></a>
    
    ## Why code runs the steps
    
    In earlier rounds:
    
    - Graders got the whole 43,000-character rubric, and its examples named seven eval cases. They now get only the routing section, and the lint refuses eval-case names.
    - After a type rename, the key and rubric disagreed, and a labeler counted a design-doc snippet as a call. Code now refuses both.
    
    ## Who judges what
    
    - Claude Sonnet, headless, and GPT-6 Sol in Codex are the graders under test. Each cites a file and line for answers read off the code, and judges type and stakes.
    - No LLM judges their answers: code scores each against the owner's key.
    - Code holds the eval to its process: it refuses a run on uncommitted rubric changes, a score after the key changed, and contradictory key answers.
    - The Claude session running the eval drafts one fix per miss; the owner approves it.
    
    <picture><source media="(prefers-color-scheme: dark)" srcset="images/answer-form-dark.png"><img alt="The owner's answer form" src="images/answer-form-light.png"></picture>
    
    *The owner sets the answer key here, before any model runs.*
    
    ## Where the rules come from
    
    The rubric in `jevaluate/rubric.md`, with each rule's source in `shared/jev-rules.md`: TypeSafe's public docs and cookbooks, or a finding from rating projects. Option-order handling, for example, comes from TypeSafe's consistency cookbook.
    
    ## Install and use
    
    Needs Claude Code with the `claude` CLI logged in, Python 3.8+ and [Jevaluate Publish](../jevaluate/). The Sol grader also needs Codex.
    
    In Claude Code:
    
    ```
    claude plugin marketplace add tiffygk/jev-mode
    claude plugin install jevaluate-eval@jev-mode
    claude plugin install jevaluate@jev-mode
    ```
    
    In Codex:
    
    ```
    codex plugin marketplace add tiffygk/jev-mode
    codex plugin add jevaluate-eval@jev-mode
    codex plugin add jevaluate@jev-mode
    ```
    
    Every command runs from a clone; all are in `commands.md`:
    
    ```
    git clone https://github.com/tiffygk/jev-mode
    cd jev-mode
    python3 jevaluate-eval/lint_materials.py
    ```
    
    Then ask Claude Code: `test the jevaluate rubric after my change`.
    
    To install every jev-mode skill at once, or to copy skills by hand, see the [repo README](../README.md#install).
    
    ## Limits
    
    It runs on Claude or Codex and needs no TypeSafe key. Jevaluate rates only projects that call Jev. An owner who misreads a rule tunes graders to that misreading. It doesn't test scores or the final verdict. A three-run judgment eval costs about 380k tokens. Not affiliated with TypeSafe.
    
    ## License
    
    PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: [open an issue](https://github.com/tiffygk/jev-mode/issues).
    
  • runners.py 6.2 KB
    Loading…
  • run_eval.py 6.7 KB
    Loading…
  • score.py 11.4 KB
    Loading…
  • SKILL.md 4.6 KB
    ---
    name: jevaluate-eval
    description: "Use when testing whether the jevaluate rubric and instructions lead graders to the right answers: after a rubric or instruction change, before freezing a rubric, before publishing ratings, or when a grader's calls look wrong."
    ---
    
    # Jevaluate Eval
    
    Tests whether graders apply the jevaluate rubric correctly, and proposes wording fixes. It never edits `rubric.md`, `read.md` or the library. If `$JEVALUATE_OVERLAY` is set, read that file first; its rules override the defaults here.
    
    The scripts are in `jevaluate-eval/` (the quiz in `jevaluate-eval/quiz/`); every command is in `commands.md`. The runners refuse while `jevaluate/` or `jevaluate-harness/` has uncommitted changes, or `main` is ahead on them: commit or merge first.
    
    ## Four instruments, in order
    Before each, tell the owner in one line which runs next and what it tests.
    
    1. **Materials check: does what a grader reads say what the rules mean?**
       - `lint_materials.py` must print "lint clean": allowed values, eval-case names, em-dashes, named files, a `jevaluate/enforcement.md` row for every rule, and no sentence over 35 words added since the newest freeze.
       - A fresh reviewer, briefed as a grader, reads them for contradictions.
       - `build_grader_view.py <out.html>` builds the step-by-step view: the rater's start files, what `step.py` serves at each phase, every stakes row, and the eval grader's prompt. The owner reviews it. Schedule that review after any planned task that moves instructions into code, never before.
    2. **Quiz: does a grader understand the rules on invented projects?**
       - Ask the owner how many scenarios they'll answer. Default 12: 8 tuning, 4 held out.
       - A writer that never sees the rules writes them (a headless call with no tools). Rewrite any that share 8 words with a rubric example or name an eval case.
       - The owner answers first, on `build_answer_form.py --count N`: type, calls Jev, verdict-1 code and stakes as read, with the type definitions beside each row. Save copies to the clipboard. `build_answer_form.py --key <answers.json>` derives kind and every n.a., and refuses contradictory answers: take each refusal line to the owner.
       - Show each disagreement with your reading, one scenario at a time, with the rule behind it. The owner rules; that is the key.
       - Draw the held-out set at random (record the seed), only after the key and any wording it triggered are final. In the results, name held-out scenarios whose discussion fed wording.
       - Red control first: `run_quiz.py --system vocab` gives only the vocabulary and should miss most answers; at 80% or more, the scenarios give their answers away. Report which rule tags the control passes: those scenarios check that the rules don't confuse a grader, not that it learned them. Then `run_quiz.py --system routing --reps 3` and `score_quiz.py`.
    3. **Judgment eval: does a grader route real projects correctly?** `run_eval.py --phase baseline|after --out <runs>/<run>` (runs folder: `$JEVALUATE_RUNS` when set, else `.work/`), then `score.py <run>`. Every gold answer must be provable from its packet; held-out labels come from blind labelers, and the owner answers before seeing them.
       - New cases: packets follow `pick_files.py`. Each blind label goes through `check_labels.py run`, which applies the same call-line rule as `library.py check` and reruns a refused label once, keeping both. The owner answers on the same form as the quiz, built with `build_answer_form.py --cards`: the same fields, the type definitions beside each row, answers kept in the browser and saved to the clipboard. Each row is an evidence card from `build_cards.py` (call lines, README claims, acting lines, one plain sentence each), never a verdict; the card script refuses type, code and stakes words. Acting lines follow the answer to whoever it reaches: quote the line that acts and a line that shows who sees or receives the result (a README usage example counts), so the owner can read stakes and type without the code.
    
    4. **Rater agreement: do two raters agree, and is either right?** For a second rater, such as another model, on real projects. Follow `rater-agreement.md`: name the tuning sample first, lead with held-out agreement and kappa (`rater_agreement/rater_agreement.py agreement`), then score both raters against an answer key the owner labels blind (`facts`, `draw`, `score`). Agreement claims consistency only.
    
    ## Rules
    - Fix wording only against tuning misses, never naming a held-out scenario or an eval case. Each fix goes to the owner as a before-and-after before the rerun.
    - Report per rule tag, propose one wording fix per missed rule, and stop.
    
  • sources.json 5.9 KB
    Loading…
  • task.md 1.7 KB
    Loading…
  • test_build_cards.py 4.9 KB
    Loading…
  • test_check_labels.py 4.3 KB
    Loading…
  • test_compare_runs.py 2.1 KB
    Loading…
  • test_evals.py 25.4 KB
    Loading…
  • test_grader_view.py 2.3 KB
    Loading…
  • test_guide.py 1.6 KB
    Loading…
  • test_lint.py 5 KB
    Loading…
  • test_pick_files.py 3.2 KB
    Loading…
  • test_rater_agreement.py 2.5 KB
    Loading…
  • test_runners.py 5.7 KB
    Loading…

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.