Claude Skill

jevaluate-harness

Use when running a jevaluate rating round, re-rating after a rubric change, changing the jevaluate rubric, or publishing ratings to the jev-mode ratings folder.

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download tiffygk-jev-mode-jevaluate-harness-30a0a86.zip · 719 KB

Install

skills CLI npx skills add https://github.com/tiffygk/jev-mode/tree/main/jevaluate-harness
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tiffygk-jev-mode@llmmart
Git git clone https://github.com/tiffygk/jev-mode.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tiffygk/jev-mode collection as a plugin from our marketplace. Git is the plain clone.

README

Jevaluate Harness

Runs Jevaluate over many projects at once, guides a rubric change, and publishes the ratings. Testing a rubric change is Jevaluate Eval.

Jevaluate rates one project on its own. This skill is the harness around it: it never judges a project or edits a rating. Without it, parallel raters collide on the shared library, a rubric change leaves every older rating stale, and nothing shows whether the change made ratings better or worse.

Who it's for

Jevaluate users who:

  • Rate a whole list of projects in one round.
  • Fork Jevaluate and change its rubric.
  • Publish a ratings folder.

It doesn't work without Jevaluate: every step runs Jevaluate's scripts, rubric or eval.

For a different LLM rating skill, this one won't run, but its pattern carries over: fixed test cases, one logger, and reviewers that see only the facts.

How it works

The skill is a short guide that opens one of three files, each a numbered list with the check that must pass:

  1. rubric-change.md changes the rubric: a meaning and an example for every value, one row per stakes level where the rule changes with stakes, and a code check for every rule. Then Jevaluate Eval tests it.
  2. rating-round.md runs a round: screen a list against two known controls, size each repo, brief raters, scan each rater's transcript, log ratings one at a time and send big verdict moves to a reviewer that sees only the facts. A split between two raters that changes the verdict, and that no written ruling settles, goes to the owner as the rule, the code and two unlabeled answers.
  3. publish.md exports a preview, has a reviewer check every page for private context, runs the release check (tests, a fresh clone, a passing eval on this rubric), and opens one pull request.

Written steps alone slipped in earlier rounds, so scripts check each one:

  • Parallel raters wrote to the shared library at the same time. Now the controller alone logs each rating, one at a time, through library.py add.
  • Sizing fetched one large repo file by file for over 15 minutes. Sizing now uses the repo's file list, and large repos are read from a shallow clone.
  • A mode the skill didn't have shipped in its README. The lint now refuses any file a skill names that doesn't exist.

harness_status.py reads your round's ledger, reruns the materials lint and the release check, and names the file to open next. A skipped step shows up there before the next one starts.

The system page also shows the routing flowchart, the eval and who is blind to what.

Install and use

Needs Claude Code or Codex, Python 3.8+, the GitHub CLI (gh, logged in) and Jevaluate Publish.

In Claude Code:

claude plugin marketplace add tiffygk/jev-mode
claude plugin install jevaluate-harness@jev-mode
claude plugin install jevaluate@jev-mode

In Codex:

codex plugin marketplace add tiffygk/jev-mode
codex plugin add jevaluate-harness@jev-mode
codex plugin add jevaluate@jev-mode

Put your never-publish regexes in $JEVALUATE_LIBRARY/private-terms.txt, and optionally set $JEVALUATE_OVERLAY to a markdown note of your own rules for these workflows (see the skill's Settings). Then ask Claude Code: run a jevaluate re-rate round.

To install every jev-mode skill at once, or to copy skills by hand, see the repo README.

Limits

Like Jevaluate, it never calls Jev, so it needs no TypeSafe API key. It rates nothing itself: each rating is a full Jevaluate run, about 100-300k tokens by repo size. Not affiliated with TypeSafe.

License

PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: open an issue.

Skill manifest

Jevaluate Harness

The maintainer's process for jevaluate-publish. One rating is jevaluate/read.md; testing the rubric is the jevaluate-eval skill. If $JEVALUATE_OVERLAY is set, read that file first; its rules override the defaults here.

Settings (yours, not in this skill)

  • $JEVALUATE_LIBRARY: the private ratings library (default ~/.claude/jevaluate-library if it exists, else ~/.jevaluate-library).
  • $PRIVATE_TERMS: your never-publish regexes, one per line. library.py export reads only $JEVALUATE_LIBRARY/private-terms.txt, so keep the file there (or symlink it) and have your pre-push hook read the same file.
  • $JEVALUATE_OVERLAY: a markdown note of your own rules for these workflows, read first; no required format.

Two principles

  • Code checks, text explains: every rule has a row in jevaluate/enforcement.md naming the check that refuses a wrong answer, or why it stays judgment.
  • Ask for what is observed: raters record type, the calls-Jev line, each decision's stakes as read and the verdict; library.py add derives kind and every n.a.

Workflows

Open the file for the workflow and follow it in order; each step names the gate that must pass.

Workflow File Done when
Change the rubric rubric-change.md jevaluate-eval passes and the owner approves the version
Rate or re-rate projects rating-round.md (rater prompt: rater-brief.md) every rating logged by library.py add, scan clean
Publish ratings publish.md one pull request, reviewed

To see where you are: python3 jevaluate-harness/scripts/harness_status.py <ledger> --workflow rubric|round (round runs through publishing). It reruns the scripted gates it has (the materials check), reads the other steps from the ledger, and names the file to open next. Ledger lines: done: <step> <YYYY-MM-DD> <evidence>. Steps: rubric = rules-written, materials-checked, quiz-passed, eval-passed, owner-approved, frozen; round = picked, sized, costs-approved, rated, scanned, adjudicated, final-gate, release-check, published.

Keep a ledger per workflow: each step, each ruling with its cost if wrong, and tokens per agent. Record rulings and handoffs where your overlay says, or in the ledger.

Enforced by

Rule Enforced by
A rater never reads the rubric, past ratings or a full-file copy directly scan_transcripts.py; test_scan.py
A freeze tag goes only on a merged commit whose rubric title carries its version freeze_tag_check.py
A workflow's scripted gates are rerun, not read from the ledger scripts/harness_status.py; test_status.py
Every release check passes before publishing scripts/check_release.py; test_release.py
Each step followed in order, rulings recorded with their cost if wrong LLM's judgment following instructions
Files (jev-mode)
  • fixtures
    • bad-rating-evidence
      • files
        • src__x.py 103 B
          Loading…
    • good-rating-evidence
      • files
        • src__x.py 103 B
          Loading…
    • bad-rating.md 1.5 KB
      Loading…
    • bad-rating.md.steps.json 517 B
      Loading…
    • good-rating.md 1.5 KB
      Loading…
    • good-rating.md.steps.json 517 B
      Loading…
  • images
    • skills-dark.png 407.6 KB · in bundle
    • skills-light.png 364.7 KB · in bundle
  • scripts
    • check_release.py 3.3 KB
      Loading…
    • dispute_packet.py 3.6 KB
      Loading…
    • harness_status.py 2.9 KB
      Loading…
  • freeze_tag_check.py 2.6 KB
    Loading…
  • publish.md 900 B
    Loading…
  • rater-brief.md 2.1 KB
    Loading…
  • rating-round.md 4.3 KB
    Loading…
  • README.md 4 KB
    # Jevaluate Harness
    
    Runs [Jevaluate](../jevaluate/) over many projects at once, guides a rubric change, and publishes the ratings. Testing a rubric change is [Jevaluate Eval](../jevaluate-eval/).
    
    Jevaluate rates one project on its own. This skill is the harness around it: it never judges a project or edits a rating. Without it, parallel raters collide on the shared library, a rubric change leaves every older rating stale, and nothing shows whether the change made ratings better or worse.
    
    ## Who it's for
    
    Jevaluate users who:
    
    - Rate a whole list of projects in one round.
    - Fork Jevaluate and change its rubric.
    - Publish a ratings folder.
    
    It doesn't work without Jevaluate: every step runs Jevaluate's scripts, rubric or eval.
    
    For a different LLM rating skill, this one won't run, but its pattern carries over: fixed test cases, one logger, and reviewers that see only the facts.
    
    ## How it works
    
    The skill is a short guide that opens one of three files, each a numbered list with the check that must pass:
    
    1. `rubric-change.md` changes the rubric: a meaning and an example for every value, one row per stakes level where the rule changes with stakes, and a code check for every rule. Then Jevaluate Eval tests it.
    2. `rating-round.md` runs a round: screen a list against two known controls, size each repo, brief raters, scan each rater's transcript, log ratings one at a time and send big verdict moves to a reviewer that sees only the facts. A split between two raters that changes the verdict, and that no written ruling settles, goes to the owner as the rule, the code and two unlabeled answers.
    3. `publish.md` exports a preview, has a reviewer check every page for private context, runs the release check (tests, a fresh clone, a passing eval on this rubric), and opens one pull request.
    
    Written steps alone slipped in earlier rounds, so scripts check each one:
    
    - Parallel raters wrote to the shared library at the same time. Now the controller alone logs each rating, one at a time, through `library.py add`.
    - Sizing fetched one large repo file by file for over 15 minutes. Sizing now uses the repo's file list, and large repos are read from a shallow clone.
    - A mode the skill didn't have shipped in its README. The lint now refuses any file a skill names that doesn't exist.
    
    `harness_status.py` reads your round's ledger, reruns the materials lint and the release check, and names the file to open next. A skipped step shows up there before the next one starts.
    
    <a href="https://tiffygk.github.io/jev-mode/system/#d1-h"><picture><source media="(prefers-color-scheme: dark)" srcset="images/skills-dark.png"><img alt="How the skills fit" src="images/skills-light.png"></picture></a>
    
    The [system page](https://tiffygk.github.io/jev-mode/system/) also shows the routing flowchart, the eval and who is blind to what.
    
    ## Install and use
    
    Needs Claude Code or Codex, Python 3.8+, the GitHub CLI (`gh`, logged in) and [Jevaluate Publish](../jevaluate/).
    
    In Claude Code:
    
    ```
    claude plugin marketplace add tiffygk/jev-mode
    claude plugin install jevaluate-harness@jev-mode
    claude plugin install jevaluate@jev-mode
    ```
    
    In Codex:
    
    ```
    codex plugin marketplace add tiffygk/jev-mode
    codex plugin add jevaluate-harness@jev-mode
    codex plugin add jevaluate@jev-mode
    ```
    
    Put your never-publish regexes in `$JEVALUATE_LIBRARY/private-terms.txt`, and optionally set `$JEVALUATE_OVERLAY` to a markdown note of your own rules for these workflows (see the skill's Settings). Then ask Claude Code: `run a jevaluate re-rate round`.
    
    To install every jev-mode skill at once, or to copy skills by hand, see the [repo README](../README.md#install).
    
    ## Limits
    
    Like Jevaluate, it never calls Jev, so it needs no TypeSafe API key. It rates nothing itself: each rating is a full Jevaluate run, about 100-300k tokens by repo size. Not affiliated with TypeSafe.
    
    ## License
    
    PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: [open an issue](https://github.com/tiffygk/jev-mode/issues).
    
  • rubric-change.md 3.3 KB
    Loading…
  • scan_transcripts.py 2.8 KB
    Loading…
  • SKILL.md 3 KB
    ---
    name: jevaluate-harness
    description: Use when running a jevaluate rating round, re-rating after a rubric change, changing the jevaluate rubric, or publishing ratings to the jev-mode ratings folder.
    ---
    
    # Jevaluate Harness
    
    The maintainer's process for `jevaluate-publish`. One rating is `jevaluate/read.md`; testing the rubric is the `jevaluate-eval` skill. If `$JEVALUATE_OVERLAY` is set, read that file first; its rules override the defaults here.
    
    ## Settings (yours, not in this skill)
    - `$JEVALUATE_LIBRARY`: the private ratings library (default `~/.claude/jevaluate-library` if it exists, else `~/.jevaluate-library`).
    - `$PRIVATE_TERMS`: your never-publish regexes, one per line. `library.py export` reads only `$JEVALUATE_LIBRARY/private-terms.txt`, so keep the file there (or symlink it) and have your pre-push hook read the same file.
    - `$JEVALUATE_OVERLAY`: a markdown note of your own rules for these workflows, read first; no required format.
    
    ## Two principles
    - **Code checks, text explains:** every rule has a row in `jevaluate/enforcement.md` naming the check that refuses a wrong answer, or why it stays judgment.
    - **Ask for what is observed:** raters record type, the calls-Jev line, each decision's stakes as read and the verdict; `library.py add` derives kind and every n.a.
    
    ## Workflows
    Open the file for the workflow and follow it in order; each step names the gate that must pass.
    
    | Workflow | File | Done when |
    |---|---|---|
    | Change the rubric | `rubric-change.md` | `jevaluate-eval` passes and the owner approves the version |
    | Rate or re-rate projects | `rating-round.md` (rater prompt: `rater-brief.md`) | every rating logged by `library.py add`, scan clean |
    | Publish ratings | `publish.md` | one pull request, reviewed |
    
    To see where you are: `python3 jevaluate-harness/scripts/harness_status.py <ledger> --workflow rubric|round` (`round` runs through publishing). It reruns the scripted gates it has (the materials check), reads the other steps from the ledger, and names the file to open next. Ledger lines: `done: <step> <YYYY-MM-DD> <evidence>`. Steps: rubric = rules-written, materials-checked, quiz-passed, eval-passed, owner-approved, frozen; round = picked, sized, costs-approved, rated, scanned, adjudicated, final-gate, release-check, published.
    
    Keep a ledger per workflow: each step, each ruling with its cost if wrong, and tokens per agent. Record rulings and handoffs where your overlay says, or in the ledger.
    
    ## Enforced by
    
    | Rule | Enforced by |
    |---|---|
    | A rater never reads the rubric, past ratings or a full-file copy directly | `scan_transcripts.py`; `test_scan.py` |
    | A freeze tag goes only on a merged commit whose rubric title carries its version | `freeze_tag_check.py` |
    | A workflow's scripted gates are rerun, not read from the ledger | `scripts/harness_status.py`; `test_status.py` |
    | Every release check passes before publishing | `scripts/check_release.py`; `test_release.py` |
    | Each step followed in order, rulings recorded with their cost if wrong | LLM's judgment following instructions |
    
  • test_dispute_packet.py 2.3 KB
    Loading…
  • test_release.py 2.1 KB
    Loading…
  • test_scan.py 4.9 KB
    Loading…
  • test_status.py 2.8 KB
    Loading…

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.