jevaluate-harness
Use when running a jevaluate rating round, re-rating after a rubric change, changing the jevaluate rubric, or publishing ratings to the jev-mode ratings folder.
Install
npx skills add https://github.com/tiffygk/jev-mode/tree/main/jevaluate-harness
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tiffygk-jev-mode@llmmart
git clone https://github.com/tiffygk/jev-mode.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tiffygk/jev-mode collection as a plugin from our marketplace. Git is the plain clone.
README
Jevaluate Harness
Runs Jevaluate over many projects at once, guides a rubric change, and publishes the ratings. Testing a rubric change is Jevaluate Eval.
Jevaluate rates one project on its own. This skill is the harness around it: it never judges a project or edits a rating. Without it, parallel raters collide on the shared library, a rubric change leaves every older rating stale, and nothing shows whether the change made ratings better or worse.
Who it's for
Jevaluate users who:
- Rate a whole list of projects in one round.
- Fork Jevaluate and change its rubric.
- Publish a ratings folder.
It doesn't work without Jevaluate: every step runs Jevaluate's scripts, rubric or eval.
For a different LLM rating skill, this one won't run, but its pattern carries over: fixed test cases, one logger, and reviewers that see only the facts.
How it works
The skill is a short guide that opens one of three files, each a numbered list with the check that must pass:
rubric-change.mdchanges the rubric: a meaning and an example for every value, one row per stakes level where the rule changes with stakes, and a code check for every rule. Then Jevaluate Eval tests it.rating-round.mdruns a round: screen a list against two known controls, size each repo, brief raters, scan each rater's transcript, log ratings one at a time and send big verdict moves to a reviewer that sees only the facts. A split between two raters that changes the verdict, and that no written ruling settles, goes to the owner as the rule, the code and two unlabeled answers.publish.mdexports a preview, has a reviewer check every page for private context, runs the release check (tests, a fresh clone, a passing eval on this rubric), and opens one pull request.
Written steps alone slipped in earlier rounds, so scripts check each one:
- Parallel raters wrote to the shared library at the same time. Now the controller alone logs each rating, one at a time, through
library.py add. - Sizing fetched one large repo file by file for over 15 minutes. Sizing now uses the repo's file list, and large repos are read from a shallow clone.
- A mode the skill didn't have shipped in its README. The lint now refuses any file a skill names that doesn't exist.
harness_status.py reads your round's ledger, reruns the materials lint and the release check, and names the file to open next. A skipped step shows up there before the next one starts.
The system page also shows the routing flowchart, the eval and who is blind to what.
Install and use
Needs Claude Code or Codex, Python 3.8+, the GitHub CLI (gh, logged in) and Jevaluate Publish.
In Claude Code:
claude plugin marketplace add tiffygk/jev-mode
claude plugin install jevaluate-harness@jev-mode
claude plugin install jevaluate@jev-mode
In Codex:
codex plugin marketplace add tiffygk/jev-mode
codex plugin add jevaluate-harness@jev-mode
codex plugin add jevaluate@jev-mode
Put your never-publish regexes in $JEVALUATE_LIBRARY/private-terms.txt, and optionally set $JEVALUATE_OVERLAY to a markdown note of your own rules for these workflows (see the skill's Settings). Then ask Claude Code: run a jevaluate re-rate round.
To install every jev-mode skill at once, or to copy skills by hand, see the repo README.
Limits
Like Jevaluate, it never calls Jev, so it needs no TypeSafe API key. It rates nothing itself: each rating is a full Jevaluate run, about 100-300k tokens by repo size. Not affiliated with TypeSafe.
License
PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: open an issue.
Skill manifest
Jevaluate Harness
The maintainer's process for jevaluate-publish. One rating is jevaluate/read.md; testing the rubric is the jevaluate-eval skill. If $JEVALUATE_OVERLAY is set, read that file first; its rules override the defaults here.
Settings (yours, not in this skill)
$JEVALUATE_LIBRARY: the private ratings library (default~/.claude/jevaluate-libraryif it exists, else~/.jevaluate-library).$PRIVATE_TERMS: your never-publish regexes, one per line.library.py exportreads only$JEVALUATE_LIBRARY/private-terms.txt, so keep the file there (or symlink it) and have your pre-push hook read the same file.$JEVALUATE_OVERLAY: a markdown note of your own rules for these workflows, read first; no required format.
Two principles
- Code checks, text explains: every rule has a row in
jevaluate/enforcement.mdnaming the check that refuses a wrong answer, or why it stays judgment. - Ask for what is observed: raters record type, the calls-Jev line, each decision's stakes as read and the verdict;
library.py addderives kind and every n.a.
Workflows
Open the file for the workflow and follow it in order; each step names the gate that must pass.
| Workflow | File | Done when |
|---|---|---|
| Change the rubric | rubric-change.md |
jevaluate-eval passes and the owner approves the version |
| Rate or re-rate projects | rating-round.md (rater prompt: rater-brief.md) |
every rating logged by library.py add, scan clean |
| Publish ratings | publish.md |
one pull request, reviewed |
To see where you are: python3 jevaluate-harness/scripts/harness_status.py <ledger> --workflow rubric|round (round runs through publishing). It reruns the scripted gates it has (the materials check), reads the other steps from the ledger, and names the file to open next. Ledger lines: done: <step> <YYYY-MM-DD> <evidence>. Steps: rubric = rules-written, materials-checked, quiz-passed, eval-passed, owner-approved, frozen; round = picked, sized, costs-approved, rated, scanned, adjudicated, final-gate, release-check, published.
Keep a ledger per workflow: each step, each ruling with its cost if wrong, and tokens per agent. Record rulings and handoffs where your overlay says, or in the ledger.
Enforced by
| Rule | Enforced by |
|---|---|
| A rater never reads the rubric, past ratings or a full-file copy directly | scan_transcripts.py; test_scan.py |
| A freeze tag goes only on a merged commit whose rubric title carries its version | freeze_tag_check.py |
| A workflow's scripted gates are rerun, not read from the ledger | scripts/harness_status.py; test_status.py |
| Every release check passes before publishing | scripts/check_release.py; test_release.py |
| Each step followed in order, rulings recorded with their cost if wrong | LLM's judgment following instructions |
Files (jev-mode)
-
fixtures
-
bad-rating-evidence
-
files
-
src__x.py 103 B
Loading…
-
-
-
good-rating-evidence
-
files
-
src__x.py 103 B
Loading…
-
-
-
bad-rating.md 1.5 KB
Loading… -
bad-rating.md.steps.json 517 B
Loading… -
good-rating.md 1.5 KB
Loading… -
good-rating.md.steps.json 517 B
Loading…
-
-
images
-
skills-dark.png 407.6 KB · in bundle
-
skills-light.png 364.7 KB · in bundle
-
-
scripts
-
check_release.py 3.3 KB
Loading… -
dispute_packet.py 3.6 KB
Loading… -
harness_status.py 2.9 KB
Loading…
-
-
freeze_tag_check.py 2.6 KB
Loading… -
publish.md 900 B
Loading… -
rater-brief.md 2.1 KB
Loading… -
rating-round.md 4.3 KB
Loading… -
README.md 4 KB
# Jevaluate Harness Runs [Jevaluate](../jevaluate/) over many projects at once, guides a rubric change, and publishes the ratings. Testing a rubric change is [Jevaluate Eval](../jevaluate-eval/). Jevaluate rates one project on its own. This skill is the harness around it: it never judges a project or edits a rating. Without it, parallel raters collide on the shared library, a rubric change leaves every older rating stale, and nothing shows whether the change made ratings better or worse. ## Who it's for Jevaluate users who: - Rate a whole list of projects in one round. - Fork Jevaluate and change its rubric. - Publish a ratings folder. It doesn't work without Jevaluate: every step runs Jevaluate's scripts, rubric or eval. For a different LLM rating skill, this one won't run, but its pattern carries over: fixed test cases, one logger, and reviewers that see only the facts. ## How it works The skill is a short guide that opens one of three files, each a numbered list with the check that must pass: 1. `rubric-change.md` changes the rubric: a meaning and an example for every value, one row per stakes level where the rule changes with stakes, and a code check for every rule. Then Jevaluate Eval tests it. 2. `rating-round.md` runs a round: screen a list against two known controls, size each repo, brief raters, scan each rater's transcript, log ratings one at a time and send big verdict moves to a reviewer that sees only the facts. A split between two raters that changes the verdict, and that no written ruling settles, goes to the owner as the rule, the code and two unlabeled answers. 3. `publish.md` exports a preview, has a reviewer check every page for private context, runs the release check (tests, a fresh clone, a passing eval on this rubric), and opens one pull request. Written steps alone slipped in earlier rounds, so scripts check each one: - Parallel raters wrote to the shared library at the same time. Now the controller alone logs each rating, one at a time, through `library.py add`. - Sizing fetched one large repo file by file for over 15 minutes. Sizing now uses the repo's file list, and large repos are read from a shallow clone. - A mode the skill didn't have shipped in its README. The lint now refuses any file a skill names that doesn't exist. `harness_status.py` reads your round's ledger, reruns the materials lint and the release check, and names the file to open next. A skipped step shows up there before the next one starts. <a href="https://tiffygk.github.io/jev-mode/system/#d1-h"><picture><source media="(prefers-color-scheme: dark)" srcset="images/skills-dark.png"><img alt="How the skills fit" src="images/skills-light.png"></picture></a> The [system page](https://tiffygk.github.io/jev-mode/system/) also shows the routing flowchart, the eval and who is blind to what. ## Install and use Needs Claude Code or Codex, Python 3.8+, the GitHub CLI (`gh`, logged in) and [Jevaluate Publish](../jevaluate/). In Claude Code: ``` claude plugin marketplace add tiffygk/jev-mode claude plugin install jevaluate-harness@jev-mode claude plugin install jevaluate@jev-mode ``` In Codex: ``` codex plugin marketplace add tiffygk/jev-mode codex plugin add jevaluate-harness@jev-mode codex plugin add jevaluate@jev-mode ``` Put your never-publish regexes in `$JEVALUATE_LIBRARY/private-terms.txt`, and optionally set `$JEVALUATE_OVERLAY` to a markdown note of your own rules for these workflows (see the skill's Settings). Then ask Claude Code: `run a jevaluate re-rate round`. To install every jev-mode skill at once, or to copy skills by hand, see the [repo README](../README.md#install). ## Limits Like Jevaluate, it never calls Jev, so it needs no TypeSafe API key. It rates nothing itself: each rating is a full Jevaluate run, about 100-300k tokens by repo size. Not affiliated with TypeSafe. ## License PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: [open an issue](https://github.com/tiffygk/jev-mode/issues). -
rubric-change.md 3.3 KB
Loading… -
scan_transcripts.py 2.8 KB
Loading… -
SKILL.md 3 KB
--- name: jevaluate-harness description: Use when running a jevaluate rating round, re-rating after a rubric change, changing the jevaluate rubric, or publishing ratings to the jev-mode ratings folder. --- # Jevaluate Harness The maintainer's process for `jevaluate-publish`. One rating is `jevaluate/read.md`; testing the rubric is the `jevaluate-eval` skill. If `$JEVALUATE_OVERLAY` is set, read that file first; its rules override the defaults here. ## Settings (yours, not in this skill) - `$JEVALUATE_LIBRARY`: the private ratings library (default `~/.claude/jevaluate-library` if it exists, else `~/.jevaluate-library`). - `$PRIVATE_TERMS`: your never-publish regexes, one per line. `library.py export` reads only `$JEVALUATE_LIBRARY/private-terms.txt`, so keep the file there (or symlink it) and have your pre-push hook read the same file. - `$JEVALUATE_OVERLAY`: a markdown note of your own rules for these workflows, read first; no required format. ## Two principles - **Code checks, text explains:** every rule has a row in `jevaluate/enforcement.md` naming the check that refuses a wrong answer, or why it stays judgment. - **Ask for what is observed:** raters record type, the calls-Jev line, each decision's stakes as read and the verdict; `library.py add` derives kind and every n.a. ## Workflows Open the file for the workflow and follow it in order; each step names the gate that must pass. | Workflow | File | Done when | |---|---|---| | Change the rubric | `rubric-change.md` | `jevaluate-eval` passes and the owner approves the version | | Rate or re-rate projects | `rating-round.md` (rater prompt: `rater-brief.md`) | every rating logged by `library.py add`, scan clean | | Publish ratings | `publish.md` | one pull request, reviewed | To see where you are: `python3 jevaluate-harness/scripts/harness_status.py <ledger> --workflow rubric|round` (`round` runs through publishing). It reruns the scripted gates it has (the materials check), reads the other steps from the ledger, and names the file to open next. Ledger lines: `done: <step> <YYYY-MM-DD> <evidence>`. Steps: rubric = rules-written, materials-checked, quiz-passed, eval-passed, owner-approved, frozen; round = picked, sized, costs-approved, rated, scanned, adjudicated, final-gate, release-check, published. Keep a ledger per workflow: each step, each ruling with its cost if wrong, and tokens per agent. Record rulings and handoffs where your overlay says, or in the ledger. ## Enforced by | Rule | Enforced by | |---|---| | A rater never reads the rubric, past ratings or a full-file copy directly | `scan_transcripts.py`; `test_scan.py` | | A freeze tag goes only on a merged commit whose rubric title carries its version | `freeze_tag_check.py` | | A workflow's scripted gates are rerun, not read from the ledger | `scripts/harness_status.py`; `test_status.py` | | Every release check passes before publishing | `scripts/check_release.py`; `test_release.py` | | Each step followed in order, rulings recorded with their cost if wrong | LLM's judgment following instructions | -
test_dispute_packet.py 2.3 KB
Loading… -
test_release.py 2.1 KB
Loading… -
test_scan.py 4.9 KB
Loading… -
test_status.py 2.8 KB
Loading…
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.