jevaluate
Use when rating whether a project, repo, skill, workflow, plugin, agent product or post uses TypeSafe's Jev (System One) model well -- "is this a good use of Jev?", "rate this Jev integration", "jevaluate <url>", or a link to something named after Jev; also when checking whether
Install
npx skills add https://github.com/tiffygk/jev-mode/tree/main/jevaluate
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tiffygk-jev-mode@llmmart
git clone https://github.com/tiffygk/jev-mode.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tiffygk/jev-mode collection as a plugin from our marketplace. Git is the plain clone.
README
Jevaluate Publish
Jevaluate Publish reads a Jev project's code and rates how well it uses Jev, for this repo's public ratings list. Every finding cites a file and line in the project and the TypeSafe doc or cookbook behind its rule.
For your own or your team's project, use Jevaluate Private: same rubric, nothing published to this repo's public ratings page.
Who it's for
- Developers choosing a Jev tool to build on.
- Enterprise teams comparing Jev integrations on one scale.
What you get
valentynkit/jev-belay at
ef719dbVerdict 4: Use it
Execution ●●● · Fit ●●○ · Coverage ●●● · Evidence ●●○
- One batched request, four atomic questions, pinned to
jev-1.13.0- Thresholds live in code; a fresh passing check is a veto Jev can't override
- Measured: AUROC 0.976 on 100 labeled stops
Top fix: the threshold was tuned on the sample it reports on. Re-check it on a fresh slice.
The rater picks the verdict, never an average; code refuses one above these rules:
| Verdict | Definition | Rule |
|---|---|---|
| 5 Learn from it | Reference-grade: follows the rules, measured on labels | Execution, Fit and Evidence all 3, with thresholds or questions retuned on its results |
| 4 Use it | Correct use, minor gaps | Execution and Fit 2 or better, no failed core fact |
| 3 Use with a fix | Right idea, fixable flaws | A failed core fact, or results claimed with no measurement |
| 2 Rework it | Core rules broken | Execution 1 or 0, or a fatal flaw: Jev computes values, or confidence is ignored on a high-stakes action |
| 1 with a code | Not rated on the scale | 1t guides and 1r Jev replacements aren't rated yet; 1a false marketing: claims Jev, never calls it; 1b: no call claimed; 1c: answers unused |
| Can't rate yet | Too little visible to judge | README only, or no traced call to Jev found yet |
How it works
- Pins the commit, fetches every file that could change the verdict, and states the cost. Above 200k tokens it stops until you approve.
- Routes the project: the code line that calls Jev (an import, README or design doc never counts), then its type.
- Answers 23 facts, each citing a file:line or quote, and rates each decision's stakes.
- Scores four dimensions.
- Compares with the closest past ratings and the project's previous rating.
- Sets the verdict, traces each fix to the failed fact and the TypeSafe page with the remedy, then logs it.
Code keeps raters on track
In round 1, a rater recorded a partial read as a full one, and a rating with 10 of 24 facts unknown passed as a 4. Now scripts run the steps:
step.pyserves the rubric one part at a time, in the order above, with only the rows that apply.- Raters can't skip ahead: routing changed after the facts is refused, the previous rating appears only after the facts and scores, and a transcript scan flags a rater that opens the rubric or past ratings.
library.py checkrefuses a rating that breaks a rule and names the line to redo;addfills in the kind, the top stakes and every "not applicable".
Where the rules come from
The 23 facts are in rubric.md. Each rule in shared/jev-rules.md cites its source: TypeSafe's docs, a cookbook, or a finding from rating projects.
Install and use
Needs Claude Code or Codex, git, Python 3.8+ and the GitHub CLI (gh, logged in).
In Claude Code:
claude plugin marketplace add tiffygk/jev-mode
claude plugin install jevaluate@jev-mode
In Codex:
codex plugin marketplace add tiffygk/jev-mode
codex plugin add jevaluate@jev-mode
Then ask your agent: jevaluate-publish https://github.com/valentynkit/jev-belay. Use a Sonnet-class model at medium effort (gpt-6-sol in Codex). A rating costs about 100-300k tokens. Ratings save to $JEVALUATE_LIBRARY, ~/.claude/jevaluate-library/ if it exists, or ~/.jevaluate-library/.
To install every jev-mode skill at once, or to copy skills by hand, see the repo README.
Published ratings
Ratings of community projects are in ratings/, under CC0. To keep a project off the list, name it in ratings-template/unpublished.txt.
Limits
It doesn't call Jev, so it needs no TypeSafe key. It reads code and never runs it. Not affiliated with TypeSafe.
License
PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: open an issue.
Skill manifest
Jevaluate Publish
Rates how well something uses Jev, TypeSafe's System One model (typed questions over a JSON state, answered in parallel with calibrated probabilities). Every rating is evidence-backed, dated, tied to a commit, and logged, so the library of past ratings anchors future ones.
How to rate
Follow read.md; scripts/step.py serves the rubric, the fix rows and past ratings one phase at a time. Jevaluate rates projects; it doesn't run or diagnose your own pipeline. If asked for that, say so and stop.
Model
Run on a Sonnet-class model at medium effort, the same one every time: ratings stay comparable only if the rater doesn't change. In Codex, run on gpt-6-sol at medium effort. In another harness, pick one model and effort and keep them. Record both in the rating (rater, effort); a rating by a different model is a different rater's.
Verdicts
Verdicts run from 5 "Learn from it" down to the verdict-1 codes and "Can't rate yet", defined in rubric.md section 5. They come from the rules, never from averaging scores. Any verdict of 3 or below states its reasoning and the core fixes.
The library
Ratings are logged outside this skill, in $JEVALUATE_LIBRARY if set, else ~/.claude/jevaluate-library/ if it exists (libraries made before Codex support), else ~/.jevaluate-library/; scripts/library.py creates it. Never overwrite a past rating: a re-rating is a new dated entry.
Sources
Only public sources: the docs at docs.typesafe.ai (page list: https://docs.typesafe.ai/llms.txt; append .md to any page for markdown) and the project's own public files. Credit a repo to its GitHub owner, never to its banner or branding. A project's claims about its own results are claims until the rating checks them.
In Codex
Codex's default sandbox blocks the network and writes outside the workspace. Fetching a repo (gh, git) and library.py add need your approval, or run with network access and the library folder as a writable root.
Files (jev-mode)
-
images
-
routing-dark.png 515.7 KB · in bundle
-
routing-light.png 464.4 KB · in bundle
-
-
ratings-template
-
LICENSE 6.9 KB · in bundle
-
README.md 2.6 KB
Loading… -
unpublished.txt 283 B
Loading…
-
-
scripts
-
conftest.py 432 B
Loading… -
coverage_manifest.py 8.5 KB
Loading… -
fetch_repo.py 4.4 KB
Loading… -
jev_callsites.py 17.7 KB
Loading… -
library.py 75.4 KB
Loading… -
rubric_text.py 8.7 KB
Loading… -
screen_list.py 6.2 KB
Loading… -
step.py 8 KB
Loading… -
test_export_names.py 2.9 KB
Loading… -
test_isolation.py 472 B
Loading… -
test_library.py 96.2 KB
Loading… -
test_lock_hardening.py 2.6 KB
Loading… -
test_lock_round3.py 5.9 KB
Loading… -
test_private.py 3.8 KB
Loading… -
test_retro_2026_10_06.py 4.9 KB
Loading… -
test_rubric_lock.py 7.5 KB
Loading… -
test_rubric_text.py 1.3 KB
Loading… -
test_scripts.py 22.5 KB
Loading… -
test_step.py 13.1 KB
Loading…
-
-
enforcement.md 7.9 KB
Loading… -
fix-catalog.md 5.6 KB
Loading… -
read.md 6.8 KB
Loading… -
README.md 4.8 KB
# Jevaluate Publish Jevaluate Publish reads a Jev project's code and rates how well it uses Jev, for this repo's public ratings list. Every finding cites a file and line in the project and the TypeSafe doc or cookbook behind its rule. For your own or your team's project, use [Jevaluate Private](../jevaluate-private/): same rubric, nothing published to this repo's public ratings page. ## Who it's for - Developers choosing a Jev tool to build on. - Enterprise teams comparing Jev integrations on one scale. ## What you get > **valentynkit/jev-belay** at `ef719db` > ### Verdict 4: Use it > Execution ●●● · Fit ●●○ · Coverage ●●● · Evidence ●●○ > > - One batched request, four atomic questions, pinned to `jev-1.13.0` > - Thresholds live in code; a fresh passing check is a veto Jev can't override > - Measured: AUROC 0.976 on 100 labeled stops > > **Top fix:** the threshold was tuned on the sample it reports on. Re-check it on a fresh slice. The rater picks the verdict, never an average; code refuses one above these rules: | Verdict | Definition | Rule | |---|---|---| | **5 Learn from it** | Reference-grade: follows the rules, measured on labels | Execution, Fit and Evidence all 3, with thresholds or questions retuned on its results | | **4 Use it** | Correct use, minor gaps | Execution and Fit 2 or better, no failed core fact | | **3 Use with a fix** | Right idea, fixable flaws | A failed core fact, or results claimed with no measurement | | **2 Rework it** | Core rules broken | Execution 1 or 0, or a fatal flaw: Jev computes values, or confidence is ignored on a high-stakes action | | **1 with a code** | Not rated on the scale | 1t guides and 1r Jev replacements aren't rated yet; [1a](rubric.md) false marketing: claims Jev, never calls it; 1b: no call claimed; 1c: answers unused | | **Can't rate yet** | Too little visible to judge | README only, or no traced call to Jev found yet | ## How it works 1. Pins the commit, fetches every file that could change the verdict, and states the cost. Above 200k tokens it stops until you approve. 2. Routes the project: the code line that calls Jev (an import, README or design doc never counts), then its type. 3. Answers 23 facts, each citing a file:line or quote, and rates each decision's stakes. 4. Scores four dimensions. 5. Compares with the closest past ratings and the project's previous rating. 6. Sets the verdict, traces each fix to the failed fact and the TypeSafe page with the remedy, then logs it. ## Code keeps raters on track In round 1, a rater recorded a partial read as a full one, and a rating with 10 of 24 facts unknown passed as a 4. Now scripts run the steps: - `step.py` serves the rubric one part at a time, in the order above, with only the rows that apply. - Raters can't skip ahead: routing changed after the facts is refused, the previous rating appears only after the facts and scores, and a transcript scan flags a rater that opens the rubric or past ratings. - `library.py check` refuses a rating that breaks a rule and names the line to redo; `add` fills in the kind, the top stakes and every "not applicable". <a href="https://tiffygk.github.io/jev-mode/system/#d2-h"><picture><source media="(prefers-color-scheme: dark)" srcset="images/routing-dark.png"><img alt="Routing flowchart" src="images/routing-light.png"></picture></a> ## Where the rules come from The 23 facts are in [`rubric.md`](rubric.md). Each rule in [`shared/jev-rules.md`](../shared/jev-rules.md) cites its source: TypeSafe's docs, a cookbook, or a finding from rating projects. ## Install and use Needs Claude Code or Codex, git, Python 3.8+ and the GitHub CLI (`gh`, logged in). In Claude Code: ``` claude plugin marketplace add tiffygk/jev-mode claude plugin install jevaluate@jev-mode ``` In Codex: ``` codex plugin marketplace add tiffygk/jev-mode codex plugin add jevaluate@jev-mode ``` Then ask your agent: `jevaluate-publish https://github.com/valentynkit/jev-belay`. Use a Sonnet-class model at medium effort (`gpt-6-sol` in Codex). A rating costs about 100-300k tokens. Ratings save to `$JEVALUATE_LIBRARY`, `~/.claude/jevaluate-library/` if it exists, or `~/.jevaluate-library/`. To install every jev-mode skill at once, or to copy skills by hand, see the [repo README](../README.md#install). ## Published ratings Ratings of community projects are in [`ratings/`](../ratings/), under CC0. To keep a project off the list, name it in [`ratings-template/unpublished.txt`](ratings-template/unpublished.txt). ## Limits It doesn't call Jev, so it needs no TypeSafe key. It reads code and never runs it. Not affiliated with TypeSafe. ## License PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: [open an issue](https://github.com/tiffygk/jev-mode/issues). -
rubric.md 32.2 KB
Loading… -
SKILL.md 2.5 KB
--- name: jevaluate-publish description: Use when rating whether a project, repo, skill, workflow, plugin, agent product or post uses TypeSafe's Jev (System One) model well -- "is this a good use of Jev?", "rate this Jev integration", "jevaluate-publish https://github.com/owner/repo", or a link to something named after Jev; also when checking whether something that calls itself Jev actually uses it. The rating goes to the public ratings list. For a person's own or team's project, jevaluate-private is the one to use. --- # Jevaluate Publish Rates how well something uses Jev, TypeSafe's System One model (typed questions over a JSON `state`, answered in parallel with calibrated probabilities). Every rating is evidence-backed, dated, tied to a commit, and logged, so the library of past ratings anchors future ones. ## How to rate Follow `read.md`; `scripts/step.py` serves the rubric, the fix rows and past ratings one phase at a time. Jevaluate rates projects; it doesn't run or diagnose your own pipeline. If asked for that, say so and stop. ## Model Run on a Sonnet-class model at medium effort, the same one every time: ratings stay comparable only if the rater doesn't change. In Codex, run on `gpt-6-sol` at medium effort. In another harness, pick one model and effort and keep them. Record both in the rating (`rater`, `effort`); a rating by a different model is a different rater's. ## Verdicts Verdicts run from 5 "Learn from it" down to the verdict-1 codes and "Can't rate yet", defined in `rubric.md` section 5. They come from the rules, never from averaging scores. Any verdict of 3 or below states its reasoning and the core fixes. ## The library Ratings are logged outside this skill, in `$JEVALUATE_LIBRARY` if set, else `~/.claude/jevaluate-library/` if it exists (libraries made before Codex support), else `~/.jevaluate-library/`; `scripts/library.py` creates it. Never overwrite a past rating: a re-rating is a new dated entry. ## Sources Only public sources: the docs at docs.typesafe.ai (page list: `https://docs.typesafe.ai/llms.txt`; append `.md` to any page for markdown) and the project's own public files. Credit a repo to its GitHub owner, never to its banner or branding. A project's claims about its own results are claims until the rating checks them. ## In Codex Codex's default sandbox blocks the network and writes outside the workspace. Fetching a repo (`gh`, `git`) and `library.py add` need your approval, or run with network access and the library folder as a writable root.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.