Claude Skill

jevaluate

Use when rating whether a project, repo, skill, workflow, plugin, agent product or post uses TypeSafe's Jev (System One) model well -- "is this a good use of Jev?", "rate this Jev integration", "jevaluate <url>", or a link to something named after Jev; also when checking whether

LLM Mart · 0 points · 1 views 53 listing impressions 0 install-command copies
Download tiffygk-jev-mode-jevaluate-30a0a86.zip · 1009 KB

Install

skills CLI npx skills add https://github.com/tiffygk/jev-mode/tree/main/jevaluate
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tiffygk-jev-mode@llmmart
Git git clone https://github.com/tiffygk/jev-mode.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tiffygk/jev-mode collection as a plugin from our marketplace. Git is the plain clone.

README

Jevaluate Publish

Jevaluate Publish reads a Jev project's code and rates how well it uses Jev, for this repo's public ratings list. Every finding cites a file and line in the project and the TypeSafe doc or cookbook behind its rule.

For your own or your team's project, use Jevaluate Private: same rubric, nothing published to this repo's public ratings page.

Who it's for

  • Developers choosing a Jev tool to build on.
  • Enterprise teams comparing Jev integrations on one scale.

What you get

valentynkit/jev-belay at ef719db

Verdict 4: Use it

Execution ●●● · Fit ●●○ · Coverage ●●● · Evidence ●●○

  • One batched request, four atomic questions, pinned to jev-1.13.0
  • Thresholds live in code; a fresh passing check is a veto Jev can't override
  • Measured: AUROC 0.976 on 100 labeled stops

Top fix: the threshold was tuned on the sample it reports on. Re-check it on a fresh slice.

The rater picks the verdict, never an average; code refuses one above these rules:

Verdict Definition Rule
5 Learn from it Reference-grade: follows the rules, measured on labels Execution, Fit and Evidence all 3, with thresholds or questions retuned on its results
4 Use it Correct use, minor gaps Execution and Fit 2 or better, no failed core fact
3 Use with a fix Right idea, fixable flaws A failed core fact, or results claimed with no measurement
2 Rework it Core rules broken Execution 1 or 0, or a fatal flaw: Jev computes values, or confidence is ignored on a high-stakes action
1 with a code Not rated on the scale 1t guides and 1r Jev replacements aren't rated yet; 1a false marketing: claims Jev, never calls it; 1b: no call claimed; 1c: answers unused
Can't rate yet Too little visible to judge README only, or no traced call to Jev found yet

How it works

  1. Pins the commit, fetches every file that could change the verdict, and states the cost. Above 200k tokens it stops until you approve.
  2. Routes the project: the code line that calls Jev (an import, README or design doc never counts), then its type.
  3. Answers 23 facts, each citing a file:line or quote, and rates each decision's stakes.
  4. Scores four dimensions.
  5. Compares with the closest past ratings and the project's previous rating.
  6. Sets the verdict, traces each fix to the failed fact and the TypeSafe page with the remedy, then logs it.

Code keeps raters on track

In round 1, a rater recorded a partial read as a full one, and a rating with 10 of 24 facts unknown passed as a 4. Now scripts run the steps:

  • step.py serves the rubric one part at a time, in the order above, with only the rows that apply.
  • Raters can't skip ahead: routing changed after the facts is refused, the previous rating appears only after the facts and scores, and a transcript scan flags a rater that opens the rubric or past ratings.
  • library.py check refuses a rating that breaks a rule and names the line to redo; add fills in the kind, the top stakes and every "not applicable".

Where the rules come from

The 23 facts are in rubric.md. Each rule in shared/jev-rules.md cites its source: TypeSafe's docs, a cookbook, or a finding from rating projects.

Install and use

Needs Claude Code or Codex, git, Python 3.8+ and the GitHub CLI (gh, logged in).

In Claude Code:

claude plugin marketplace add tiffygk/jev-mode
claude plugin install jevaluate@jev-mode

In Codex:

codex plugin marketplace add tiffygk/jev-mode
codex plugin add jevaluate@jev-mode

Then ask your agent: jevaluate-publish https://github.com/valentynkit/jev-belay. Use a Sonnet-class model at medium effort (gpt-6-sol in Codex). A rating costs about 100-300k tokens. Ratings save to $JEVALUATE_LIBRARY, ~/.claude/jevaluate-library/ if it exists, or ~/.jevaluate-library/.

To install every jev-mode skill at once, or to copy skills by hand, see the repo README.

Published ratings

Ratings of community projects are in ratings/, under CC0. To keep a project off the list, name it in ratings-template/unpublished.txt.

Limits

It doesn't call Jev, so it needs no TypeSafe key. It reads code and never runs it. Not affiliated with TypeSafe.

License

PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: open an issue.

Skill manifest

Jevaluate Publish

Rates how well something uses Jev, TypeSafe's System One model (typed questions over a JSON state, answered in parallel with calibrated probabilities). Every rating is evidence-backed, dated, tied to a commit, and logged, so the library of past ratings anchors future ones.

How to rate

Follow read.md; scripts/step.py serves the rubric, the fix rows and past ratings one phase at a time. Jevaluate rates projects; it doesn't run or diagnose your own pipeline. If asked for that, say so and stop.

Model

Run on a Sonnet-class model at medium effort, the same one every time: ratings stay comparable only if the rater doesn't change. In Codex, run on gpt-6-sol at medium effort. In another harness, pick one model and effort and keep them. Record both in the rating (rater, effort); a rating by a different model is a different rater's.

Verdicts

Verdicts run from 5 "Learn from it" down to the verdict-1 codes and "Can't rate yet", defined in rubric.md section 5. They come from the rules, never from averaging scores. Any verdict of 3 or below states its reasoning and the core fixes.

The library

Ratings are logged outside this skill, in $JEVALUATE_LIBRARY if set, else ~/.claude/jevaluate-library/ if it exists (libraries made before Codex support), else ~/.jevaluate-library/; scripts/library.py creates it. Never overwrite a past rating: a re-rating is a new dated entry.

Sources

Only public sources: the docs at docs.typesafe.ai (page list: https://docs.typesafe.ai/llms.txt; append .md to any page for markdown) and the project's own public files. Credit a repo to its GitHub owner, never to its banner or branding. A project's claims about its own results are claims until the rating checks them.

In Codex

Codex's default sandbox blocks the network and writes outside the workspace. Fetching a repo (gh, git) and library.py add need your approval, or run with network access and the library folder as a writable root.

Files (jev-mode)
  • images
    • routing-dark.png 515.7 KB · in bundle
    • routing-light.png 464.4 KB · in bundle
  • ratings-template
    • LICENSE 6.9 KB · in bundle
    • README.md 2.6 KB
      Loading…
    • unpublished.txt 283 B
      Loading…
  • scripts
    • conftest.py 432 B
      Loading…
    • coverage_manifest.py 8.5 KB
      Loading…
    • fetch_repo.py 4.4 KB
      Loading…
    • jev_callsites.py 17.7 KB
      Loading…
    • library.py 75.4 KB
      Loading…
    • rubric_text.py 8.7 KB
      Loading…
    • screen_list.py 6.2 KB
      Loading…
    • step.py 8 KB
      Loading…
    • test_export_names.py 2.9 KB
      Loading…
    • test_isolation.py 472 B
      Loading…
    • test_library.py 96.2 KB
      Loading…
    • test_lock_hardening.py 2.6 KB
      Loading…
    • test_lock_round3.py 5.9 KB
      Loading…
    • test_private.py 3.8 KB
      Loading…
    • test_retro_2026_10_06.py 4.9 KB
      Loading…
    • test_rubric_lock.py 7.5 KB
      Loading…
    • test_rubric_text.py 1.3 KB
      Loading…
    • test_scripts.py 22.5 KB
      Loading…
    • test_step.py 13.1 KB
      Loading…
  • enforcement.md 7.9 KB
    Loading…
  • fix-catalog.md 5.6 KB
    Loading…
  • read.md 6.8 KB
    Loading…
  • README.md 4.8 KB
    # Jevaluate Publish
    
    Jevaluate Publish reads a Jev project's code and rates how well it uses Jev, for this repo's public ratings list. Every finding cites a file and line in the project and the TypeSafe doc or cookbook behind its rule.
    
    For your own or your team's project, use [Jevaluate Private](../jevaluate-private/): same rubric, nothing published to this repo's public ratings page.
    
    ## Who it's for
    
    - Developers choosing a Jev tool to build on.
    - Enterprise teams comparing Jev integrations on one scale.
    
    ## What you get
    
    > **valentynkit/jev-belay** at `ef719db`
    > ### Verdict 4: Use it
    > Execution ●●● · Fit ●●○ · Coverage ●●● · Evidence ●●○
    >
    > - One batched request, four atomic questions, pinned to `jev-1.13.0`
    > - Thresholds live in code; a fresh passing check is a veto Jev can't override
    > - Measured: AUROC 0.976 on 100 labeled stops
    >
    > **Top fix:** the threshold was tuned on the sample it reports on. Re-check it on a fresh slice.
    
    The rater picks the verdict, never an average; code refuses one above these rules:
    
    | Verdict | Definition | Rule |
    |---|---|---|
    | **5&nbsp;Learn&nbsp;from&nbsp;it** | Reference-grade: follows the rules, measured on labels | Execution, Fit and Evidence all 3, with thresholds or questions retuned on its results |
    | **4&nbsp;Use&nbsp;it** | Correct use, minor gaps | Execution and Fit 2 or better, no failed core fact |
    | **3&nbsp;Use&nbsp;with&nbsp;a&nbsp;fix** | Right idea, fixable flaws | A failed core fact, or results claimed with no measurement |
    | **2&nbsp;Rework&nbsp;it** | Core rules broken | Execution 1 or 0, or a fatal flaw: Jev computes values, or confidence is ignored on a high-stakes action |
    | **1&nbsp;with&nbsp;a&nbsp;code** | Not rated on the scale | 1t guides and 1r Jev replacements aren't rated yet; [1a](rubric.md) false marketing: claims Jev, never calls it; 1b: no call claimed; 1c: answers unused |
    | **Can't&nbsp;rate&nbsp;yet** | Too little visible to judge | README only, or no traced call to Jev found yet |
    
    ## How it works
    
    1. Pins the commit, fetches every file that could change the verdict, and states the cost. Above 200k tokens it stops until you approve.
    2. Routes the project: the code line that calls Jev (an import, README or design doc never counts), then its type.
    3. Answers 23 facts, each citing a file:line or quote, and rates each decision's stakes.
    4. Scores four dimensions.
    5. Compares with the closest past ratings and the project's previous rating.
    6. Sets the verdict, traces each fix to the failed fact and the TypeSafe page with the remedy, then logs it.
    
    ## Code keeps raters on track
    
    In round 1, a rater recorded a partial read as a full one, and a rating with 10 of 24 facts unknown passed as a 4. Now scripts run the steps:
    
    - `step.py` serves the rubric one part at a time, in the order above, with only the rows that apply.
    - Raters can't skip ahead: routing changed after the facts is refused, the previous rating appears only after the facts and scores, and a transcript scan flags a rater that opens the rubric or past ratings.
    - `library.py check` refuses a rating that breaks a rule and names the line to redo; `add` fills in the kind, the top stakes and every "not applicable".
    
    <a href="https://tiffygk.github.io/jev-mode/system/#d2-h"><picture><source media="(prefers-color-scheme: dark)" srcset="images/routing-dark.png"><img alt="Routing flowchart" src="images/routing-light.png"></picture></a>
    
    ## Where the rules come from
    
    The 23 facts are in [`rubric.md`](rubric.md). Each rule in [`shared/jev-rules.md`](../shared/jev-rules.md) cites its source: TypeSafe's docs, a cookbook, or a finding from rating projects.
    
    ## Install and use
    
    Needs Claude Code or Codex, git, Python 3.8+ and the GitHub CLI (`gh`, logged in).
    
    In Claude Code:
    
    ```
    claude plugin marketplace add tiffygk/jev-mode
    claude plugin install jevaluate@jev-mode
    ```
    
    In Codex:
    
    ```
    codex plugin marketplace add tiffygk/jev-mode
    codex plugin add jevaluate@jev-mode
    ```
    
    Then ask your agent: `jevaluate-publish https://github.com/valentynkit/jev-belay`. Use a Sonnet-class model at medium effort (`gpt-6-sol` in Codex). A rating costs about 100-300k tokens. Ratings save to `$JEVALUATE_LIBRARY`, `~/.claude/jevaluate-library/` if it exists, or `~/.jevaluate-library/`.
    
    To install every jev-mode skill at once, or to copy skills by hand, see the [repo README](../README.md#install).
    
    ## Published ratings
    
    Ratings of community projects are in [`ratings/`](../ratings/), under CC0. To keep a project off the list, name it in [`ratings-template/unpublished.txt`](ratings-template/unpublished.txt).
    
    ## Limits
    
    It doesn't call Jev, so it needs no TypeSafe key. It reads code and never runs it. Not affiliated with TypeSafe.
    
    ## License
    
    PolyForm Noncommercial 1.0.0: free for personal and noncommercial use, with credit. Company or paid use needs a commercial license: [open an issue](https://github.com/tiffygk/jev-mode/issues).
    
  • rubric.md 32.2 KB
    Loading…
  • SKILL.md 2.5 KB
    ---
    name: jevaluate-publish
    description: Use when rating whether a project, repo, skill, workflow, plugin, agent product or post uses TypeSafe's Jev (System One) model well -- "is this a good use of Jev?", "rate this Jev integration", "jevaluate-publish https://github.com/owner/repo", or a link to something named after Jev; also when checking whether something that calls itself Jev actually uses it. The rating goes to the public ratings list. For a person's own or team's project, jevaluate-private is the one to use.
    ---
    
    # Jevaluate Publish
    
    Rates how well something uses Jev, TypeSafe's System One model (typed questions over a JSON `state`, answered in parallel with calibrated probabilities). Every rating is evidence-backed, dated, tied to a commit, and logged, so the library of past ratings anchors future ones.
    
    ## How to rate
    
    Follow `read.md`; `scripts/step.py` serves the rubric, the fix rows and past ratings one phase at a time. Jevaluate rates projects; it doesn't run or diagnose your own pipeline. If asked for that, say so and stop.
    
    ## Model
    
    Run on a Sonnet-class model at medium effort, the same one every time: ratings stay comparable only if the rater doesn't change. In Codex, run on `gpt-6-sol` at medium effort. In another harness, pick one model and effort and keep them. Record both in the rating (`rater`, `effort`); a rating by a different model is a different rater's.
    
    ## Verdicts
    
    Verdicts run from 5 "Learn from it" down to the verdict-1 codes and "Can't rate yet", defined in `rubric.md` section 5. They come from the rules, never from averaging scores. Any verdict of 3 or below states its reasoning and the core fixes.
    
    ## The library
    
    Ratings are logged outside this skill, in `$JEVALUATE_LIBRARY` if set, else `~/.claude/jevaluate-library/` if it exists (libraries made before Codex support), else `~/.jevaluate-library/`; `scripts/library.py` creates it. Never overwrite a past rating: a re-rating is a new dated entry.
    
    ## Sources
    
    Only public sources: the docs at docs.typesafe.ai (page list: `https://docs.typesafe.ai/llms.txt`; append `.md` to any page for markdown) and the project's own public files. Credit a repo to its GitHub owner, never to its banner or branding. A project's claims about its own results are claims until the rating checks them.
    
    ## In Codex
    
    Codex's default sandbox blocks the network and writes outside the workspace. Fetching a repo (`gh`, `git`) and `library.py add` need your approval, or run with network access and the library folder as a writable root.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.