interface:
  display_name: "Agent Plugin Eval"
  short_description: "Audit and compare portable agent plugins against the Agent Plugins specification"
  default_prompt: "Use $agent-plugin-eval to audit this plugin repository against Agent Plugins 1.0.0 and return an evidence-cited scorecard."

# Jev Integration for Agent Plugin Eval

## Overview

This skill integrates with [TypeSafe Jev](https://github.com/AugmentHCI/typesafe-jev) for subjective quality evaluation of agent plugins. Jev provides calibrated probability judgments (Score) and categorical classifications (Noul) that augment the deterministic checks in `inspect_plugin.py`.

## When to Use Jev

| Evaluation Type | Use Jev? | Rationale |
|-----------------|----------|-----------|
| **Axes 1-3 Conformance** | No | Deterministic checks (JSON validity, schema presence, field existence) |
| **Axis 4 Product Quality** | **Yes (Score)** | Subjective assessment of UX, documentation clarity, error handling |
| **Axis 2 Quality Criteria** | **Yes (Score)** | Schema design elegance, naming consistency, API coherence |
| **Gate Classification** | **Yes (Noul)** | Categorical pass/fail on structure, coherence, validity |
| **Secret Detection** | **Yes (Noul)** | Pattern-based suspicion (confirmed requires external scanner) |
| **Quality Checklist** | **Yes (Noul)** | Present/missing classification for files |

## Discovery Protocol

Before invoking Jev, check availability:

```python
from typesafe import jev_available

if jev_available():
    from typesafe import Score, Noul
    # Use Jev for subjective criteria
else:
    # Fall back to heuristic scoring
```

## Question Categories

### Score Questions (0.0-1.0 probability)

Used for quality dimensions where gradations matter:

| ID | Axis.Criterion | Evaluates |
|----|----------------|-----------|
| `ux_coherence` | 4.6 | Tool interaction coherence |
| `documentation_clarity` | 4.8 | Docs completeness and clarity |
| `error_handling_quality` | 4.9 | Error message quality |
| `input_validation_robustness` | 2.15 | Validation thoroughness |
| `schema_design_quality` | 2.16 | Schema design quality |
| `naming_consistency` | 2.17 | Naming conventions |
| `api_design_elegance` | 2.18 | API simplicity and elegance |

### Noul Questions (categorical labels)

Used for binary or few-class decisions:

| Category | Questions | Labels |
|----------|-----------|--------|
| **Gates (G1-G4)** | 4 | conformant/valid/coherent vs malformed/invalid/incoherent |
| **Secrets** | 4 | suspected / not_suspected |
| **Quality Checklist** | 7 | present / missing |
| **Component Validity** | 4 | valid / invalid / missing |

## Integration Points

### 1. Score Integration (Axis 2 & 4)

```python
def evaluate_quality_with_jev(context: dict) -> dict:
    """Evaluate subjective quality criteria using Jev Score."""
    if not jev_available():
        return heuristic_quality_score(context)

    from typesafe import Score

    scores = {}
    questions = load_jev_questions()["questions"]["score"]

    for q in questions:
        ctx = extract_context(context, q["context_required"])
        result = Score(q["question"], context=ctx)
        scores[q["id"]] = {
            "probability": result.probability,
            "axis": q["axis"],
            "criterion": q["criterion"]
        }

    return aggregate_axis_scores(scores)
```

### 2. Gate Classification (Noul)

```python
def classify_gates_with_jev(context: dict) -> dict:
    """Classify conformance gates using Jev Noul."""
    if not jev_available():
        return deterministic_gate_check(context)

    from typesafe import Noul

    gates = {}
    questions = [q for q in load_jev_questions()["questions"]["noul"]
                 if "gate" in q]

    for q in questions:
        ctx = extract_context(context, q["context_required"])
        result = Noul(q["question"], labels=q["labels"], context=ctx)
        gates[q["gate"]] = {
            "label": result.label,
            "passed": result.label == q["labels"][0]  # First label is positive
        }

    return gates
```

### 3. Secret Detection (Noul)

```python
def detect_secrets_with_jev(context: dict) -> dict:
    """Flag potential secrets for human review using Jev Noul."""
    if not jev_available():
        return regex_secret_scan(context)

    from typesafe import Noul

    findings = {}
    questions = [q for q in load_jev_questions()["questions"]["noul"]
                 if q.get("category") == "secrets"]

    for q in questions:
        ctx = extract_context(context, q["context_required"])
        result = Noul(q["question"], labels=q["labels"], context=ctx)
        findings[q["id"]] = {
            "suspected": result.label == "suspected",
            "confidence": result.probability
        }

    # Flag for human review if any suspected
    return {
        "findings": findings,
        "requires_review": any(f["suspected"] for f in findings.values())
    }
```

## Score Aggregation

Jev Score returns probabilities [0.0, 1.0]. Map to rubric scores:

```python
def aggregate_axis_scores(scores: dict, max_per_axis: int = 25) -> dict:
    """Aggregate Jev scores to rubric scale."""
    axis_scores = {}

    for axis_num in [2, 4]:
        axis_items = [s for s in scores.values() if s["axis"] == axis_num]
        if axis_items:
            avg_prob = sum(s["probability"] for s in axis_items) / len(axis_items)
            axis_scores[f"axis_{axis_num}"] = round(avg_prob * max_per_axis, 1)

    return axis_scores
```

## Fallback Behavior

When Jev is unavailable, the skill falls back to:

1. **Quality scoring**: Heuristic checks (file existence, size, pattern matching)
2. **Gate classification**: Deterministic JSON/schema validation
3. **Secret detection**: Regex pattern matching (high false positive rate)

The fallback is functional but less nuanced than Jev evaluation.

## Context Preparation

Each question specifies required context. Extract from plugin repository:

```python
CONTEXT_EXTRACTORS = {
    "plugin_manifest": lambda repo: read_file(repo, "plugin.json"),
    "tool_definitions": lambda repo: extract_tools(repo),
    "readme": lambda repo: read_file(repo, "README.md"),
    "skill_files": lambda repo: glob_read(repo, "**/SKILL.md"),
    "source_files_sample": lambda repo: sample_source(repo, max_files=10),
    "file_listing": lambda repo: list_files(repo),
    # ... etc
}
```

## Output Format

Jev-enhanced evaluation adds a `jev` section to the report:

```json
{
  "score": 78,
  "axes": { ... },
  "gates": { ... },
  "jev": {
    "available": true,
    "quality_scores": {
      "ux_coherence": 0.72,
      "documentation_clarity": 0.85,
      ...
    },
    "gate_classifications": {
      "G1": {"label": "conformant", "passed": true},
      ...
    },
    "secret_findings": {
      "requires_review": false,
      "findings": { ... }
    }
  }
}
```

# Output Template

Use the user's language. Preserve this section order, but keep empty prose
brief. Omit the comparison section unless `compare` is supplied.

```markdown
# Agent Plugin Evaluation — {plugin name}

> Verdict: {PASS / PARTIAL / FAIL} — {one-sentence reason}
> Evaluated: {date}
> Source: {path or URL} ({commit when available})
> Standard: Agent Plugins {version} ({status}), retrieved {date}
> Audit mode: static / authorized dynamic checks

## Summary

| Metric | Value |
|---|---|
| Conformance | {PASS / PARTIAL / FAIL} |
| Raw score | {n}/100 |
| Final score | {n}/100 {cap explanation, if any} |
| Grade | {A/B/C/D/F} |
| Portable components | {skills count}, {MCP server count} |
| Client extensions | {namespaces or none} |
| Criteria scored / N/A | {n} / {n} |

## Release blockers

1. **{finding}** — `{file:line or JSON pointer}`; spec §{section}.
   Impact: {official failure boundary}. Fix: {smallest conforming change}.

Write “None found” when the gate is PASS. Do not mix recommendations here.

## Scorecard

| # | Criterion | Weight | Score | Evidence |
|---|---|---:|---:|---|
| 1 | Root package and manifest | 3x | {n}/100 | {citation} |
| ... | ... | ... | ... | ... |
| 18 | Maintenance and release hygiene | 1x | {n}/100 | {citation} |

Use `N/A` plus a reason for conditional criteria.

## Findings by component

### Manifest and package
{findings, including non-fatal deviations}

### Skills
{one row or paragraph per discovered Skill; include conformance and quality}

### MCP servers
{one row or paragraph per server; include transport and failure boundary}

### Client extensions
{portable-vs-client-specific assessment}

### Security and containment
{resolved paths, secret scan, and execution-risk findings; never reveal values;
distinguish suspected from confirmed credentials and state the corroboration}

## Prioritized actions

1. **P0 — {action}.** Evidence: `{file:line}`. Verify by: {check}.
2. **P1 — {action}.** Evidence: `{file:line}`. Verify by: {check}.
3. **P2 — {action}.** Evidence: `{file:line}`. Verify by: {check}.

## Limits

{Anything not executed or verified, unsupported extension specs, network limits,
or ambiguity. State that static safety boundaries were intentional.}
```

## Comparison mode

Append after both independent evaluations:

```markdown
## Comparison — {A} vs {B}

| Metric | {A} | {B} | Delta / winner |
|---|---:|---:|---|
| Conformance | {status} | {status} | {result} |
| Raw score | {n} | {n} | {signed delta} |
| Final score | {n} | {n} | {signed delta} |
| Grade | {grade} | {grade} | {result} |

| # | Criterion | {A} | {B} | Delta |
|---|---|---:|---:|---:|
| 1 | Root package and manifest | {n} | {n} | {signed delta} |
| ... | ... | ... | ... | ... |
| 18 | Maintenance and release hygiene | {n} | {n} | {signed delta} |

### Decision

{Which plugin is more conformant, which has better uncapped design quality,
and which is safer to release. Do not collapse those into one vague winner.}

### Shared and unique actions

- **Both:** {shared action}
- **{A}:** {specific action}
- **{B}:** {specific action}
```

# Agent Plugin Evaluation Rubric

Score each applicable criterion from 0–100 and multiply by its weight. Mark a
conditional criterion `N/A` only when the relevant component is absent; exclude
N/A from numerator and denominator. Cite concrete `file:line`, JSON pointer, or
resolved-path evidence for every score.

## Scale

| Score | Meaning |
|---|---|
| 0 | Missing or wholly broken |
| 1–25 | Token attempt; major requirements unmet |
| 26–50 | Partial implementation with serious gaps |
| 51–75 | Functional, with meaningful improvements needed |
| 76–90 | Strong; minor gaps |
| 91–100 | Exemplary and release-ready |

## Axis 1 — Portable conformance

| # | Criterion | Weight | Key question |
|---|---|---:|---|
| 1 | Root package and manifest | 3x | Is there exactly one regular root `plugin.json`, with all resolved package paths contained? |
| 2 | Schema, required fields, and name | 3x | Does the manifest target a supported canonical schema and satisfy every fatal required-field/name rule? |
| 3 | Closed metadata model | 2x | Are top-level and author fields closed, optional types correct, and metadata internally accurate? |
| 4 | Fixed discovery and version coherence | 2x | Are standard components only in fixed locations, with matching spec versions and correct filesystem kinds? |

## Axis 2 — Components and integration

| # | Criterion | Weight | Applies when | Key question |
|---|---|---:|---|---|
| 5 | Agent Skills conformance | 2x | At least one Skill is discovered | Does every immediate Skill satisfy the Agent Skills format and containment rules? |
| 6 | Agent Skills quality | 1x | At least one Skill is discovered | Are triggers, instructions, resources, validation, and progressive disclosure useful and precise? Aggregate all Skills; identify outliers. |
| 7 | MCP configuration conformance | 2x | `mcp.json` exists | Are the closed top level and every server variant valid under the matching schema? |
| 8 | MCP runtime portability | 2x | At least one stdio server exists | Are command, args, cwd, env, placeholders, bundled executables, and dependencies deterministic across installations? |
| 9 | Remote MCP transport quality | 2x | At least one remote server exists | Are URL, TLS, headers, transport choice, redirect assumptions, and authentication boundaries safe and portable? |
| 10 | Extension isolation | 1x | Client-specific data/files exist | Are extensions correctly reverse-domain namespaced without masquerading as portable core? |

## Axis 3 — Safety and resilience

| # | Criterion | Weight | Key question |
|---|---|---:|---|
| 11 | Path containment | 3x | Do symlinks, commands, working directories, and package-relative references remain within their required roots after resolution? |
| 12 | Secrets and least exposure | 3x | Are visible manifests/configs free of credentials and unnecessarily sensitive literals? Are suspected values redacted in the report? |
| 13 | Component independence | 2x | Can valid Skills and MCP entries remain useful when an unrelated component is absent, invalid, unsupported, or unavailable? |
| 14 | Static audit safety | 1x | Can reviewers and clients inspect/install the package without executing hidden setup, hooks, or side effects? |

## Axis 4 — Product quality

| # | Criterion | Weight | Key question |
|---|---|---:|---|
| 15 | Cohesion and portable value | 2x | Do the components form one understandable plugin and deliver meaningful value on conformant clients? |
| 16 | Discoverability and documentation | 1x | Do description, version, repository, license, keywords, and user guidance make purpose, trust, setup, and limits clear? |
| 17 | Validation evidence | 2x | Are schemas, Skills, scripts, and MCP behavior covered by safe reproducible checks or fixtures? Do tests exercise failure boundaries? |
| 18 | Maintenance and release hygiene | 1x | Are versions, dependencies, generated artifacts, executable bits, ignored files, and client extensions intentional and maintainable? |

## Weighted result and gates

Compute the uncapped weighted score:

```text
raw = sum(score × weight) / sum(applicable weights)
```

Then apply the conformance gate from `spec-checklist.md`:

| Gate | Cap | Meaning |
|---|---:|---|
| PASS | none | No normative violation found |
| PARTIAL | 59 | At least one non-fatal deviation or invalid/skipped component |
| FAIL | 39 | Plugin-level fatal failure, root-manifest escape, or confirmed credential |

Grades use the final capped score: A 80–100, B 60–79, C 40–59, D 20–39,
F 0–19. Always report both raw and final scores when a cap applies.

## Comparison rules

- Score each plugin independently, including its N/A denominator and gate.
- Use final score for the headline delta; show raw delta separately when either
  plugin is capped.
- Compare conformance statuses directly; do not imply that a numerical lead can
  compensate for a release blocker.
- Distinguish shared weaknesses from plugin-specific regressions and end with a
  per-plugin action list.

# Agent Plugins 1.0.0 Conformance Checklist

Use this snapshot for Agent Plugins `1.0.0`. The canonical normative text at
<https://agent-plugins.org/specification> governs; JSON Schemas are supporting
validation artifacts. Version 1.0.0 was published as a Working Draft, so verify
the canonical source when evaluating another version or a request for “latest.”

## 1. Package and manifest

- Require a regular `plugin.json` at the plugin root. Its resolved path and all
  package-supplied paths must remain inside the resolved plugin root.
- Require a JSON object with canonical `$schema` and `name`.
- For 1.0.0, require
  `https://agent-plugins.org/schemas/1.0.0/plugin.schema.json`.
- Permit only `$schema`, `name`, `version`, `description`, `author`, `homepage`,
  `repository`, `license`, `keywords`, and `extensions` at the top level.
- Constrain `name` to 1–64 lowercase `a-z`, digits, hyphens, and periods; start
  and end alphanumeric; reject `--` and `..`.
- Validate optional field types. `author` may contain only string-valued
  `name`, `email`, and `url`. Semantic Versioning and SPDX are recommended,
  not required by type validation.
- Treat unknown top-level fields as reported-and-ignored schema violations,
  not fatal loader errors. Treat a non-object `extensions` similarly. Other
  manifest schema violations reject the plugin.

## 2. Discovery and skills

- Discover Skills only from immediate child directories of `skills/` that
  contain a regular file named exactly `SKILL.md`; do not recurse deeper.
- Validate every discovered skill against <https://agentskills.io/specification>.
  Skip an invalid skill without disabling valid siblings or MCP.
- Treat an absent `skills/` as valid. If present but not a directory, mark the
  Skills component type invalid while continuing with other types.
- Do not treat client-native folders or manifest entries as portable component
  discovery. Agent Plugins v1 defines exactly Skills and MCP servers.

## 3. MCP configuration

- Discover MCP only from regular root `mcp.json`. If absent, MCP is simply not
  provided. If present with the wrong filesystem kind, MCP is invalid.
- Require a JSON object containing only `$schema` and `mcpServers`, both
  required. For 1.0.0 use
  `https://agent-plugins.org/schemas/1.0.0/mcp.schema.json`.
- Require the MCP schema version to match `plugin.json`. A top-level MCP error
  disables MCP but not Skills.
- Validate each server independently as exactly one closed variant:

  - `stdio`: required string `command`; optional string-array `args`,
    string-map `env`, and string `cwd`.
  - `streamable-http` or deprecated `sse`: required absolute `url`; optional
    string-map `headers`.

- For `stdio`, keep `command` one executable token: a bare executable or a
  plugin-relative path beginning `./`. Do not expand placeholders in it.
- Allow `cwd` only as `./...`, `${PLUGIN_ROOT}[/...]`, or
  `${PLUGIN_DATA}[/...]`, with post-resolution containment.
- Expand exactly `${PLUGIN_ROOT}` and `${PLUGIN_DATA}`, once and
  non-recursively, only in `args`, `env` values, and `cwd`. Preserve unknown
  placeholder-like text literally. Forbid reserved names in configured `env`.
- For remote transports, require absolute HTTP(S), no userinfo or fragment,
  and HTTPS except exact localhost or loopback IPs. Treat header names
  case-insensitively and reject duplicates by casing.
- Never embed credentials in `env` or `headers`. Agent Plugins v1 has no
  portable OAuth or credential-reference fields.

## 4. Client extensions

- Put client-specific manifest data under `extensions` keys using stable
  reverse-domain namespaces whose values are objects.
- Put client-specific files in root directories named exactly after their
  namespaces. Either manifest data or a directory may exist independently.
- Do not assign portable semantics to extension contents. Evaluate an
  extension against its owning client's specification only when the user asks
  for that compatibility profile.

## 5. Failure boundaries and evaluation gate

| Finding | Official loading boundary | Evaluation gate |
|---|---|---|
| Missing/unreadable root manifest, unsupported schema, invalid required field, or fatal manifest schema error | Reject plugin; load no components | `FAIL` |
| Package or configured path resolves outside allowed root/data boundary | Reject/skip at the narrowest normative boundary; deny access | `FAIL` when the root manifest escapes; otherwise `PARTIAL` at the affected boundary |
| Unknown root manifest field or non-object `extensions` | Report and ignore field; continue | `PARTIAL` |
| `skills/` or `mcp.json` present with wrong filesystem kind | Disable that component type; continue | `PARTIAL` |
| Invalid individual Skill | Skip that Skill; continue | `PARTIAL` |
| Invalid MCP top level or version mismatch | Disable MCP; continue | `PARTIAL` |
| Invalid/unsupported/failing MCP server entry | Skip that server; continue | `PARTIAL` |
| Missing optional component location | No error | No downgrade |
| Confirmed embedded credential in visible package config | Unsafe to release | `FAIL` |

A secret-like field name or literal is not confirmation by itself. Keep a
heuristic match as a redacted suspicion until corroborated by a recognized live
credential format, trusted scanner, repository provenance, or the user. Do not
authenticate with a suspected value as a validation technique.

Do not promote `PARTIAL` to `FAIL` merely because one component fails. Do not
downgrade a normative MUST violation to a recommendation because a specific
client is permissive.

## 6. Primary sources

- [Agent Plugins overview](https://agent-plugins.org/)
- [Agent Plugins Specification 1.0.0](https://agent-plugins.org/specification)
- [Plugin JSON Schema 1.0.0](https://agent-plugins.org/schemas/1.0.0/plugin.schema.json)
- [MCP JSON Schema 1.0.0](https://agent-plugins.org/schemas/1.0.0/mcp.schema.json)
- [Vercel announcement](https://vercel.com/blog/introducing-agent-plugins)

The Vercel announcement is useful context for the portability goal but is not
normative. Client-specific creator tooling (e.g., OpenAI plugin-creator,
Anthropic Claude Desktop, Cursor MCP integration) is a compatibility reference,
not a replacement for the portable specification.

#!/usr/bin/env python3
"""Static Agent Plugins 1.0.0 inspector; never executes plugin code.

The scanner finds one plugin root in a repository, checks high-confidence
manifest/MCP/path rules, and emits evidence leads. Human review remains
required for Agent Skill quality, client extensions, and contextual findings.
"""

from __future__ import annotations

import argparse
import ipaddress
import json
import os
from pathlib import Path
import re
import sys
from typing import Any
from urllib.parse import urlsplit


PLUGIN_SCHEMA = "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json"
MCP_SCHEMA = "https://agent-plugins.org/schemas/1.0.0/mcp.schema.json"
PLUGIN_FIELDS = {
    "$schema", "name", "version", "description", "author", "homepage",
    "repository", "license", "keywords", "extensions",
}
NAME_RE = re.compile(r"^(?=.{1,64}$)[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$")
NAMESPACE_RE = re.compile(r"^[a-z0-9][a-z0-9-]*(?:\.[a-z0-9][a-z0-9-]*)+$")
SKILL_NAME_RE = re.compile(r"^(?=.{1,64}$)[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$")
SECRET_KEY_RE = re.compile(
    r"(?:authorization|api[-_]?key|access[-_]?token|(?:^|[-_])token(?:$|[-_])|secret|password|passwd)", re.I
)
HEADER_NAME_RE = re.compile(r"^[!#$%&'*+.^_`|~0-9A-Za-z-]+$")


class Report:
    def __init__(self, target: Path) -> None:
        self.target = str(target)
        self.root: Path | None = None
        self.findings: list[dict[str, str]] = []
        self.skills = 0
        self.mcp_servers = 0
        self.extensions: list[str] = []

    def add(self, severity: str, code: str, path: str, message: str, spec: str) -> None:
        self.findings.append({
            "severity": severity,
            "code": code,
            "path": path,
            "message": message,
            "spec": spec,
        })

    @property
    def gate(self) -> str:
        severities = {item["severity"] for item in self.findings}
        if "fatal" in severities or "security" in severities:
            return "FAIL"
        if "partial" in severities:
            return "PARTIAL"
        return "PASS"

    def payload(self) -> dict[str, Any]:
        counts: dict[str, int] = {}
        for finding in self.findings:
            key = finding["severity"]
            counts[key] = counts.get(key, 0) + 1
        return {
            "target": self.target,
            "plugin_root": str(self.root) if self.root else None,
            "spec_version": "1.0.0",
            "gate": self.gate,
            "counts": counts,
            "inventory": {
                "skills": self.skills,
                "mcp_servers": self.mcp_servers,
                "extension_namespaces": self.extensions,
            },
            "findings": self.findings,
            "notice": "Static evidence leads only; confirm in source before scoring.",
        }


def relative(path: Path, root: Path) -> str:
    try:
        return str(path.relative_to(root)) or "."
    except ValueError:
        return str(path)


def plugin_candidates(target: Path) -> list[Path]:
    if (target / "plugin.json").exists() or (target / "plugin.json").is_symlink():
        return [target]
    found: list[Path] = []
    for base, dirs, files in os.walk(target, followlinks=False):
        dirs[:] = [d for d in dirs if d not in {".git", "node_modules", ".venv", "venv"}]
        base_path = Path(base)
        if "plugin.json" in files or (base_path / "plugin.json").is_symlink():
            found.append(base_path)
    return sorted(set(found))


def load_json(path: Path, report: Report, severity: str, code: str, spec: str) -> Any | None:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (OSError, UnicodeError, json.JSONDecodeError) as exc:
        report.add(severity, code, str(path), f"Cannot parse JSON: {exc}", spec)
        return None


def contained(path: Path, root: Path) -> bool:
    try:
        path.resolve(strict=False).relative_to(root.resolve(strict=True))
        return True
    except (OSError, ValueError):
        return False


def check_symlinks(root: Path, report: Report) -> None:
    for base, dirs, files in os.walk(root, followlinks=False):
        for name in [*dirs, *files]:
            path = Path(base) / name
            if path.is_symlink() and not contained(path, root):
                report.add(
                    "partial", "PATH_ESCAPE", relative(path, root),
                    "Symlink resolves outside the plugin root.", "§4.1",
                )


def valid_name(name: Any) -> bool:
    return (
        isinstance(name, str)
        and bool(NAME_RE.fullmatch(name))
        and "--" not in name
        and ".." not in name
    )


def check_manifest(root: Path, report: Report) -> dict[str, Any] | None:
    path = root / "plugin.json"
    if not path.is_file() or not contained(path, root):
        report.add("fatal", "MANIFEST_MISSING", "plugin.json", "Root plugin.json is not a contained regular file.", "§4.1, §5.1")
        return None
    data = load_json(path, report, "fatal", "MANIFEST_JSON", "§5.2")
    if data is None:
        return None
    if not isinstance(data, dict):
        report.add("fatal", "MANIFEST_OBJECT", "plugin.json", "Manifest top level must be an object.", "§5.2")
        return None
    if data.get("$schema") != PLUGIN_SCHEMA:
        report.add("fatal", "PLUGIN_SCHEMA", "plugin.json#/$schema", f"Expected canonical Agent Plugins 1.0.0 schema {PLUGIN_SCHEMA}.", "§5.2")
    if not valid_name(data.get("name")):
        report.add("fatal", "PLUGIN_NAME", "plugin.json#/name", "Name violates the 1-64 character lowercase naming constraints.", "§5.3, §5.5")
    for key in sorted(set(data) - PLUGIN_FIELDS):
        report.add("partial", "UNKNOWN_MANIFEST_FIELD", f"plugin.json#/{key}", "Unknown top-level field is non-conforming and must be reported and ignored.", "§5.2")

    string_fields = {"version", "description", "homepage", "repository", "license"}
    for key in sorted(string_fields & set(data)):
        if not isinstance(data[key], str):
            report.add("fatal", "METADATA_TYPE", f"plugin.json#/{key}", "Metadata field must be a string.", "§5.4")
    if "keywords" in data and not (
        isinstance(data["keywords"], list)
        and all(isinstance(item, str) for item in data["keywords"])
    ):
        report.add("fatal", "KEYWORDS_TYPE", "plugin.json#/keywords", "keywords must be an array of strings.", "§5.4")
    if "author" in data:
        author = data["author"]
        if not isinstance(author, dict):
            report.add("fatal", "AUTHOR_TYPE", "plugin.json#/author", "author must be an object.", "§5.4")
        else:
            extra = set(author) - {"name", "email", "url"}
            if extra or any(not isinstance(value, str) for value in author.values()):
                report.add("fatal", "AUTHOR_FIELDS", "plugin.json#/author", "author may contain only string name, email, and url fields.", "§5.4")
    if "extensions" in data:
        extensions = data["extensions"]
        if not isinstance(extensions, dict):
            report.add("partial", "EXTENSIONS_OBJECT", "plugin.json#/extensions", "Non-object extensions is reported and ignored.", "§8.1")
        else:
            for namespace, value in extensions.items():
                report.extensions.append(namespace)
                if not NAMESPACE_RE.fullmatch(namespace):
                    report.add("partial", "EXTENSION_NAMESPACE", f"plugin.json#/extensions/{namespace}", "Extension key is not a clear reverse-domain namespace.", "§8")
                if not isinstance(value, dict):
                    report.add("partial", "EXTENSION_VALUE", f"plugin.json#/extensions/{namespace}", "Extension namespace value must be an object for a conforming package.", "§8.1")
    return data


def parse_frontmatter(path: Path) -> dict[str, str] | None:
    try:
        lines = path.read_text(encoding="utf-8").splitlines()
    except (OSError, UnicodeError):
        return None
    if not lines or lines[0].strip() != "---":
        return None
    try:
        end = next(i for i, line in enumerate(lines[1:], 1) if line.strip() == "---")
    except StopIteration:
        return None
    result: dict[str, str] = {}
    body = lines[1:end]
    i = 0
    while i < len(body):
        line = body[i]
        match = re.match(r"^([A-Za-z0-9_-]+):\s*(.*?)\s*$", line)
        if match:
            key, raw = match.groups()
            if raw in {">", "|", ">-", "|-", ">+", "|+"}:
                parts: list[str] = []
                i += 1
                while i < len(body) and (not body[i].strip() or body[i][0].isspace()):
                    parts.append(body[i].strip())
                    i += 1
                result[key] = "\n".join(parts).strip()
                continue
            result[key] = raw.strip("'\"")
        i += 1
    return result


def check_skills(root: Path, report: Report) -> None:
    skills = root / "skills"
    if not skills.exists() and not skills.is_symlink():
        return
    if not skills.is_dir() or not contained(skills, root):
        report.add("partial", "SKILLS_KIND", "skills", "Present skills path is not a contained directory.", "§6.2")
        return
    for child in sorted(skills.iterdir()):
        skill_file = child / "SKILL.md"
        if not child.is_dir() or not skill_file.is_file():
            continue
        report.skills += 1
        if not contained(skill_file, root):
            report.add("partial", "SKILL_ESCAPE", relative(skill_file, root), "Discovered SKILL.md resolves outside the plugin root.", "§4.1, §7.1")
            continue
        frontmatter = parse_frontmatter(skill_file)
        if frontmatter is None:
            report.add("partial", "SKILL_FRONTMATTER", relative(skill_file, root), "SKILL.md lacks parseable YAML frontmatter; run an Agent Skills validator.", "§7.1")
            continue
        name = frontmatter.get("name", "")
        description = frontmatter.get("description", "")
        if not name or not description:
            report.add("partial", "SKILL_REQUIRED", relative(skill_file, root), "Skill frontmatter is missing name or description.", "§7.1")
        if name and (not SKILL_NAME_RE.fullmatch(name) or "--" in name):
            report.add("partial", "SKILL_NAME_FORMAT", relative(skill_file, root), "Skill name violates Agent Skills naming constraints.", "§7.1 / Agent Skills")
        if len(description) > 1024:
            report.add("partial", "SKILL_DESCRIPTION", relative(skill_file, root), "Skill description exceeds 1024 characters.", "§7.1 / Agent Skills")
        if name and name != child.name:
            report.add("partial", "SKILL_NAME", relative(skill_file, root), "Skill frontmatter name does not match its immediate directory.", "§7.1")


def looks_like_secret(key: str, value: str) -> bool:
    if not SECRET_KEY_RE.search(key) or not value.strip():
        return False
    return "${" not in value and not re.search(r"(?:example|placeholder|replace|dummy|public)", value, re.I)


def check_stdio(name: str, server: dict[str, Any], root: Path, report: Report) -> None:
    pointer = f"mcp.json#/mcpServers/{name}"
    allowed = {"type", "command", "args", "env", "cwd"}
    if set(server) - allowed:
        report.add("partial", "MCP_STDIO_FIELDS", pointer, "stdio entry contains fields outside its closed variant.", "§7.2.1")
    command = server.get("command")
    if not isinstance(command, str) or not command:
        report.add("partial", "MCP_COMMAND", pointer + "/command", "stdio command must be a non-empty executable token.", "§7.2.1")
    elif "${" in command or (not command.startswith("./") and any(ch.isspace() for ch in command)) or command.startswith("../"):
        report.add("partial", "MCP_COMMAND_TOKEN", pointer + "/command", "command must be one bare or ./ plugin-relative token without placeholder expansion.", "§7.2.1")
    elif command.startswith("./") and not contained(root / command[2:], root):
        report.add("partial", "MCP_COMMAND_ESCAPE", pointer + "/command", "Plugin-relative command resolves outside the plugin root.", "§4.1, §7.2.1")
    args = server.get("args")
    if args is not None and not (isinstance(args, list) and all(isinstance(v, str) for v in args)):
        report.add("partial", "MCP_ARGS", pointer + "/args", "args must be an array of strings.", "§7.2.1")
    env = server.get("env")
    if env is not None:
        if not isinstance(env, dict) or not all(isinstance(k, str) and isinstance(v, str) for k, v in env.items()):
            report.add("partial", "MCP_ENV", pointer + "/env", "env must be an object of string values.", "§7.2.1")
        else:
            for key, value in env.items():
                if key in {"PLUGIN_ROOT", "PLUGIN_DATA"}:
                    report.add("partial", "MCP_RESERVED_ENV", pointer + f"/env/{key}", "Reserved plugin environment variables are client-supplied.", "§9.2")
                if looks_like_secret(key, value):
                    report.add("warning", "POSSIBLE_SECRET", pointer + f"/env/{key}", "Possible embedded credential in visible env config (value redacted); confirm before assigning FAIL.", "§9.2")
    cwd = server.get("cwd")
    if cwd is not None:
        valid_form = isinstance(cwd, str) and (
            cwd.startswith("./")
            or cwd == "${PLUGIN_ROOT}" or cwd.startswith("${PLUGIN_ROOT}/")
            or cwd == "${PLUGIN_DATA}" or cwd.startswith("${PLUGIN_DATA}/")
        )
        if not valid_form:
            report.add("partial", "MCP_CWD", pointer + "/cwd", "cwd is not a permitted plugin/data-rooted form.", "§7.2.1")
        elif cwd.startswith("./") and not contained(root / cwd[2:], root):
            report.add("partial", "MCP_CWD_ESCAPE", pointer + "/cwd", "cwd resolves outside the plugin root.", "§4.1, §7.2.1")


def is_loopback(host: str | None) -> bool:
    if host == "localhost":
        return True
    if not host:
        return False
    try:
        return ipaddress.ip_address(host).is_loopback
    except ValueError:
        return False


def check_remote(name: str, server: dict[str, Any], report: Report) -> None:
    pointer = f"mcp.json#/mcpServers/{name}"
    if set(server) - {"type", "url", "headers"}:
        report.add("partial", "MCP_REMOTE_FIELDS", pointer, "Remote entry contains fields outside its closed variant.", "§7.2.1")
    raw_url = server.get("url")
    valid = isinstance(raw_url, str)
    try:
        parsed = urlsplit(raw_url) if valid else None
        hostname = parsed.hostname if parsed else None
        userinfo = bool(parsed and (parsed.username or parsed.password))
    except ValueError:
        parsed = None
        hostname = None
        userinfo = False
    if not valid or parsed is None or parsed.scheme not in {"http", "https"} or not hostname or userinfo or parsed.fragment:
        report.add("partial", "MCP_URL", pointer + "/url", "Remote URL must be absolute HTTP(S), without userinfo or fragment.", "§7.2.1")
    elif parsed.scheme == "http" and not is_loopback(hostname):
        report.add("partial", "MCP_TLS", pointer + "/url", "Non-loopback MCP endpoint must use HTTPS.", "§7.2.1")
    headers = server.get("headers")
    if headers is not None:
        if not isinstance(headers, dict) or not all(isinstance(k, str) and isinstance(v, str) for k, v in headers.items()):
            report.add("partial", "MCP_HEADERS", pointer + "/headers", "headers must be an object of strings.", "§7.2.1")
        else:
            lowered: set[str] = set()
            for key, value in headers.items():
                if not HEADER_NAME_RE.fullmatch(key) or "\r" in value or "\n" in value:
                    report.add("partial", "MCP_HEADER_SYNTAX", pointer + f"/headers/{key}", "Header name or value is not a valid HTTP field.", "§7.2.1")
                if key.lower() in lowered:
                    report.add("partial", "MCP_HEADER_DUPLICATE", pointer + "/headers", "Header names duplicate case-insensitively.", "§7.2.1")
                lowered.add(key.lower())
                if looks_like_secret(key, value):
                    report.add("warning", "POSSIBLE_SECRET", pointer + f"/headers/{key}", "Possible embedded credential in visible headers (value redacted); confirm before assigning FAIL.", "§7.2.1")


def check_mcp(root: Path, report: Report) -> None:
    path = root / "mcp.json"
    if not path.exists() and not path.is_symlink():
        return
    if not path.is_file() or not contained(path, root):
        report.add("partial", "MCP_KIND", "mcp.json", "Present mcp.json is not a contained regular file.", "§6.2")
        return
    data = load_json(path, report, "partial", "MCP_JSON", "§7.2.2")
    if data is None:
        return
    if not isinstance(data, dict):
        report.add("partial", "MCP_OBJECT", "mcp.json", "MCP top level must be an object.", "§7.2.1")
        return
    if set(data) != {"$schema", "mcpServers"}:
        report.add("partial", "MCP_TOP_LEVEL", "mcp.json", "MCP top level must contain exactly $schema and mcpServers.", "§7.2.1")
    if data.get("$schema") != MCP_SCHEMA:
        report.add("partial", "MCP_SCHEMA", "mcp.json#/$schema", f"Expected canonical matching schema {MCP_SCHEMA}.", "§7.2.1, §10.1")
    servers = data.get("mcpServers")
    if not isinstance(servers, dict):
        report.add("partial", "MCP_SERVERS", "mcp.json#/mcpServers", "mcpServers must be an object.", "§7.2.1")
        return
    report.mcp_servers = len(servers)
    for name, server in servers.items():
        pointer = f"mcp.json#/mcpServers/{name}"
        if not isinstance(server, dict):
            report.add("partial", "MCP_SERVER_OBJECT", pointer, "Server entry must be an object.", "§7.2.1")
            continue
        server_type = server.get("type")
        if server_type == "stdio":
            check_stdio(name, server, root, report)
        elif server_type in {"streamable-http", "sse"}:
            check_remote(name, server, report)
        else:
            report.add("partial", "MCP_SERVER_TYPE", pointer + "/type", "Unknown or missing MCP transport type.", "§7.2.1")


def inspect(target: Path) -> Report:
    report = Report(target)
    if not target.is_dir():
        report.add("fatal", "TARGET", str(target), "Target is not a directory.", "Package model")
        return report
    candidates = plugin_candidates(target)
    if len(candidates) != 1:
        report.add("fatal", "PLUGIN_ROOT_AMBIGUOUS", str(target), f"Expected exactly one plugin root; found {len(candidates)}.", "§4, §5.1")
        return report
    root = candidates[0].resolve(strict=True)
    report.root = root
    check_symlinks(root, report)
    check_manifest(root, report)
    check_skills(root, report)
    check_mcp(root, report)
    return report


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("target", type=Path)
    parser.add_argument("--json", action="store_true", dest="as_json")
    args = parser.parse_args()
    report = inspect(args.target.expanduser().resolve(strict=False))
    payload = report.payload()
    if args.as_json:
        print(json.dumps(payload, indent=2, ensure_ascii=False))
    else:
        print(f"gate: {payload['gate']}")
        print(f"plugin root: {payload['plugin_root'] or 'unresolved'}")
        print(f"skills: {payload['inventory']['skills']}")
        print(f"MCP servers: {payload['inventory']['mcp_servers']}")
        for item in payload["findings"]:
            print(f"[{item['severity'].upper()}] {item['code']} {item['path']}: {item['message']} ({item['spec']})")
    return {"PASS": 0, "PARTIAL": 1, "FAIL": 2}[report.gate]


if __name__ == "__main__":
    sys.exit(main())

{
  "version": "1.0",
  "skill": "agent-plugin-eval",
  "description": "Jev questions for subjective plugin quality evaluation",
  "questions": {
    "score": [
      {
        "id": "ux_coherence",
        "axis": 4,
        "criterion": 6,
        "question": "Does the plugin provide a coherent, intuitive experience when tools are used together?",
        "context_required": ["plugin_manifest", "tool_definitions", "sample_interactions"]
      },
      {
        "id": "documentation_clarity",
        "axis": 4,
        "criterion": 8,
        "question": "Is the documentation clear, complete, and useful for both users and integrators?",
        "context_required": ["readme", "skill_instructions", "steering_files"]
      },
      {
        "id": "error_handling_quality",
        "axis": 4,
        "criterion": 9,
        "question": "Are errors handled gracefully with clear, actionable messages?",
        "context_required": ["tool_definitions", "error_examples", "validation_logic"]
      },
      {
        "id": "input_validation_robustness",
        "axis": 2,
        "criterion": 15,
        "question": "Is input validation thorough without being overly restrictive?",
        "context_required": ["schema_definitions", "validation_rules"]
      },
      {
        "id": "schema_design_quality",
        "axis": 2,
        "criterion": 16,
        "question": "Are the JSON schemas well-designed with appropriate constraints and descriptions?",
        "context_required": ["tool_schemas", "type_definitions"]
      },
      {
        "id": "naming_consistency",
        "axis": 2,
        "criterion": 17,
        "question": "Are naming conventions consistent and semantically meaningful across tools?",
        "context_required": ["tool_names", "parameter_names", "resource_names"]
      },
      {
        "id": "api_design_elegance",
        "axis": 2,
        "criterion": 18,
        "question": "Is the API design elegant, avoiding unnecessary complexity?",
        "context_required": ["tool_definitions", "workflow_patterns"]
      }
    ],
    "noul": [
      {
        "id": "gate_manifest_structure",
        "gate": "G1",
        "question": "Does the manifest follow valid JSON structure with required fields (name, version)?",
        "labels": ["conformant", "malformed"],
        "context_required": ["plugin_manifest_raw"]
      },
      {
        "id": "gate_schema_validity",
        "gate": "G2",
        "question": "Are all tool schemas valid JSON Schema with proper type definitions?",
        "labels": ["valid", "invalid"],
        "context_required": ["tool_schemas_raw"]
      },
      {
        "id": "gate_skill_coherence",
        "gate": "G3",
        "question": "Do bundled skills have coherent instructions that match tool capabilities?",
        "labels": ["coherent", "incoherent"],
        "context_required": ["skill_files", "tool_definitions"]
      },
      {
        "id": "gate_transport_config",
        "gate": "G4",
        "question": "Is the transport configuration (stdio/http) properly specified and consistent?",
        "labels": ["valid", "misconfigured"],
        "context_required": ["transport_config", "server_entry"]
      },
      {
        "id": "secret_env_vars",
        "category": "secrets",
        "question": "Do environment variables appear to contain hardcoded secrets rather than references?",
        "labels": ["suspected", "not_suspected"],
        "context_required": ["env_definitions", "config_files"]
      },
      {
        "id": "secret_in_code",
        "category": "secrets",
        "question": "Does the source code contain patterns suggesting embedded credentials?",
        "labels": ["suspected", "not_suspected"],
        "context_required": ["source_files_sample"]
      },
      {
        "id": "secret_in_docs",
        "category": "secrets",
        "question": "Do documentation or example files contain what appear to be real credentials?",
        "labels": ["suspected", "not_suspected"],
        "context_required": ["readme", "examples"]
      },
      {
        "id": "secret_exposed_endpoint",
        "category": "secrets",
        "question": "Are there hardcoded URLs or endpoints that might expose internal systems?",
        "labels": ["suspected", "not_suspected"],
        "context_required": ["source_files_sample", "config_files"]
      },
      {
        "id": "checklist_readme_exists",
        "category": "quality_checklist",
        "question": "Is there a README with installation and usage instructions?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing"]
      },
      {
        "id": "checklist_license",
        "category": "quality_checklist",
        "question": "Is there a LICENSE file with a recognized open source license?",
        "labels": ["present", "missing_or_unclear"],
        "context_required": ["file_listing", "license_content"]
      },
      {
        "id": "checklist_changelog",
        "category": "quality_checklist",
        "question": "Is there a CHANGELOG or version history?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing"]
      },
      {
        "id": "checklist_tests",
        "category": "quality_checklist",
        "question": "Are there automated tests for the plugin functionality?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing", "test_files"]
      },
      {
        "id": "checklist_ci",
        "category": "quality_checklist",
        "question": "Is there CI/CD configuration for automated testing?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing", "ci_config"]
      },
      {
        "id": "checklist_examples",
        "category": "quality_checklist",
        "question": "Are there usage examples or sample interactions?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing", "examples_dir"]
      },
      {
        "id": "checklist_security_policy",
        "category": "quality_checklist",
        "question": "Is there a SECURITY.md or security policy?",
        "labels": ["present", "missing"],
        "context_required": ["file_listing"]
      },
      {
        "id": "component_plugin_json",
        "category": "component_validity",
        "question": "Is plugin.json present and parseable?",
        "labels": ["valid", "invalid", "missing"],
        "context_required": ["plugin_json_raw"]
      },
      {
        "id": "component_mcp_json",
        "category": "component_validity",
        "question": "Is mcp.json present and properly structured?",
        "labels": ["valid", "invalid", "missing"],
        "context_required": ["mcp_json_raw"]
      },
      {
        "id": "component_skill_md",
        "category": "component_validity",
        "question": "Are SKILL.md files present with proper frontmatter?",
        "labels": ["valid", "invalid", "missing"],
        "context_required": ["skill_files_raw"]
      },
      {
        "id": "component_steering",
        "category": "component_validity",
        "question": "Are steering files properly formatted with valid YAML frontmatter?",
        "labels": ["valid", "invalid", "not_applicable"],
        "context_required": ["steering_files_raw"]
      }
    ]
  },
  "usage": {
    "score_aggregation": "For Axis 2 and 4 quality scores, average Jev Score responses (0.0-1.0) and scale to rubric (0-25 per axis)",
    "gate_evaluation": "Gates pass if Jev Noul returns positive label (conformant/valid/coherent)",
    "secret_detection": "Flag for human review if any secret question returns suspected",
    "checklist_scoring": "Count present labels, report as X/7 completeness"
  }
}

#!/usr/bin/env python3
"""Compute an Agent Plugin evaluation score with a conformance gate.

Usage:
    score.py [--gate pass|partial|fail] 1:80:3 2:65:3 ... 8:NA:2

Each criterion is ID:SCORE:WEIGHT. NA or N/A excludes a criterion from both
sums. PARTIAL caps the final score at 59; FAIL caps it at 39.
"""

from __future__ import annotations

import argparse
import sys


CAPS = {"pass": None, "partial": 59.0, "fail": 39.0}


def grade(score: float) -> str:
    if score >= 80:
        return "A"
    if score >= 60:
        return "B"
    if score >= 40:
        return "C"
    if score >= 20:
        return "D"
    return "F"


def parse_triple(value: str) -> tuple[str, float | None, float]:
    try:
        criterion, score_text, weight_text = value.split(":")
    except ValueError as exc:
        raise argparse.ArgumentTypeError(
            f"{value!r}: expected criterion:score:weight"
        ) from exc
    if not criterion:
        raise argparse.ArgumentTypeError("criterion ID cannot be empty")
    try:
        weight = float(weight_text)
    except ValueError as exc:
        raise argparse.ArgumentTypeError(f"{value!r}: weight must be numeric") from exc
    if weight <= 0:
        raise argparse.ArgumentTypeError(f"{value!r}: weight must be positive")
    if score_text.strip().upper() in {"NA", "N/A"}:
        return criterion, None, weight
    try:
        score = float(score_text)
    except ValueError as exc:
        raise argparse.ArgumentTypeError(f"{value!r}: score must be 0-100 or NA") from exc
    if not 0 <= score <= 100:
        raise argparse.ArgumentTypeError(f"{value!r}: score outside 0-100")
    return criterion, score, weight


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--gate", choices=CAPS, default="pass")
    parser.add_argument("criteria", nargs="+", type=parse_triple)
    args = parser.parse_args()

    numerator = 0.0
    denominator = 0.0
    excluded: list[str] = []
    seen: set[str] = set()
    for criterion, score, weight in args.criteria:
        if criterion in seen:
            parser.error(f"duplicate criterion ID: {criterion}")
        seen.add(criterion)
        if score is None:
            excluded.append(criterion)
            continue
        numerator += score * weight
        denominator += weight

    if denominator == 0:
        parser.error("no applicable criteria")

    raw = numerator / denominator
    cap = CAPS[args.gate]
    final = min(raw, cap) if cap is not None else raw

    print(f"gate = {args.gate.upper()}")
    print(f"applicable criteria = {len(args.criteria) - len(excluded)}")
    print(f"N/A = {', '.join(excluded) or 'none'}")
    print(f"sum(score x weight) = {numerator:g}")
    print(f"sum(weight) = {denominator:g}")
    print(f"raw score = {raw:.2f}")
    if cap is None:
        print("cap = none")
    else:
        print(f"cap = {cap:g} ({args.gate.upper()} gate)")
    print(f"final score = {final:.2f}")
    print(f"grade = {grade(final)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())