Claude Cursor Skill

llm-evaluation

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

LLM Mart · 0 points · 16 views 0 listing impressions 0 install-command copies
Download wshobson-agents-plugins_llm-application-dev_skills_llm-evaluation-554237f.zip · 6 KB

Install

skills CLI npx skills add https://github.com/wshobson/agents/tree/main/plugins/llm-application-dev/skills/llm-evaluation
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install wshobson-agents@llmmart
Git git clone https://github.com/wshobson/agents.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole wshobson/agents collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

When to Use This Skill

  • Measuring LLM application performance systematically
  • Comparing different models or prompts
  • Detecting performance regressions before deployment
  • Validating improvements from prompt changes
  • Building confidence in production systems
  • Establishing baselines and tracking progress over time
  • Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

Text Generation:

  • BLEU: N-gram overlap (translation)
  • ROUGE: Recall-oriented (summarization)
  • METEOR: Semantic similarity
  • BERTScore: Embedding-based similarity
  • Perplexity: Language model confidence

Classification:

  • Accuracy: Percentage correct
  • Precision/Recall/F1: Class-specific performance
  • Confusion Matrix: Error patterns
  • AUC-ROC: Ranking quality

Retrieval (RAG):

  • MRR: Mean Reciprocal Rank
  • NDCG: Normalized Discounted Cumulative Gain
  • Precision@K: Relevant in top K
  • Recall@K: Coverage in top K

2. Human Evaluation

Manual assessment for quality aspects difficult to automate.

Dimensions:

  • Accuracy: Factual correctness
  • Coherence: Logical flow
  • Relevance: Answers the question
  • Fluency: Natural language quality
  • Safety: No harmful content
  • Helpfulness: Useful to the user

3. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

Approaches:

  • Pointwise: Score individual responses
  • Pairwise: Compare two responses
  • Reference-based: Compare to gold standard
  • Reference-free: Judge without ground truth

Quick Start

from dataclasses import dataclass
from typing import Callable
import numpy as np

@dataclass
class Metric:
    name: str
    fn: Callable

    @staticmethod
    def accuracy():
        return Metric("accuracy", calculate_accuracy)

    @staticmethod
    def bleu():
        return Metric("bleu", calculate_bleu)

    @staticmethod
    def bertscore():
        return Metric("bertscore", calculate_bertscore)

    @staticmethod
    def custom(name: str, fn: Callable):
        return Metric(name, fn)

class EvaluationSuite:
    def __init__(self, metrics: list[Metric]):
        self.metrics = metrics

    async def evaluate(self, model, test_cases: list[dict]) -> dict:
        results = {m.name: [] for m in self.metrics}

        for test in test_cases:
            prediction = await model.predict(test["input"])

            for metric in self.metrics:
                score = metric.fn(
                    prediction=prediction,
                    reference=test.get("expected"),
                    context=test.get("context")
                )
                results[metric.name].append(score)

        return {
            "metrics": {k: np.mean(v) for k, v in results.items()},
            "raw_scores": results
        }

# Usage
suite = EvaluationSuite([
    Metric.accuracy(),
    Metric.bleu(),
    Metric.bertscore(),
    Metric.custom("groundedness", check_groundedness)
])

test_cases = [
    {
        "input": "What is the capital of France?",
        "expected": "Paris",
        "context": "France is a country in Europe. Paris is its capital."
    },
]

results = await suite.evaluate(model=your_model, test_cases=test_cases)

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Files (agents)
  • references
    • details.md 13.7 KB
      Loading…
  • SKILL.md 3.8 KB
    ---
    name: llm-evaluation
    description: Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
    ---
    
    # LLM Evaluation
    
    Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
    
    ## When to Use This Skill
    
    - Measuring LLM application performance systematically
    - Comparing different models or prompts
    - Detecting performance regressions before deployment
    - Validating improvements from prompt changes
    - Building confidence in production systems
    - Establishing baselines and tracking progress over time
    - Debugging unexpected model behavior
    
    ## Core Evaluation Types
    
    ### 1. Automated Metrics
    
    Fast, repeatable, scalable evaluation using computed scores.
    
    **Text Generation:**
    
    - **BLEU**: N-gram overlap (translation)
    - **ROUGE**: Recall-oriented (summarization)
    - **METEOR**: Semantic similarity
    - **BERTScore**: Embedding-based similarity
    - **Perplexity**: Language model confidence
    
    **Classification:**
    
    - **Accuracy**: Percentage correct
    - **Precision/Recall/F1**: Class-specific performance
    - **Confusion Matrix**: Error patterns
    - **AUC-ROC**: Ranking quality
    
    **Retrieval (RAG):**
    
    - **MRR**: Mean Reciprocal Rank
    - **NDCG**: Normalized Discounted Cumulative Gain
    - **Precision@K**: Relevant in top K
    - **Recall@K**: Coverage in top K
    
    ### 2. Human Evaluation
    
    Manual assessment for quality aspects difficult to automate.
    
    **Dimensions:**
    
    - **Accuracy**: Factual correctness
    - **Coherence**: Logical flow
    - **Relevance**: Answers the question
    - **Fluency**: Natural language quality
    - **Safety**: No harmful content
    - **Helpfulness**: Useful to the user
    
    ### 3. LLM-as-Judge
    
    Use stronger LLMs to evaluate weaker model outputs.
    
    **Approaches:**
    
    - **Pointwise**: Score individual responses
    - **Pairwise**: Compare two responses
    - **Reference-based**: Compare to gold standard
    - **Reference-free**: Judge without ground truth
    
    ## Quick Start
    
    ```python
    from dataclasses import dataclass
    from typing import Callable
    import numpy as np
    
    @dataclass
    class Metric:
        name: str
        fn: Callable
    
        @staticmethod
        def accuracy():
            return Metric("accuracy", calculate_accuracy)
    
        @staticmethod
        def bleu():
            return Metric("bleu", calculate_bleu)
    
        @staticmethod
        def bertscore():
            return Metric("bertscore", calculate_bertscore)
    
        @staticmethod
        def custom(name: str, fn: Callable):
            return Metric(name, fn)
    
    class EvaluationSuite:
        def __init__(self, metrics: list[Metric]):
            self.metrics = metrics
    
        async def evaluate(self, model, test_cases: list[dict]) -> dict:
            results = {m.name: [] for m in self.metrics}
    
            for test in test_cases:
                prediction = await model.predict(test["input"])
    
                for metric in self.metrics:
                    score = metric.fn(
                        prediction=prediction,
                        reference=test.get("expected"),
                        context=test.get("context")
                    )
                    results[metric.name].append(score)
    
            return {
                "metrics": {k: np.mean(v) for k, v in results.items()},
                "raw_scores": results
            }
    
    # Usage
    suite = EvaluationSuite([
        Metric.accuracy(),
        Metric.bleu(),
        Metric.bertscore(),
        Metric.custom("groundedness", check_groundedness)
    ])
    
    test_cases = [
        {
            "input": "What is the capital of France?",
            "expected": "Paris",
            "context": "France is a country in Europe. Paris is its capital."
        },
    ]
    
    results = await suite.evaluate(model=your_model, test_cases=test_cases)
    ```
    
    ## Detailed patterns and worked examples
    
    Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
    
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.