Claude Skill

A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

LLM Mart · 0 points · 0 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download agentskillexchange-skills-skills_ab-test-agent-skills-mcp-changes-with-caliper-07beb56.zip · 0 KB

agentskillexchange/skills 45 57 forks MIT Updated 11h ago

Part of agentskillexchange/skills — 249 skills

Install

skills CLI

npx skills add https://github.com/agentskillexchange/skills/tree/main/skills/ab-test-agent-skills-mcp-changes-with-caliper

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install agentskillexchange-skills@llmmart

Git git clone https://github.com/agentskillexchange/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole agentskillexchange/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

Prerequisites

Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test

Installation

Install or set up from the source-backed instructions:

Install the CLI with pipx install caliper-eval for direct runs, or add the agent skill with npx skills@latest add edonadei/caliper; then create an .eval.yaml, run caliper run ... --k 3, run an ablated control with --ablate, and compare results with caliper compare.

Source: https://github.com/edonadei/caliper

Documentation

https://github.com/edonadei/caliper

Source

Agent Skill Exchange

Files (skills)

SKILL.md 1.4 KB

---
name: "A/B test agent skills and MCP changes with Caliper"
slug: "ab-test-agent-skills-mcp-changes-with-caliper"
description: "Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable."
github_stars: 175
verification: "security_reviewed"
source: "https://github.com/edonadei/caliper"
author: "edonadei"
publisher_type: "individual"
category: "Security & Verification"
framework: "Multi-Framework"
tool_ecosystem:
  github_repo: "edonadei/caliper"
  github_stars: 175
---

# A/B test agent skills and MCP changes with Caliper

Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.

## Prerequisites

Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test

## Installation

Install or set up from the source-backed instructions:

Install the CLI with `pipx install caliper-eval` for direct runs, or add the agent skill with `npx skills@latest add edonadei/caliper`; then create an `.eval.yaml`, run `caliper run ... --k 3`, run an ablated control with `--ablate`, and compare results with `caliper compare`.

- Source: https://github.com/edonadei/caliper

## Documentation

- https://github.com/edonadei/caliper

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/ab-test-agent-skills-mcp-changes-with-caliper/)