Mcp Apache Spark History Server
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
Connect AI agents and engineers to Apache Spark History Server for intelligent job analysis, performance monitoring, and investigation
Important
✨ NEW — Spark History Server CLI is now available
A standalone Go binary that queries Spark History Server directly from your terminal — no MCP, no AI framework, no daemon process. Inspect jobs, compare runs, investigate failures, and script against the Spark REST API.
This project provides two interfaces to your Spark History Server data:
🛠️ SHS CLI (shs) |
⚡ MCP Server | |
|---|---|---|
| For | Engineers, shell scripts, CI/CD, coding agents | AI agents and MCP-compatible clients |
| Mental model | "I know the command I want to run" | "Agent, investigate this Spark app" |
| Install | Single static binary — no dependencies | Python 3.12+, uv |
| Get started | CLI docs → | MCP docs → |
🛠️ SHS CLI (shs) — For Engineers & Scripts
A standalone Go binary. Query your Spark History Server directly from the terminal, shell scripts, or CI/CD pipelines. Also works as a skill for coding agents like Claude Code and Kiro.
Install
# Auto-detect latest version, OS, and architecture
VERSION=$(curl -s https://api.github.com/repos/kubeflow/mcp-apache-spark-history-server/releases | grep -m1 '"tag_name": "cli/' | cut -d'"' -f4 | sed 's|cli/||')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
[ "$ARCH" = "x86_64" ] && ARCH="amd64"
[ "$ARCH" = "aarch64" ] && ARCH="arm64"
curl -sSL "https://github.com/kubeflow/mcp-apache-spark-history-server/releases/download/cli%2F${VERSION}/shs-${VERSION}-${OS}-${ARCH}.tar.gz" | tar xz
sudo mv shs /usr/local/bin/
Quick Start
# Generate a config file
shs setup config > config.yaml # then set your Spark History Server URL
# Explore applications
shs apps
shs jobs -a APP_ID --status failed
shs stages -a APP_ID --sort duration
shs compare apps --app-a APP1 --app-b APP2
# Use as a skill with Claude Code or Kiro
shs setup skill > ~/.claude/skills/spark-history.md
CLI documentation for full usage, or check out a real-world example of Claude Code comparing two TPC-DS 3TB benchmark runs.
⚡ MCP Server — For AI Agents
An MCP (Model Context Protocol) server that exposes Spark History Server data as tools for AI agents. Agents query your Spark infrastructure using natural language — the server handles tool selection, multi-server routing, and structured data retrieval.
Use the MCP server when you want an AI agent to conduct multi-step investigations, synthesize findings across tools, or answer natural-language questions about your Spark applications.
Install
# Run directly with uvx (no install needed)
uvx --from mcp-apache-spark-history-server spark-mcp
# Or install with pip
uv tool install mcp-apache-spark-history-server
spark-mcp
The package is published to PyPI.
Coding Agent Integration
Register the server with a single command. Both examples run it over stdio via uvx.
With no config file present, the server defaults to a Spark History Server at http://localhost:18080;
point it elsewhere with a config file or SHS_SERVERS__LOCAL__URL.
Claude Code (claude mcp add):
From the project's README.