Mcp Apache Spark History Server

MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

LLM Mart 1 views 220 listing impressions

Connect AI agents and engineers to Apache Spark History Server for intelligent job analysis, performance monitoring, and investigation


Important

✨ NEW — Spark History Server CLI is now available

SHS CLI

A standalone Go binary that queries Spark History Server directly from your terminal — no MCP, no AI framework, no daemon process. Inspect jobs, compare runs, investigate failures, and script against the Spark REST API.

Get started with the SHS CLI →


This project provides two interfaces to your Spark History Server data:

🛠️ SHS CLI (shs) ⚡ MCP Server
For Engineers, shell scripts, CI/CD, coding agents AI agents and MCP-compatible clients
Mental model "I know the command I want to run" "Agent, investigate this Spark app"
Install Single static binary — no dependencies Python 3.12+, uv
Get started CLI docs → MCP docs →

📺 See it in action: Watch the demo video


🛠️ SHS CLI (shs) — For Engineers & Scripts

A standalone Go binary. Query your Spark History Server directly from the terminal, shell scripts, or CI/CD pipelines. Also works as a skill for coding agents like Claude Code and Kiro.

Install

# Auto-detect latest version, OS, and architecture
VERSION=$(curl -s https://api.github.com/repos/kubeflow/mcp-apache-spark-history-server/releases | grep -m1 '"tag_name": "cli/' | cut -d'"' -f4 | sed 's|cli/||')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
[ "$ARCH" = "x86_64" ] && ARCH="amd64"
[ "$ARCH" = "aarch64" ] && ARCH="arm64"

curl -sSL "https://github.com/kubeflow/mcp-apache-spark-history-server/releases/download/cli%2F${VERSION}/shs-${VERSION}-${OS}-${ARCH}.tar.gz" | tar xz
sudo mv shs /usr/local/bin/

Quick Start

# Generate a config file
shs setup config > config.yaml   # then set your Spark History Server URL

# Explore applications
shs apps
shs jobs -a APP_ID --status failed
shs stages -a APP_ID --sort duration
shs compare apps --app-a APP1 --app-b APP2

# Use as a skill with Claude Code or Kiro
shs setup skill > ~/.claude/skills/spark-history.md

CLI documentation for full usage, or check out a real-world example of Claude Code comparing two TPC-DS 3TB benchmark runs.


⚡ MCP Server — For AI Agents

An MCP (Model Context Protocol) server that exposes Spark History Server data as tools for AI agents. Agents query your Spark infrastructure using natural language — the server handles tool selection, multi-server routing, and structured data retrieval.

Use the MCP server when you want an AI agent to conduct multi-step investigations, synthesize findings across tools, or answer natural-language questions about your Spark applications.

Install

# Run directly with uvx (no install needed)
uvx --from mcp-apache-spark-history-server spark-mcp

# Or install with pip
uv tool install mcp-apache-spark-history-server
spark-mcp

The package is published to PyPI.

Coding Agent Integration

Register the server with a single command. Both examples run it over stdio via uvx. With no config file present, the server defaults to a Spark History Server at http://localhost:18080; point it elsewhere with a config file or SHS_SERVERS__LOCAL__URL.

Claude Code (claude mcp add):

From the project's README.

Related servers

MimiOps is an opinionated MCP server for Kubernetes

28 views

Read and write RAGE Package Format archives

23 views

vSphere with Tanzu (VKS): Namespace and TanzuKubernetesCluster lifecycle. Requires vSphere 8.x+.

20 views

MCP server for Geargrafx PC Engine / TurboGrafx-16 emulator

19 views