Benchwarmers

Ask a collection of LLMs at once through your Benchwarmers server; verdicts pick the next panel.

LLM Mart 0 views 44 listing impressions

Benchwarmers is a self-hosted panel gateway for LLMs. Your app asks for one of your collections, and the whole panel answers: the models you picked for that kind of work, plus explorer seats that rotate in the models you have used least. Every answer comes back side by side, labelled by model and seat. You check them against the real work and record a verdict, and those verdicts decide who sits on the next panel.

What it is best at:

  • Asking several models at once. One request, in the OpenAI, Anthropic, Gemini or Ollama format, a panel of answers. Where they disagree is where to look.
  • Agents too. Claude Code and other tool-calling clients use a collection through route mode: one member per turn, kept for the conversation, with failover, and every call on the record.
  • Reviewing. Code review, claims about code, designs: different models catch different things, and two models agreeing is not proof.
  • Playing to each model's strengths. A collection per kind of work (reviewers, designers, enumerators), each with the models that do it best.
  • Giving every model playing time. Explorers cycle through the rest of the bench, so new and free models get tried beside the ones you trust, without you choosing them by hand.
  • Benchmarking on your own projects. A review suite seeded from your own repositories, plus your verdicts on real work, instead of public leaderboards.
  • Knowing who to bench. Who is confidently wrong, who has stopped answering (retired automatically), and who is resting after a rate limit.
  • Keeping free and cheap models useful. Health probes, rate limits taken from what providers say and enforced, and one endpoint for every app, over any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
Related tools Examples What they do What Benchwarmers does differently
Gateways and routers LiteLLM, OpenRouter, Portkey One request to one model, with fallback Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record
Councils and ensembles LLM Council, Mixture-of-Agents Several models merged into one answer Keeps every answer, because who said what is the evidence; one merged summary only if you ask
Side-by-side chat Open WebUI, LMArena Several answers shown together Records a verdict per answer and uses them to pick the next panel
Evaluation harnesses promptfoo, Inspect, SWE-bench Score models offline on fixed test sets Scores them on your real work and your own code, and keeps rotating new models in

Contents

Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is in USAGE.md. Running it on a server or a hosting platform is in HOSTING.md. A coding agent setting it up for someone should start with AGENTS.md.

Try it

From the project's README.

Related servers

Let LLM help you achieve your regression analysis with Stata

21 views

Web search, deep research, prediction markets & crypto data for AI agents. Pay per call via x402.

21 views

vSphere with Tanzu (VKS): Namespace and TanzuKubernetesCluster lifecycle. Requires vSphere 8.x+.

20 views

Durable agentic memory, encrypted at rest. Fully offline: no network, no API key, no cloud.

19 views