{
  "schema_version": 1,
  "skill_name": "vllm",
  "evals": [
    {
      "id": "serving-config-review",
      "prompt": "A team wants to serve a 70B instruct model (bf16) on two A100 80GB GPUs in one node with a 32K context window. Draft the vllm serve command with serving flags, explain the memory and parallelism reasoning, and state what to verify after startup.",
      "expected_output": "A vllm serve command that pins the model and revision, serves a stable API name, and configures parallelism and memory deliberately: --model <repo> --revision <sha> --served-model-name <api-name> --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.9 --trust-remote-code only if the model card requires it. The reasoning explains that a 70B bf16 checkpoint needs roughly 140GB of weights, so two 80GB GPUs with tensor parallelism are required (weights plus KV cache plus activations must fit under the per-instance gpu-memory-utilization cap), that --pipeline-parallel-size would only be needed across nodes, that --max-model-len 32K directly sizes the KV cache (larger context means fewer concurrent sequences), and that TP requires NVLink and equal per-GPU memory. Post-startup verification: the startup log shows memory profiling completed and the GPU KV cache size, /v1/models lists the served name, and a representative chat request returns tokens. The record goes into the serving config template so the deployment is reproducible and rollback is a redeploy of the same pinned image and arguments.",
      "assertions": [
        "The command pins model, revision, served-model-name, tensor-parallel-size 2, max-model-len, and gpu-memory-utilization",
        "The 70B bf16 weight size (~140GB) is used to justify two A100 80GB GPUs under the memory-utilization cap",
        "max-model-len is tied to KV cache sizing and concurrent-sequence capacity",
        "Verification includes startup memory profiling, /v1/models, and a representative chat request",
        "The serving config template is used so the deployment is reproducible and rollback is a pinned-image redeploy"
      ]
    },
    {
      "id": "quantization-and-parallelism-sizing",
      "prompt": "Our 8B model in bf16 takes 16GB of VRAM and we want to serve 256 concurrent long-context requests. We have a single 80GB H100. Should we quantize, enable tensor parallelism, or both, and what trade-offs should we record before changing anything?",
      "expected_output": "A sizing analysis that measures before optimizing: establish the baseline with the current bf16 serving config, record the reported GPU KV cache size and max concurrency for the target context, then evaluate options against that baseline. On a single H100, tensor parallelism is not applicable (TP shards across multiple GPUs) so the levers are quantization and KV cache settings: an FP8 or INT4 weight format shrinks weights and frees GPU memory for a larger KV cache, --kv-cache-dtype fp8 shrinks the cache for long contexts on supported hardware, and --max-model-len or --max-num-seqs bound concurrency. The trade-offs are stated explicitly: quantization changes output quality and requires kernels that match the hardware generation, and more KV cache (longer context or higher concurrency) increases gpu-cache-usage until requests wait. The change record freezes version, model, quantization, context, batching, and workload, and verification re-runs the frozen benchmark plus a quality spot-check rather than assuming the quantized model matches bf16 behavior.",
      "assertions": [
        "The response measures the baseline (KV cache size, max concurrency) before optimizing",
        "Tensor parallelism is correctly deemed not applicable on a single GPU",
        "Quantization (FP8/INT4 weights) and kv-cache-dtype fp8 are evaluated as the memory levers, with quality trade-offs stated",
        "KV cache pressure is tied to gpu-cache-usage and waiting requests",
        "Verification re-runs the frozen benchmark and a quality spot-check under matched conditions"
      ]
    },
    {
      "id": "openai-api-integration",
      "prompt": "We started vllm serve with a chat model and the OpenAI SDK returns a 404 for /v1/chat/completions, while /v1/models works. Diagnose the likely causes and describe how to verify the fix, without exposing the server.",
      "expected_output": "A diagnostic sequence over the read-only probes: confirm /health and /v1/models respond, then check what /v1/models actually lists and compare it with the model name the client sends, since a served-model-name mismatch or an unserved model id produces 404-style failures on inference routes. The next checks are version and load state, and the server log for whether the model loaded and which endpoints are registered on this release, because vLLM serves /v1/chat/completions only for text-generation models that carry a chat template; a model without a chat template cannot serve chat and needs --chat-template, and some pooling or embedding models do not register chat endpoints at all. The fix is verified by sending one bounded chat request with the exact served model name and checking for generated tokens, and by confirming the server is only reachable on the intended interface with API auth or a reverse proxy in place. The response does not expose logs or tokens and treats any bind-address or API-key change as a confirmed mutation.",
      "assertions": [
        "Diagnosis starts with /health, /v1/models, and the served model name versus the client's model field",
        "Chat templates and model type (generative versus pooling) are named as the cause of missing /v1/chat/completions routes",
        "The fix is verified with one bounded chat request returning generated tokens",
        "Exposure controls (bind address, API keys, reverse proxy) are checked without dumping logs or secrets",
        "A bind or auth change is treated as a mutation requiring confirmation"
      ]
    },
    {
      "id": "benchmark-run-review",
      "prompt": "A colleague claims a vLLM upgrade doubled our token throughput: 420 tok/s before, 850 tok/s after. Review their evidence before we roll the upgrade to production.",
      "expected_output": "A skeptical review of the benchmark evidence: the comparison is only meaningful if every other condition was frozen, so the reviewer asks for or reconstructs the matched conditions — pinned vLLM versions on both sides, identical model and revision, same quantization and dtype, same tensor parallelism, max-model-len, max-num-seqs, max-num-batched-tokens, GPU model and driver, dataset, prompt and output lengths, request rate, and concurrency — and requires the raw benchmark records from vllm bench serve showing request throughput, output token throughput, TTFT/TPOT/ITL percentiles, and variance across repetitions. A throughput doubling without changes to hardware or workload is a red flag for a mismatch (for example comparing different concurrency, a different dataset, or fp8 versus bf16). The reviewer also separates throughput from latency: if the goal is user-facing latency, TPOT and TTFT under realistic load matter more than raw tok/s. The upgrade only ships if the matched benchmark confirms the claim and the run record template was used on both sides.",
      "assertions": [
        "The review demands matched conditions (version, model, quant, parallelism, context, batching, GPU, workload) on both sides",
        "Raw vllm bench serve output with throughput, TTFT/TPOT/ITL percentiles, and variance is required, not a single number",
        "A suspicious doubling is probed for a mismatch in concurrency, dataset, or precision",
        "Throughput and latency (TTFT/TPOT under load) are treated as separate metrics",
        "The benchmark run record template is required on both sides before the upgrade ships"
      ]
    },
    {
      "id": "continuous-batching-tuning",
      "prompt": "Our vLLM server reports gpu_cache_usage_perc above 0.95 with requests waiting, and p50 latency is fine but p99 spiked. We already run with default batching settings. What should we change, and what should we measure?",
      "expected_output": "A one-variable-at-a-time tuning plan anchored on evidence: the cache-usage and waiting-request metrics say the deployment is at KV cache capacity, so the options are to bound concurrency (lower --max-num-seqs to cap sequences per iteration), reduce per-request token demand (lower --max-model-len or cap output tokens), enable or widen chunked prefill (--enable-chunked-prefill with a deliberate --max-num-batched-tokens) so prefill no longer blocks decode, or scale out, rather than raising --max-num-seqs further which would worsen p99. Prefix caching (--enable-prefix-caching) is evaluated if requests share system prompts, since a higher hit rate lowers effective input tokens. Every change is measured with the frozen benchmark plus the /metrics counters (gpu_cache_usage_perc, num_requests_running/waiting) before and after, one change at a time, and p99 is reported with its distribution rather than a single run.",
      "assertions": [
        "The plan ties gpu_cache_usage_perc near 1.0 with waiting requests to KV cache capacity, not to raising max-num-seqs",
        "Concrete levers are named: max-num-seqs, max-model-len or output cap, chunked prefill with max-num-batched-tokens, or scale-out",
        "Prefix caching is evaluated for shared-prefix workloads",
        "Changes are made one at a time with the frozen benchmark and /metrics counters before and after",
        "p99 is reported with distribution or variance, not a single run"
      ]
    },
    {
      "id": "upgrade-rollback-troubleshooting",
      "prompt": "We upgraded the vllm/vllm-openai image from :v0.25.1 to :v0.26.0 and the server now crashes at startup with a CUDA out of memory error, even though the model did not change. Diagnose and decide: fix forward or roll back?",
      "expected_output": "A bounded diagnosis that starts with the read-only evidence: confirm the exact image digest on both versions, the serving args, the model revision, and the GPU inventory (nvidia-smi), then compare the startup logs of the failing v0.26.0 run against the v0.25.1 run. OOM at startup after an engine upgrade with the same model points at changed memory defaults or changed flag semantics: vLLM releases change default KV cache allocation, max-model-len derivation, CUDA graph capture, and flag names, so the same config may now reserve more memory; the fix-forward path re-reads the v0.26.0 release notes for changed defaults, lowers gpu-memory-utilization or max-model-len on the scratch instance, validates the model loads and passes the frozen benchmark, and only then redeploys. The rollback path is already cheap because the previous pinned image and serving config were recorded: redeploy vllm/vllm-openai:v0.25.1 with the old args, verify /health, /v1/models, and one representative request, and re-run the benchmark. The response decides rollback first if the team cannot validate forward in the window, and never mutates the production deployment without a human directive naming target, scope, and rollback path.",
      "assertions": [
        "Diagnosis uses exact image digests, serving args, model revision, nvidia-smi, and side-by-side startup logs",
        "Changed memory defaults and flag semantics across releases are named as the leading OOM-after-upgrade cause",
        "The fix-forward path validates on a scratch instance with release notes and the frozen benchmark before redeploying",
        "The rollback path redeploys the pinned previous image plus recorded config and verifies at the delivery boundary",
        "No production mutation happens without a human directive naming target, scope, and rollback path"
      ]
    }
  ]
}

# vLLM Operations — Source Index

> **Last Updated:** 2026-08-03

This index tracks the authoritative upstream sources behind the vLLM operational
skill and the refresh procedure for keeping it current. vLLM moves fast — flags,
defaults, and endpoint behavior change between releases; treat any claim here as
version-sensitive and re-verify against the installed release.

## Canonical sources

| Topic | Source |
|---|---|
| vLLM documentation (latest) | https://docs.vllm.ai/en/latest/ |
| vLLM documentation (stable release) | https://docs.vllm.ai/en/stable/ |
| Releases and release notes | https://github.com/vllm-project/vllm/releases |
| Docker deployment | https://docs.vllm.ai/en/latest/deployment/docker/ |
| Kubernetes deployment | https://docs.vllm.ai/en/latest/deployment/k8s/ |
| Online serving (OpenAI-compatible API) | https://docs.vllm.ai/en/latest/serving/online_serving/ |
| Engine arguments | https://docs.vllm.ai/en/latest/configuration/engine_args/ |
| Quantization | https://docs.vllm.ai/en/latest/features/quantization/index.html |
| Benchmark CLI (`vllm bench`) | https://docs.vllm.ai/en/latest/benchmarking/cli/ |
| vLLM paper (PagedAttention, SOSP 2023) | https://arxiv.org/abs/2309.06180 |
| Continuous batching explainer (Anyscale, by Cade Daniel et al.) | https://www.anyscale.com/blog/continuous-batching-llm-inference |

## Version observations (as of this refresh)

- Latest release: **v0.26.0** (published 2026-07-27). Release cadence is roughly
  monthly; docs publish `stable` and `latest` streams plus per-version archives.
- The official Docker image is `vllm/vllm-openai` on Docker Hub, with
  `vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants;
  the XPU image is official starting with v0.26.0.
- Engine argument documentation moved to `configuration/engine_args`; engine
  args are also available as JSON-style CLI arguments (`--json-arg.key value`).
- Benchmarking is now a first-class CLI: `vllm bench serve` (online serving),
  `vllm bench throughput` (offline), plus latency-focused and multimodal
  variants, replacing the older `benchmark_serving.py`/`benchmark_throughput.py`
  scripts. The docs recommend the external GuideLLM framework for production
  capacity testing.
- `--prefix-caching` and `--performance-mode` (balanced/interactivity/throughput)
  are current flags on the `vllm serve` command line; older doc pages may still
  show earlier spellings.
- Default `--gpu-memory-utilization` is 0.92 (per instance). `--kv-cache-dtype`
  accepts `auto`, `bfloat16`, `float16`, `fp8` (`fp8_e4m3`), `int8_per_token_head`,
  `nvfp4`, and other hardware-specific values on CUDA 11.8+.
- vLLM 0.26.0 dependencies include Transformers 5.13, FlashInfer 0.6.14, and
  NIXL 1.3.1; GPU support spans NVIDIA (Ampere+ for most quantized kernels),
  AMD ROCm, and Intel XPU, with a CPU backend for testing.

## Refresh procedure

1. Re-check the sources above for a new release and read its release notes for
   changed or removed engine arguments and changed defaults.
2. Update the version observations that changed (defaults, flag names, endpoint
   behavior, hardware support).
3. Re-verify the SKILL.md scope keyword sweep and the routing links to
   `ml-engineering` and `llama-cpp` still resolve.
4. Re-run the bundled probe against a test server and confirm every check still
   parses: `scripts/vllm-health --url http://127.0.0.1:8000 --json`.

## Related skill sources

- `ml-engineering` owns serving methodology: engine selection, quantization
  decisions, deployment plans, and regression triage. Its references are the
  source for engine-spanning decisions; this skill covers vLLM operation itself.
- `llama-cpp` owns the llama.cpp stack (GGUF, llama-server). Do not duplicate
  its content here.

# vLLM Deployment: Docker and Kubernetes

> **Last Updated:** 2026-08-03
> Sources: https://docs.vllm.ai/en/latest/deployment/docker/ and
> https://docs.vllm.ai/en/latest/deployment/k8s/

## Docker

The official image is `vllm/vllm-openai` (NVIDIA CUDA), with
`vllm/vllm-openai-rocm` (AMD) and `vllm/vllm-openai-xpu` (Intel) variants. The
image entrypoint is `vllm serve`, so engine arguments follow the image tag.

```bash
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.26.0 \
    --model Qwen/Qwen3-0.6B
```

### Container details that matter

- **Shared memory**: use `--ipc=host` or a `--shm-size` (for example `--shm-size=2g`).
  PyTorch uses shared memory between processes, particularly for tensor-parallel
  inference; the default 64 MB `/dev/shm` in a container is too small and causes
  obscure crashes at load time.
- **HF token**: pass `--env "HF_TOKEN=$HF_TOKEN"` for gated models. Never commit
  the token; mount the cache volume so weights are reused across restarts.
- **CUDA compatibility**: on hosts whose driver is older than the toolkit in the
  image, set `VLLM_ENABLE_CUDA_COMPATIBILITY=1` (only for select datacenter GPUs).
- **Compile cache**: vLLM compiles `torch.compile` artifacts into
  `VLLM_CACHE_ROOT` (default `~/.cache/vllm`). Mount a named volume there so the
  second and later containers start fast instead of recompiling.
- **Non-root**: the image ships a `vllm` user (UID 2000, GID 0). Run with
  `--user 2000:0` and mount writable paths under `/home/vllm` (for example
  `/home/vllm/.cache/huggingface`) rather than `/root`. The `vllm-openai-nonroot`
  image target supports OpenShift-style arbitrary UIDs within group 0.
- **Pin tags**: use a release tag (`vllm/vllm-openai:v0.26.0`), not `latest`.
  Optional dependencies (audio, gRPC, etc.) are not in the base image; layer
  them on with `uv pip install --system vllm[extra]==<same-version>`.

## Kubernetes

A native deployment: Deployment + Service, GPU resource limits, a model-cache
volume, an `emptyDir` shared-memory volume, and `/health` probes.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vllm
    spec:
      volumes:
        - name: cache-volume
          persistentVolumeClaim:
            claimName: vllm-models
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: "2Gi"
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.26.0
          command: ["/bin/sh", "-c"]
          args:
            - "vllm serve <model> --served-model-name <name> --trust-remote-code"
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token-secret
                  key: token
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          volumeMounts:
            - mountPath: /root/.cache/huggingface
              name: cache-volume
            - mountPath: /dev/shm
              name: shm
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 5
```

### K8s details that matter

- **GPU scheduling**: request `nvidia.com/gpu: "1"` (NVIDIA device plugin) or
  `amd.com/gpu` (AMD k8s device plugin). Tensor parallelism across GPUs in one
  pod uses `--tensor-parallel-size` equal to the GPU count; the pod then needs
  a larger `/dev/shm` (size it at 2-8 GiB) and, on some platforms, host IPC.
- **Probes**: vLLM's `/health` endpoint reports ready only after the model
  finishes loading, which can take minutes for large models. Set
  `initialDelaySeconds` and `failureThreshold` high enough; a container killed
  by the probe loop logs `KeyboardInterrupt: terminated` and shows
  `failed startup probe, will be restarted` in `kubectl get events`. To find
  the right threshold, remove the probes, time startup, then restore them.
- **gRPC**: pass `--grpc` (requires `vllm[grpc]` in the image) and replace the
  HTTP probes with `grpc` probes; the server then implements the standard
  gRPC health-checking protocol and returns `NOT_SERVING` while loading or
  shutting down.
- **Alternatives**: the upstream docs also cover Helm, KServe, KubeRay,
  NVIDIA Dynamo, and the vllm-project production-stack integrations. For this
  skill's scope, the native manifest above is the reference pattern; the
  infrastructure for those frameworks routes to `kubernetes`.

## Verification at the delivery boundary

- `kubectl logs -l app.kubernetes.io/name=vllm` shows `Application startup
  complete` and `Uvicorn running on http://0.0.0.0:8000`.
- `vllm-health --url <service-url> --check health --check models --json` shows
  `/health` 200 and the served model.
- A bounded request returns generated tokens; a PVC-backed model cache means
  the next pod starts without re-downloading weights.

# vLLM Model Configuration: Quantization, Tensor Parallelism, KV Cache

> **Last Updated:** 2026-08-03
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
> https://docs.vllm.ai/en/latest/features/quantization/index.html

## Model identity and context

- `--model` is the Hugging Face repo or a local path; `--revision` pins a branch,
  tag, or commit so weight provenance is reproducible. `--tokenizer` overrides
  the tokenizer; `--chat-template` supplies a Jinja2 chat template when the
  model card lacks one (without one, chat requests error).
- `--max-model-len` bounds prompt + output per request and accepts human-readable
  values (`32k` = 32,000; `32K` = 32,768). Unset, it derives from the model
  config. `-1`/`auto` picks the largest length that fits GPU memory, capped by
  the model's trained context. This flag is the dominant driver of KV cache size:
  halving it roughly doubles the concurrent sequences a GPU can hold.
- `--dtype` selects weight/activation precision (`auto`, `bfloat16`, `float16`,
  `float32`, `half`). `auto` uses FP16 for FP32/FP16 models and BF16 for BF16
  models; some quantized formats are recommended at a specific dtype (for
  example `half` for AWQ).

## Quantization-aware serving

- **Let the checkpoint declare its scheme.** vLLM first checks the model's
  `quantization_config`; `--quantization`/`-q` is for cases where the config is
  missing or needs overriding. Serving a checkpoint with the wrong method fails
  to load or silently degrades.
- **Supported methods (as of v0.26.0)**: GPTQ, AWQ, bitsandbytes (load-time
  quantization), GGUF, LLM Compressor FP8/INT8/INT4, NVIDIA Model Optimizer
  (NVFP4/MXFP4/FP8), TorchAO, and online quantization. The current index lives
  at https://docs.vllm.ai/en/latest/features/quantization/index.html.
- **Hardware coupling**: kernel support varies by GPU generation — for example
  Marlin (GPTQ/AWQ/FP8/FP4) requires Turing+ and is NVIDIA-only; FP8 W8A8 needs
  Ada/Hopper; GGUF and bitsandbytes span more platforms. Check the compatibility
  table before choosing a quantized checkpoint for a GPU fleet.
- **KV cache quantization**: `--kv-cache-dtype` (`auto`, `bfloat16`, `float16`,
  `fp8` = `fp8_e4m3`, `int8_per_token_head`, `nvfp4`, ...) shrinks the attention
  cache for long-context workloads on CUDA 11.8+. This is a quality-vs-capacity
  decision; spot-check outputs before trusting it at scale.
- **Dated-source rule**: quantization support changes every release. Never
  assume a method that worked on one version works on the next; the source
  index records the refresh date.

## Tensor, pipeline, and data parallelism

- `--tensor-parallel-size N` (`-tp`) shards each layer's weights and attention
  across N GPUs in one node. It requires fast interconnect (NVLink or high-speed
  NIC) and equal per-GPU memory. Startup runs a memory-profiling pass and logs
  the resulting KV cache size — that log line is the evidence the model fits.
- `--pipeline-parallel-size N` (`-pp`) splits layers across ranks/nodes for
  models too large for one node's aggregate memory; it adds inter-stage
  communication latency.
- `--data-parallel-size N` (`-dp`) replicates the model across groups for
  throughput scaling of small models. `--expert-parallel-size` shards MoE
  experts. The product of TP × PP × DP (× EP for MoE) must equal the world size.
- **Gotchas**: tensor-parallel workers must all see the same visible GPU set
  (`CUDA_VISIBLE_DEVICES` applies to the whole process group); mixed GPU models
  or different per-GPU memory cause the memory profile to fail; multi-node TP
  needs `--distributed-executor-backend mp` with `--master-addr`/`--master-port`
  reachable across nodes.

## Memory budgeting

- `--gpu-memory-utilization` (default 0.92) caps the fraction of GPU memory the
  model executor may use — weights plus KV cache plus activation buffers. It is
  per-instance and does not account for other processes on the GPU.
- Weights dominate first: a bf16 70B needs ~140 GB before any KV cache, so it
  needs two 80 GB GPUs (TP=2) or quantization. Only after weights fit does the
  KV cache decide concurrency: the engine logs `GPU KV cache size: N tokens`
  and `Maximum concurrency for M tokens per request: K`; record both.
- `--cpu-offload-gb` moves weights/KV to CPU per GPU (a virtual memory increase
  at the cost of PCIe-bound performance) — a stopgap for fitting a model, not a
  performance feature.
- `--load-format` (auto, safetensors, npcache, bitsandbytes, sharded_state, ...)
  controls how weights are read; `--safetensors-load-strategy eager` avoids
  random reads on network filesystems (NFS/Lustre) at the cost of CPU RAM.

## Configuration record

Every non-default choice goes into `templates/serving-config.md`. The record is
what makes a deployment reproducible and rollback a redeploy of the previous
pinned image plus record.

# vLLM OpenAI-Compatible API Surface

> **Last Updated:** 2026-08-03
> Source: https://docs.vllm.ai/en/latest/serving/online_serving/

## Basic endpoints

| Endpoint | Purpose |
|---|---|
| `/health` | Liveness; returns 200 only when the engine is ready to serve |
| `/version` | vLLM version information |
| `/v1/models` | List of served models (the `served-model-name`s clients must use) |
| `/load` | Server load metrics |
| `/metrics` | Prometheus-compatible metrics (`vllm:*` counters and gauges) |

## Inference endpoints

- `/v1/completions` — text generation (no chat template needed). Note: the
  `suffix` parameter is not supported.
- `/v1/chat/completions` — chat; requires the model to carry a chat template in
  its tokenizer config (or pass `--chat-template <file|string>`). The `user`
  parameter is ignored. `parallel_tool_calls` controls whether more than one
  tool call per response is allowed.
- `/v1/responses` (+ `/v1/responses/{id}/cancel`) — the OpenAI Responses API,
  for text-generation models.
- `/v1/embeddings` — for embedding/pooling models.
- `/v1/audio/transcriptions` and `/v1/audio/translations` — ASR models.
- Anthropic Messages API (`/v1/messages`, `/v1/messages/count_tokens`) and
  gRPC (`--grpc`) are also served on recent releases.

Chat models whose card lacks a template, and pooling/embedding models, will not
serve chat routes; a 404 on `/v1/chat/completions` while `/v1/models` works is
the classic symptom.

## Request/response facts

- The `model` field in requests must match a `--served-model-name` (or the
  `--model` value if none was set). `--served-model-name` accepts multiple names
  and aliases; the response echoes the first.
- Streaming (`"stream": true`) emits `choices[].delta` chunks with a terminal
  `finish_reason`. Tool calling requires server flags: `--enable-auto-tool-choice
  --tool-call-parser openai` (parser per model family). Structured outputs use
  xgrammar or guidance backends.
- Sampling parameters (temperature, top-p, top-k, max_tokens, stop, logprobs)
  are per-request; `--max-logprobs` caps logprobs server-wide.

## Exposure and security

- Start on loopback (`--host 127.0.0.1`) and verify with `vllm-health` and a
  bounded request before any wider bind. Exposing beyond loopback requires an
  explicit decision: bind address, API key (`--api-key`), TLS or a trusted
  reverse proxy, firewall, CORS, and rate limiting.
- Development-only and destructive routes must never be exposed in production:
  `/reset_prefix_cache`, `/reset_mm_cache`, weight-transfer endpoints
  (`/start_weight_update`, `/update_weights`), profiling (`/start_profile`),
  and `/collective_rpc`. vLLM gates some of these behind
  `VLLM_SERVER_DEV_MODE=1` — do not enable dev mode on production ingress.
- `--enable-log-requests` at debug level logs prompt text; keep it off or
  redacted in shared/audited sessions. Do not print or `tee` unredacted
  environment files, `.env` contents, or HF tokens.

## Verification checklist

1. `/health` returns 200.
2. `vllm-health --check models --json` lists the served model name(s).
3. A bounded chat/completion request returns generated tokens and a
   `finish_reason`.
4. Streaming and tool-calling paths are verified with the exact client that
   production uses (not assumed from endpoint existence).

# vLLM Benchmarking: Throughput and Latency

> **Last Updated:** 2026-08-03
> Source: https://docs.vllm.ai/en/latest/benchmarking/cli/

## Metrics that matter

- **Throughput**: request throughput (req/s), output token throughput (tok/s),
  and total token throughput (input + output tok/s). Output token throughput is
  the number users feel; total token throughput includes prompt processing.
- **Latency**: TTFT (time to first token — what users perceive as the start of
  a response), TPOT (time per output token, excluding the first), and ITL
  (inter-token latency, including scheduling jitter). Report mean, median, and
  p99 — p99 under load is the SLO-relevant number.
- Throughput and latency are different axes: raising concurrency raises
  throughput until the KV cache saturates, then latency degrades. State which
  axis a change optimizes before measuring it.

## `vllm bench serve` (online serving benchmark)

Run against a live server with a representative dataset:

```bash
vllm bench serve \
  --backend vllm \
  --model <model-name> \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 \
  --request-rate inf \
  --max-concurrency 64 \
  --save-result --result-dir ./log
```

Output includes: `Successful requests`, `Benchmark duration (s)`, `Total input
tokens`, `Total generated tokens`, `Request throughput (req/s)`, `Output token
throughput (tok/s)`, `Total token throughput (tok/s)`, and mean/median/p99 of
TTFT, TPOT, and ITL.

### Load pattern control

- `--request-rate`: `inf` sends all requests immediately (maximum throughput
  test); a finite rate (requests/second) simulates arrival traffic with a
  Poisson process (`--burstiness 1.0`), bursty Gamma traffic (`0.1-0.5`), or
  uniform spacing (`2.0-5.0`).
- `--max-concurrency`: caps outstanding requests, simulating a load balancer or
  gateway limit. The most common production pattern is `--request-rate=inf
  --max-concurrency=<limit>`.
- Datasets: `sharegpt` (realistic chat), `custom` (a JSONL of `{"prompt": ...}`
  entries), synthetic random lengths, and HF-hosted sets. For capacity planning
  use the KV-cache-derived maximum concurrency the server logs at startup as
  the ceiling (80-90% of it for realistic tests).

## `vllm bench throughput` (offline benchmark)

Measures raw engine throughput without the HTTP path:

```bash
vllm bench throughput \
  --model <model-name> \
  --input-len 512 --output-len 128 \
  --num-prompts 1000
```

Use it for engine-only comparisons (kernel, quantization, parallelism); it is
not a user-facing latency measurement. `vllm bench latency` is the
latency-oriented offline variant.

## Comparable evidence (the skill's core rule)

Benchmark numbers are only comparable under **matched conditions**. The
[benchmark run record template](../templates/benchmark-run-record.md) freezes:

- vLLM version/image tag/digest, model + revision, quantization + dtype;
- parallelism (TP/PP/DP), `max-model-len`, KV cache dtype, memory utilization;
- batching knobs (`max-num-seqs`, `max-num-batched-tokens`, chunked prefill,
  prefix caching), performance mode;
- GPU model/count/driver/interconnect, host, thermal, background load;
- dataset, prompt/output lengths, request rate, concurrency, sampling params;
- repetitions and variance.

Never compare numbers from different versions, models, quants, parallelism,
contexts, batches, or workloads as if one variable changed. Re-run the frozen
benchmark after every serving-arg change and record both sides in the template.

## Production capacity testing

For SLA validation and capacity planning the upstream docs recommend the
external GuideLLM framework (live progress, automatic reports). This skill's
scope is the bundled `vllm bench` tools plus the run-record discipline; route
a GuideLLM setup as its own tooling decision.

# vLLM Continuous Batching and Tuning

> **Last Updated:** 2026-08-03
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/ and
> https://www.anyscale.com/blog/continuous-batching-llm-inference

## How continuous batching works

vLLM batches requests **continuously** (iteration-level scheduling): the
scheduler admits new sequences whenever capacity frees up, mixing prefill and
decode in the same iteration instead of waiting for a whole batch to finish.
PagedAttention stores KV blocks in a paged table so memory is allocated per
token-block rather than per whole sequence, which is what makes high occupancy
and large effective concurrency possible. The practical consequences:

- A single model replica serves many concurrent requests; throughput rises with
  concurrency until the KV cache or the batch limits bind.
- Long prefill requests can block decode of other requests unless chunked
  prefill is enabled, which is exactly what spikes p99 latency under mixed
  workloads.

## The knobs

| Flag | Default behavior | What it does |
|---|---|---|
| `--max-num-seqs` | convenience default (set per deployment) | Max sequences per scheduling iteration. Higher = more batching and throughput, higher per-request latency and KV pressure. |
| `--max-num-batched-tokens` | convenience default (set per deployment) | Max tokens per iteration. Together with chunked prefill it bounds how much prefill work can run between decode steps. |
| `--enable-chunked-prefill` | disabled by default on most configs | Lets a prefill request be chunked across iterations so it shares time with decode; the fix for prefill-blocking-decode latency spikes. |
| `--enable-prefix-caching` | disabled | Reuses KV blocks for shared request prefixes (chat system prompts, RAG contexts); hit rate shows up in `/metrics` and lower effective input tokens. |
| `--performance-mode` | `balanced` | Kernel-level trade: `interactivity` favors low end-to-end latency at small batch sizes; `throughput` favors aggregate tok/s at high concurrency. |
| `--optimization-level` | `-O2` | Startup-time vs performance trade for `torch.compile` (`-O0` fastest start, `-O3` best performance). |

Both `--max-num-seqs` and `--max-num-batched-tokens` accept human-readable
sizes (`1k`, `2M`, ...). `--long-prefill-token-threshold` sets when a prefill is
treated as long for chunked-prefill scheduling.

## Tuning discipline

1. **Establish the baseline first**: freeze the serving config record, run the
   benchmark, and record `vllm:gpu_cache_usage_perc`,
   `vllm:num_requests_running`, and `vllm:num_requests_waiting` at the peak.
2. **Change one knob at a time** and re-run the frozen benchmark. Two changes
   in one iteration make the delta unassignable.
3. **Read the signals**: cache usage near 1.0 with requests waiting means the
   KV cache is the binding constraint — lower `--max-num-seqs`, shorten
   `--max-model-len` or cap output tokens, or scale out. It does not mean
   raise the batch limits further.
4. **Match the knob to the symptom**: latency spikes under mixed load →
   chunked prefill + a deliberate `--max-num-batched-tokens`; low utilization
   at steady state → raise `--max-num-seqs` within the cache budget;
   shared-prefix workloads → `--enable-prefix-caching`.
5. **Verify at the boundary**: re-run the benchmark and confirm the declared
   metric moved and the guardrail metrics (p99 TTFT/TPOT) did not regress,
   with variance reported across repetitions.

## Capacity math from the startup log

At startup the engine reports something like:

```
GPU KV cache size: 15,728,640 tokens
Maximum concurrency for 8,192 tokens per request: 1920
```

`max_concurrency = kv_cache_size / max_model_len`. Use 80-90% of that figure as
the `--max-concurrency` for capacity-planning benchmarks, and treat the size as
the ceiling when deciding whether to raise concurrency or shorten context.

# vLLM GPU Operation, Upgrade, and Rollback

> **Last Updated:** 2026-08-03
> Sources: https://docs.vllm.ai/en/latest/configuration/engine_args/,
> https://docs.vllm.ai/en/latest/deployment/docker/,
> https://github.com/vllm-project/vllm/releases

## GPU operation

- **Inventory first**: `nvidia-smi` (NVIDIA) or `rocm-smi` (AMD) shows device
  list, per-GPU memory, utilization, temperature, power, and ECC/error state.
  Record it before and after changes; a thermal or ECC change invalidates
  benchmark comparisons.
- **Device selection**: `CUDA_VISIBLE_DEVICES="0,1" vllm serve ...` restricts
  which GPUs the process sees; `--tensor-parallel-size` counts the *visible*
  devices in order. Device names are the container's view, not the host's —
  verify with `nvidia-smi -L` inside the container.
- **Per-GPU memory matters**: tensor parallelism assumes equal per-GPU memory.
  Mixed-capacity GPUs or other workloads sharing a GPU break the memory
  profile; pin the serving process to dedicated devices.
- **Observability**: `/metrics` exposes `vllm:gpu_cache_usage_perc` (KV cache
  pressure), `vllm:num_requests_running`, `vllm:num_requests_waiting`, and
  token counters. Prometheus scraping of `/metrics` is the standard wiring;
  alert on cache usage near 1.0 with requests waiting (capacity), repeated OOM,
  and probe failures.
- **OOM at startup** usually means weights + KV cache do not fit: reduce
  `--max-model-len`, switch to a quantized checkpoint, add GPUs, or (as a
  stopgap) `--cpu-offload-gb`. Lowering `--gpu-memory-utilization` does not
  help if the weights alone exceed memory.
- **OOM mid-run** means KV cache pressure at peak concurrency: shorten context,
  lower `--max-num-seqs`, or scale out replicas.

## Upgrade

1. **Pin before you start**: image tag (or `pip install vllm==<version>`),
   model revision, and the full serving command from the serving config record.
   Without pins, upgrades are unreproducible and rollback is guesswork.
2. **Read the release notes for the full span** (for example v0.25.1 → v0.26.0):
   vLLM deprecates and renames flags, changes defaults (memory allocation,
   `max-model-len` derivation, CUDA graph capture), and adjusts kernel/hardware
   support between releases. A config that ran on one minor may not behave the
   same on the next.
3. **Rehearse on a scratch instance** with the real model and workload: install
   the new version, apply the config, confirm the model loads, and run the
   frozen benchmark.
4. **Swap with a rollback plan**: deploy the new pinned image, verify at the
   delivery boundary (`/health`, `/v1/models`, a representative request,
   re-benchmark), and keep the previous image + config ready to reapply.

## Rollback

- **Rollback is a redeploy of the previous pinned image + serving config.**
  Because the config record is versioned and the image tag is pinned, the old
  state is reproducible by construction.
- Verify the rollback the same way as an upgrade: `/health`, `/v1/models`, one
  representative request, and the frozen benchmark. Do not assume the old
  behavior returned because the old image is back.
- Watch for **config drift across versions**: KV cache layout, defaults, and
  flag names change between releases. A rollback must restore the *recorded*
  config, not the current one.

## Troubleshooting table

| Symptom | First evidence to collect | Likely causes and next step |
|---|---|---|
| Server crashes at startup (OOM) | Side-by-side startup logs, `nvidia-smi`, image digests | Weights+KV don't fit; changed defaults after upgrade. Reduce `max-model-len`/memory utilization on scratch, or roll back the pinned image |
| `/health` 200 but requests fail | `/v1/models`, one bounded request | Model not loaded, served-name mismatch, missing chat template; check `/load` and startup log |
| 404 on `/v1/chat/completions` | `/v1/models`, model type | Pooling/embedding model or missing chat template; see `references/03-openai-api.md` |
| High p99 latency | TTFT vs TPOT split, cache-usage metrics | Prefill blocking decode → chunked prefill; KV pressure → bound concurrency |
| Requests waiting, cache ~100% | `vllm:gpu_cache_usage_perc`, waiting gauge | At capacity: scale out or reduce context/concurrency; do not raise batch limits |
| Container killed at startup | `kubectl get events`, log `KeyboardInterrupt: terminated` | Probe `failureThreshold` too low for model load time; raise it |
| Slow restart every time | Startup duration, `VLLM_CACHE_ROOT` | Compile cache not persisted; mount `~/.cache/vllm` volume |

## Hard boundaries

- No restart, redeploy, scale, or upgrade without an explicit human directive
  naming the target and a stated rollback path. Read-only discovery may proceed
  without confirmation.
- Never compare performance across versions, models, quants, parallelism,
  contexts, batches, or workloads as if one variable changed.
- Never expose development-only endpoints, request logs with prompt content, or
  credentials to chat or logs.

# vLLM Benchmark Run Record

One record per benchmark run. Numbers are only comparable across runs with
matched frozen conditions — change one variable at a time and state it here.

## Objective

- Hypothesis: _[fill: what change is being evaluated]_
- Primary metric and threshold: _[fill: e.g. output tok/s >= 1.2x baseline]_
- Guardrail metrics and thresholds: _[fill: e.g. p99 TTFT < 2s]_
- Workload represented: _[fill: production traffic shape, request mix]_

## Frozen conditions

- vLLM version / image tag / digest: _[fill: e.g. vllm/vllm-openai:v0.26.0]_
- Model, revision, quantization, dtype: _[fill: repo/name@commit, method, dtype]_
- Tensor / pipeline / data parallel sizes: _[fill: e.g. tp=2, pp=1]_
- Max model len / KV cache dtype / gpu memory utilization: _[fill: values]_
- Max num seqs / batched tokens / chunked prefill / prefix caching: _[fill: values]_
- GPU model, count, driver, interconnect: _[fill: e.g. 2x A100 80GB, NVLink]_
- Host, thermal, background load: _[fill: machine, cooling, concurrent jobs]_

## Benchmark invocation

- Serving command (from the serving config record): _[fill: reference or command]_
- Tool and command: _[fill: vllm bench serve / vllm bench throughput / guideLLM]_
- Dataset: _[fill: sharegpt / custom jsonl path / prompt lengths]_
- Number of prompts / request rate / burstiness / max concurrency: _[fill: values]_
- Sampling parameters: _[fill: temperature, top-p, max_tokens]_
- Warmup / repetitions / delay: _[fill: e.g. 50 prompts warmup, 3 repetitions]_

## Raw results

- Raw output path: _[fill: saved vllm bench output or --save-result file]_
- Request throughput (req/s): _[fill: value]_
- Output token throughput (tok/s): _[fill: value]_
- Total token throughput (tok/s): _[fill: value]_
- TTFT mean / median / p99 (ms): _[fill: values]_
- TPOT mean / median / p99 (ms): _[fill: values]_
- ITL mean / median / p99 (ms): _[fill: values]_
- GPU KV cache usage peak: _[fill: vllm:gpu_cache_usage_perc peak]_

## Compared variable (only one)

- Baseline value: _[fill: reference the previous record]_
- Candidate value: _[fill: this record's change]_
- All other known differences: _[fill: none, or list any drift]_

## Conclusion

- Outcome vs threshold: _[fill: met / not met / inconclusive]_
- Variance across repetitions: _[fill: spread of the primary metric]_
- Decision and next experiment: _[fill: ship, roll back, or next single-variable change]_

# vLLM Serving Configuration Record

Fill this record before launching or changing a vLLM server. It is the rollback
unit: the previous record plus the previous pinned image is the rollback path.

## Deployment identity

- Requested outcome: _[fill: what the deployment must do and for whom]_
- Deployment type: _[fill: bare vllm serve / Docker / Kubernetes]_
- Target and scope confirmed with: _[fill: who confirmed, when]_
- Rollback path: _[fill: previous image tag + previous record]_

## Pinned artifacts

- vLLM image or version: _[fill: vllm/vllm-openai:v0.26.0 or pip vllm==...]_
- Image digest (when available): _[fill: sha256:...]_
- Model repository and revision: _[fill: repo/name@commit or tag]_
- Served model name(s): _[fill: names clients must use in /v1 requests]_
- Tokenizer / chat template override: _[fill: path or "from model card"]_
- Config file or command source: _[fill: path to the recorded vllm serve args]_

## Model and engine configuration

- Dtype: _[fill: auto / bfloat16 / float16 / float32]_
- Quantization: _[fill: none / gptq / awq / fp8 / gguf / ...; match weight format]_
- Tensor parallel size: _[fill: 1 / N GPUs in node]_
- Pipeline parallel size: _[fill: 1 / number of nodes]_
- Data / expert parallel size (if used): _[fill: 1 / ...]_
- Max model length: _[fill: prompt + output bound, e.g. 32768]_
- GPU memory utilization: _[fill: 0.0-1.0, default 0.92]_
- KV cache dtype: _[fill: auto / fp8 / ...]_
- CPU offload GB (if used): _[fill: 0 or GiB per GPU]_

## Batching and scheduling

- Max sequences per iteration: _[fill: --max-num-seqs value or default]_
- Max batched tokens per iteration: _[fill: --max-num-batched-tokens value]_
- Chunked prefill: _[fill: enabled / disabled / default]_
- Prefix caching: _[fill: enabled / disabled]_
- Performance mode: _[fill: balanced / interactivity / throughput]_

## Environment and access

- Host / cluster: _[fill: node, cluster, namespace]_
- GPUs (model, count, driver): _[fill: e.g. 2x A100 80GB, driver 560.x]_
- Bind address and port: _[fill: 127.0.0.1:8000 or explicit exposure]_
- Authentication / TLS / proxy: _[fill: API key, TLS terminator, reverse proxy]_
- Model cache mount and HF token handling: _[fill: volume path; token never stored here]_

## Startup verification

- [ ] `/health` returns 200
- [ ] `/v1/models` lists the served model name
- [ ] A representative request returns generated tokens
- [ ] Startup log records memory profiling and GPU KV cache size: _[fill: tokens]_
- [ ] `/metrics` shows expected `vllm:gpu_cache_usage_perc` baseline: _[fill: value]_

## Changes from the previous record

- _[fill: what changed, why, and which benchmark run record backs it]_

#!/usr/bin/env python3
"""Deterministic tests for the vllm/scripts/vllm-health probe.

Runs the script as a subprocess so the tests exercise the real CLI surface
(--help, --json, --check subsets, exit codes, JSON payloads). A local
stdlib HTTP server stubs the vLLM endpoints (/health, /version, /v1/models,
/load, /metrics), so no external network or vLLM server is needed. Also
asserts the read-only contract: the script never opens files in write mode.
"""
import json
import socket
import subprocess
import sys
import threading
import time
import unittest
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
SCRIPT = ROOT / "scripts" / "vllm-health"


def run_script(*args):
    return subprocess.run(
        [sys.executable, str(SCRIPT), *args],
        capture_output=True,
        text=True,
        timeout=30,
    )


def load_json(proc):
    return json.loads(proc.stdout)


class StubVLLMServer:
    """Minimal read-only stub of the vLLM HTTP surface."""

    def __init__(self):
        handler = self._make_handler()
        self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
        self.port = self.server.server_address[1]
        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)

    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass

            def do_GET(self):
                if self.path == "/health":
                    self._send(200, "OK")
                elif self.path == "/version":
                    self._send(200, "v0.26.0")
                elif self.path == "/v1/models":
                    self._send(
                        200,
                        json.dumps({"object": "list", "data": [{"id": "test-model"}]}),
                    )
                elif self.path == "/load":
                    self._send(200, json.dumps({"model": "test-model", "state": "OK"}))
                elif self.path == "/metrics":
                    self._send(200, "vllm:gpu_cache_usage_perc 0.42\nvllm:num_requests_running 3\n")
                else:
                    self._send(404, "not found")

            def _send(self, code, body):
                payload = body.encode("utf-8")
                self.send_response(code)
                self.send_header("Content-Type", "text/plain")
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)

        return Handler

    def start(self):
        self.thread.start()

    def stop(self):
        self.server.shutdown()
        self.server.server_close()

    def url(self):
        return f"http://127.0.0.1:{self.port}"


class SlowVLLMServer(StubVLLMServer):
    """A stub that responds slowly, for timeout testing."""

    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass

            def do_GET(self):
                time.sleep(2.0)
                payload = b"OK"
                self.send_response(200)
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)

        return Handler


class LargeMetricsServer(StubVLLMServer):
    """A stub whose /metrics body exceeds the probe's read bound."""

    @staticmethod
    def _make_handler():
        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass

            def do_GET(self):
                if self.path == "/metrics":
                    payload = (b"vllm:gpu_cache_usage_perc 0.5\n" + b"x" * 80 * 1024)
                else:
                    payload = b"OK"
                self.send_response(200)
                self.send_header("Content-Length", str(len(payload)))
                self.end_headers()
                self.wfile.write(payload)

        return Handler


def free_port():
    """Return a currently-unused local port (socket is closed after)."""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(("127.0.0.1", 0))
    port = sock.getsockname()[1]
    sock.close()
    return port


class HelpTests(unittest.TestCase):
    def test_help_exits_zero_and_advertises_capabilities(self):
        proc = run_script("--help")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("--json", proc.stdout)
        self.assertIn("--check", proc.stdout)
        self.assertIn("health", proc.stdout)

    def test_usage_error_exits_two(self):
        proc = run_script("--not-a-real-flag")
        self.assertEqual(proc.returncode, 2)


class ProbeTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.server = StubVLLMServer()
        cls.server.start()

    @classmethod
    def tearDownClass(cls):
        cls.server.stop()

    def test_health_check_passes(self):
        proc = run_script("--url", self.server.url(), "--check", "health")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("OK", proc.stdout)
        self.assertIn("health", proc.stdout)

    def test_models_check_parses_served_models(self):
        proc = run_script("--url", self.server.url(), "--check", "models", "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        checks = payload["checks"]
        self.assertEqual(checks[0]["ok"], True)
        self.assertIn("test-model", checks[0]["model_ids"])

    def test_all_checks_pass_with_json(self):
        proc = run_script("--url", self.server.url(), "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        self.assertEqual(len(payload["checks"]), 5)
        for check in payload["checks"]:
            self.assertTrue(check["ok"], f"{check['name']} should pass: {check}")

    def test_metrics_check_reads_key_vllm_gauges(self):
        proc = run_script("--url", self.server.url(), "--check", "metrics", "--json")
        self.assertEqual(proc.returncode, 0)
        payload = load_json(proc)
        observed = payload["checks"][0]["key_metrics_present"]
        self.assertTrue(observed["vllm:gpu_cache_usage_perc"])
        self.assertTrue(observed["vllm:num_requests_running"])

    def test_unreachable_server_fails_with_exit_one(self):
        proc = run_script("--url", f"http://127.0.0.1:{free_port()}", "--check", "health")
        self.assertEqual(proc.returncode, 1)
        self.assertIn("FAIL", proc.stdout)

    def test_timeout_emits_documented_exit_code_124(self):
        slow = SlowVLLMServer()
        slow.start()
        try:
            proc = run_script("--url", slow.url(), "--check", "health", "--timeout", "0.5")
            self.assertEqual(proc.returncode, 124)
            self.assertIn("timed out", proc.stdout)
        finally:
            slow.stop()

    def test_metrics_read_is_bounded_and_reports_truncation(self):
        large = LargeMetricsServer()
        large.start()
        try:
            proc = run_script("--url", large.url(), "--check", "metrics", "--json")
            self.assertEqual(proc.returncode, 0)
            payload = load_json(proc)
            check = payload["checks"][0]
            self.assertTrue(check["ok"])
            self.assertTrue(check["truncated"])
            self.assertEqual(check["bytes_read"], 64 * 1024)
        finally:
            large.stop()

    def test_models_check_parses_from_stub(self):
        """The models check parses the stub's served model list (positive path)."""
        proc = run_script(
            "--url", f"http://127.0.0.1:{self.server.port}", "--check", "models", "--json"
        )
        payload = load_json(proc)
        self.assertEqual(payload["checks"][0]["ok"], True)


class ReadOnlyContractTests(unittest.TestCase):
    def test_script_never_opens_files_for_writing(self):
        source = (SCRIPT).read_text(encoding="utf-8")
        for line in source.splitlines():
            stripped = line.strip()
            if stripped.startswith("#"):
                continue
            self.assertNotIn("'w'", stripped)
            self.assertNotIn('"w"', stripped)
            self.assertNotIn("'a'", stripped)
            self.assertNotIn('"a"', stripped)

    def test_script_only_issues_get_requests(self):
        source = (SCRIPT).read_text(encoding="utf-8")
        self.assertIn('method="GET"', source)


if __name__ == "__main__":
    unittest.main()