{
  "schema_version": 1,
  "skill_name": "ml-engineering",
  "evals": [
    {
      "id": "fine-tuning-plan-review",
      "prompt": "A teammate wants to fine-tune a 7B model on 40,000 internal support tickets to improve answer quality. Their plan: full fine-tuning on a single A100, 20 epochs, no baseline eval, a random 80/20 split of the tickets for train and eval, and they will report the final eval loss as the success metric. Review this fine-tuning plan against ml-engineering methodology and tell them what to change.",
      "expected_output": "A fine-tuning plan review that reorders the work around measurement: run a baseline with the base model against the eval set before spending any training budget, and decide whether fine-tuning is even needed. The reviewer questions the full fine-tuning choice for a 7B on one GPU (LoRA or QLoRA fits the compute budget and the data size) and replaces 20 epochs with early stopping driven by a held-out validation split. The eval split must not be randomly carved from the same tickets used for training: a leakage audit (for example with the eval-set overlap checker) and a time- or source-based split is required so the reported numbers mean something. Success is defined by a small set of task-specific eval cases with a rubric, not by eval loss, and every run records seed, data version, hyperparameters, and environment so it can be reproduced.",
      "assertions": [
        "The review demands a base-model baseline against the eval set before training budget is spent",
        "The review recommends LoRA or QLoRA over full fine-tuning given the data size and single-GPU budget",
        "The review flags the random 80/20 split as a contamination risk and requires a leakage audit or time/source-based split",
        "The review rejects eval loss as the success metric and requires task-specific eval cases with a rubric",
        "The review requires a reproducibility record covering seed, data version, hyperparameters, and environment"
      ]
    },
    {
      "id": "eval-set-design",
      "prompt": "We are about to fine-tune a model on customer-support conversations and need an eval set to decide whether the fine-tune ships. Design the eval set: how many examples, what should they cover, how do we score them, and how do we keep the eval set from leaking into training data?",
      "expected_output": "An eval-set design built around the deployment task: 50-200 representative examples drawn from real production traffic (plus synthetic edge cases), each paired with a reference answer and a scoring rubric that defines correct, partially correct, and incorrect for that task. The set is stratified to include common intents, rare intents, and known failure modes, and it is versioned and frozen before training starts so it cannot silently drift. Leakage control is explicit: examples are deduplicated against the training corpus, the eval-set overlap checker runs between the train and eval corpora before every training run, and the set is rotated or replenished periodically so the model cannot memorize it. Per-example scores are tracked, not just the aggregate, so a regression in one capability is visible.",
      "assertions": [
        "The design specifies 50-200 representative examples with reference answers and a scoring rubric",
        "The design covers stratification across common intents, rare intents, and known failure modes",
        "The design freezes and versions the eval set before training starts",
        "The design requires a leakage/overlap check between the eval set and training corpus before each run",
        "The design tracks per-example scores so regressions in individual capabilities are visible"
      ]
    },
    {
      "id": "quantization-decision",
      "prompt": "We serve a 70B model and want to cut serving cost by quantizing it. Someone proposed jumping straight to 4-bit weights to double throughput. Walk me through how to decide whether to quantize, what to compare, and how to prove the choice is safe.",
      "expected_output": "A quantization decision process that measures before it optimizes: establish the baseline quality of the unquantized model on the production eval set first, then treat each candidate (GGUF q4/q5/q8, GPTQ, AWQ, bitsandbytes) as a hypothesis to test against that baseline. Each candidate is evaluated with the same eval set and the same calibration-data strategy, comparing quality deltas per capability as well as VRAM footprint, tokens-per-second, and latency under production concurrency. The decision is recorded with the measured numbers, the hardware and serving engine used, and the quality threshold the model must stay above; if a quantized variant regresses a critical capability, it is rejected or shipped only for the workloads where the regression does not matter. The eval set and the recording keep the decision reproducible and comparable when a new model or engine version lands.",
      "assertions": [
        "The process measures the unquantized baseline on the production eval set before quantizing",
        "The comparison covers GGUF/GPTQ/AWQ/bitsandbytes candidates with the same eval set and calibration strategy",
        "The comparison measures quality deltas per capability plus VRAM, throughput, and latency under production concurrency",
        "The decision is recorded with measured numbers, hardware, serving engine, and an explicit quality threshold",
        "A variant that regresses a critical capability is rejected or limited to workloads where the regression is acceptable"
      ]
    },
    {
      "id": "deployment-plan",
      "prompt": "Our fine-tuned LoRA adapter passed eval and needs to go to production serving. Design the deployment plan: serving stack, hardware, rollout, and how we know the deployed model is actually better and not silently broken.",
      "expected_output": "A deployment plan that treats the model like any other artifact: the LoRA adapter and base model are versioned and pinned together, the serving engine (for example llama.cpp or vLLM) is chosen by the latency and concurrency budget, and GPU sizing is derived from measured tokens-per-second under load rather than model-card arithmetic. Rollout is staged: shadow or canary traffic first, compared against the incumbent model with the production eval set, then a gradual ramp with an automatic rollback if latency, error rate, or a sampled eval score crosses the threshold. Monitoring is set up before release: request latency, token throughput, GPU utilization, and a continuously sampled eval score so a regression surfaces within the deployment window instead of in next week's report. The plan names the rollback path (revert the adapter version, keep the serving config) and who signs off on each stage.",
      "assertions": [
        "The plan pins model and adapter versions together and derives GPU sizing from measured throughput under load",
        "The plan chooses a serving engine (llama.cpp or vLLM) against the latency and concurrency budget",
        "The plan uses shadow or canary rollout compared against the incumbent model before ramping",
        "The plan defines an automatic rollback trigger on latency, error rate, or sampled eval score",
        "The plan requires pre-release monitoring and continuous sampled eval scoring during the rollout"
      ]
    },
    {
      "id": "regression-triage",
      "prompt": "After we shipped a fine-tuning update, aggregate benchmark scores went up, but one capability subset dropped noticeably: the model now fumbles multi-turn repair dialogs that it handled before. Triage this regression and decide what to do.",
      "expected_output": "A regression triage that disaggregates before concluding: the aggregate improvement is real but hides the subset loss, so the first step is per-subset and per-example comparison against the previous run, with the eval set frozen at the same version. The triage checks whether the drop is a genuine behavior change or an eval artifact: whether examples leaked or were reshuffled between the train and eval corpora (an overlap check), whether the eval set was rotated, and whether the delta is within run-to-run noise. If the regression is real, the likely causes are examined: data mix drift, the fine-tune overfitting the new examples, or a capability trade-off introduced by the update. The decision is explicit: retrain with a corrected data mix, tune the LoRA rank or learning rate, or roll back the adapter while keeping the eval-regression table as the record of what was measured and decided.",
      "assertions": [
        "The triage disaggregates results per subset and per example against the previous run on a frozen eval set",
        "The triage rules out eval artifacts including train/eval overlap or leakage and eval-set rotation",
        "The triage checks whether the delta is within run-to-run noise before calling it a real regression",
        "The triage investigates data-mix drift, overfitting, and capability trade-offs as root causes",
        "The triage ends with an explicit decision: retrain, adjust hyperparameters, or roll back, recorded in the eval-regression table"
      ]
    },
    {
      "id": "training-run-reproducibility",
      "prompt": "A colleague trained a model a month ago and cannot reproduce it now: same script, same data folder name, different result. What information was missing from their record, and what should a training-run record capture so this never happens again?",
      "expected_output": "A reproducibility diagnosis that lists what a training-run record must pin down: the base model identity and commit or hash, the exact data version (folder name is not a version; the record needs a hash or dataset revision), the full training configuration (seed, optimizer, learning rate and schedule, warmup, gradient accumulation, batch size, epochs, precision and mixed-precision settings, LoRA rank/alpha/target modules if applicable), the framework and library versions, and the hardware. The record also captures the eval methodology so the same eval set and scoring code produce the same numbers, and it stores the trained artifact path and its hash. The colleague's missing pieces are almost certainly the seed, the data version hash, and the environment pin; the fix is a filled training-run record template committed next to the training code so a future run starts from a reproducible baseline instead of a folder name.",
      "assertions": [
        "The diagnosis requires base-model identity and data version by hash, not folder name",
        "The diagnosis requires the full training configuration including seed, optimizer, schedule, batch size, and precision",
        "The diagnosis requires framework and library versions plus hardware details",
        "The diagnosis requires pinning the eval methodology so scores are comparable across runs",
        "The fix is a committed training-run record template that captures all of the above"
      ]
    },
    {
      "id": "metric-and-stochastic-comparison",
      "prompt": "Two model updates have nearly identical average accuracy, but one improves a rare high-harm class and has higher latency. Design the evaluation decision.",
      "expected_output": "A decision that defines task harms and thresholds, reports per-class and per-example results, measures latency/cost, repeats stochastic comparisons under a declared protocol, and treats an aggregate tie as insufficient evidence.",
      "assertions": [
        "The response chooses metrics from task harms and decision thresholds",
        "The response disaggregates rare or critical capabilities instead of relying on average accuracy",
        "The response measures latency or cost alongside quality",
        "The response repeats stochastic comparisons with controlled seeds or a declared sampling protocol",
        "The response treats aggregate similarity as inconclusive when critical capability or operational constraints differ"
      ]
    },
    {
      "id": "end-to-end-model-lineage",
      "prompt": "A production model artifact passed evaluation, but nobody can link it to the exact feature cutoff, dataset revision, training configuration, or deployed image. Decide whether it can be promoted and specify the missing record.",
      "expected_output": "Promotion is blocked or explicitly marked non-reproducible until immutable links connect source, dataset, features and temporal cutoff, splits, code/config, run artifacts, evaluation, registry, serving configuration, and endpoint. The record names owners and timestamps.",
      "assertions": [
        "The response does not treat a model file or experiment URL as sufficient lineage",
        "The response requires immutable identifiers for data, features and cutoff, code/configuration, artifacts, evaluation, registry, serving, and deployment",
        "The response blocks or clearly marks promotion as non-reproducible when links are missing",
        "The response assigns owners and timestamps to lineage links",
        "The response preserves a reproducible record for later diagnosis or compliance"
      ]
    },
    {
      "id": "temporal-feature-parity",
      "prompt": "Offline evaluation is strong, but online performance dropped after launch. The feature team says the definitions are the same. Diagnose the risk and define the validation.",
      "expected_output": "A diagnosis that checks event time versus availability time, lookback windows, source revisions, and offline/online transformation parity using replay or shadow data. The response treats training-serving skew as a leading hypothesis before retraining.",
      "assertions": [
        "The response checks event time, availability time, source revision, and permitted lookback windows",
        "The response validates semantic parity with replay or shadow samples",
        "The response identifies training-serving skew as a plausible cause before retraining",
        "The response checks that training and evaluation exclude information unavailable at prediction time",
        "The response records the feature definition and cutoff version used by the deployed model"
      ]
    },
    {
      "id": "drift-response-decision",
      "prompt": "A deployed classifier has input drift and a delayed label signal. Define the response before automatically retraining.",
      "expected_output": "A bounded drift response that checks schema/source changes, seasonality, segment mix, label delay, and metric integrity; defines thresholds, sample windows, owners, and critical-capability impact; and chooses among observation, data repair, recalibration, retraining, human review, rollback, or retirement with candidate validation.",
      "assertions": [
        "The response treats drift as an investigation signal rather than automatic proof that retraining is correct",
        "The response checks schema/source changes, seasonality or segment mix, label delay, and metric integrity",
        "The response defines threshold, minimum window or sample, owner, and critical-capability impact",
        "The response includes bounded options such as data repair, human review, rollback, or retirement",
        "The response validates a candidate against a frozen regression set and recent production slice before ramping"
      ]
    }
  ]
}

# Evaluation, lineage, and production response

Use this reference for model-quality decisions that must remain comparable after training, adaptation, deployment, or data change.

## Configuration and metric choice

Start with the deployment decision, not a convenient metric. State the task, harm of errors, decision threshold, acceptable tradeoffs, and the smallest representative evaluation set. Select metrics that expose those risks (for example, per-class precision/recall for imbalanced classification, calibration when probabilities drive action, and latency/throughput when serving capacity is part of the decision). Record the metric definition, aggregation, weighting, and pass threshold.

For stochastic training or generation, repeat the same comparison with controlled seeds or a declared sampling protocol. Report the distribution or uncertainty across repeats and inspect per-capability/per-example deltas. An aggregate change within observed run-to-run variation is inconclusive; a statistically significant result still does not authorize a release if a critical capability regresses.

## End-to-end lineage

A promotable model needs links across the whole chain:

`source snapshot → dataset revision → feature definitions and cutoff → split/eval revision → code and configuration → run/artifacts → evaluation report → registered model → serving image/configuration → deployed endpoint`

Store immutable identifiers or hashes for each link, plus the owner, timestamp, environment, and transformation boundary. A model file or experiment URL alone is insufficient. If a link is missing, label the result non-reproducible and block promotion until the gap is accepted by the responsible owner.

## Temporal feature parity

For every feature, define its event time, availability time, source revision, transformation, and permitted lookback window. Training and batch evaluation must use only data available at the prediction timestamp. Online serving must run the same semantic transformation or consume a versioned feature definition. Compare offline and online distributions on a replay or shadow sample; investigate differences as training-serving skew before interpreting quality changes.

## Drift and response

Monitor system signals (errors, latency, throughput, resource saturation), input/feature distributions, prediction distributions, and outcome quality when labels arrive. Define thresholds, minimum sample/window, owner, and action before deployment. A drift alert is a signal to investigate, not proof that retraining is correct: check schema/source changes, seasonality, segment mix, label delay, and metric integrity first.

The response options are bounded: continue with a documented observation, repair the data/feature path, recalibrate or retrain, route to human review, roll back to the last known-good model, or retire the service. Choose using the critical-capability threshold, lineage completeness, cost/latency budget, and rollback readiness. Validate the candidate on the frozen regression set and a recent production slice before ramping traffic.

## Training, adaptation, and serving tradeoffs

Treat prompting, adapter fine-tuning, full fine-tuning, retrieval augmentation, quantization, and a serving-engine change as different hypotheses. Compare them against the same baseline and task eval, while recording data freshness, quality delta by capability, latency/throughput, memory, cost, operational complexity, and rollback path. Keep the smallest intervention that clears the target; do not spend training budget to solve a data, retrieval, or serving bottleneck without evidence.

This is methodology. Commands and lifecycle details for a named model server, orchestrator, registry, feature store, or monitoring product belong to that tool’s skill.

# Model Evaluation

## Benchmark Selection

| What you want to measure | Recommended benchmarks |
|--------------------------|----------------------|
| General reasoning | MMLU-Pro, GPQA, ARC-Challenge |
| Code generation | HumanEval+, SWE-Bench, BigCodeBench |
| Instruction following | MT-Bench, AlpacaEval, Arena-Hard |
| Tool calling | BFCL, ToolBench, jdhodges tool-call eval |
| Safety | TruthfulQA, BBQ, ToxiGen |
| Math | GSM8K, MATH, AIME |

## Custom Eval Design

When off-the-shelf benchmarks don't capture your domain:

1. **Collect 50-200 representative examples** from your actual use cases
2. **Define a scoring rubric** — what constitutes correct, partially correct, and incorrect
3. **Include adversarial examples** — edge cases, ambiguous inputs, known failure modes
4. **Run baseline (base model) first** — establish the ceiling before fine-tuning
5. **Track per-example** — aggregate scores hide regressions in specific capabilities

## Regression Tracking

| Before change | After change | Interpretation |
|---|---|---|
| Score A | Score A ± noise | No detectable effect |
| Score A | Score A - delta | Regression — investigate |
| Score A | Score A + delta | Improvement — verify on held-out set |
| Score A, B regressed | Score A + delta, B regressed | Tradeoff — intentional? |

# Fine-Tuning

## Approach Selection

| Approach | When to use | VRAM | Data needed |
|----------|-------------|------|-------------|
| LoRA | Adapting to a new domain/task, limited VRAM | 8-16 GB for 7B | 100-10K examples |
| QLoRA | Very limited VRAM, large base model | 6-12 GB for 7B | 100-10K examples |
| Full fine-tune | Maximum capability shift, sufficient VRAM | 40-80 GB for 7B | 10K+ examples |
| DoRA | Weight-decomposed adaptation, better than LoRA | +5% over LoRA | Same as LoRA |

## Training Config Checklist

- [ ] Seed set for reproducibility
- [ ] Learning rate schedule selected (cosine, linear, constant)
- [ ] Warmup steps configured (typically 5-10% of total)
- [ ] Gradient accumulation steps set
- [ ] Mixed precision (bf16/fp16) enabled if hardware supports
- [ ] Evaluation during training (every N steps)
- [ ] Checkpoint saving with best-model tracking (eval loss)
- [ ] WandB or local logging configured
- [ ] Data shuffled before each epoch
- [ ] Train/validation split verified (no cross-contamination)

## Validation Strategy

| Data regime | Validation approach |
|-------------|-------------------|
| < 500 examples | K-fold cross-validation (k=5) |
| 500-10K | 80/10/10 train/val/test split |
| 10K+ | 90/5/5 split with stratified sampling |
| Imbalanced classes | Stratified split by class distribution |

# ML Model Quantization & Inference Serving — Reference Manual

> **Purpose:** Engineering methodology reference for production ML serving. Covers quantization approaches, calibration strategies, quality assessment, mixed precision, KV cache optimization, and inference serving architectures.
>
> **Last updated:** 2025-06-05

---

## Table of Contents

1. [Quantization Fundamentals](#1-quantization-fundamentals)
2. [Quantization Approaches Compared](#2-quantization-approaches-compared)
   - 2.1 GGUF
   - 2.2 GPTQ
   - 2.3 AWQ
   - 2.4 bitsandbytes NF4
   - 2.5 EXL2
   - 2.6 HQQ
   - 2.7 Comparison Table
3. [Calibration Data Strategies](#3-calibration-data-strategies)
4. [Quality Impact Assessment](#4-quality-impact-assessment)
5. [Mixed Precision Patterns](#5-mixed-precision-patterns)
6. [KV Cache Quantization](#6-kv-cache-quantization)
7. [Quantization Workflow](#7-quantization-workflow)
8. [Inference Serving Architecture](#8-inference-serving-architecture)
   - 8.1 vLLM
   - 8.2 llama.cpp Server
   - 8.3 Text Generation Inference (TGI)
   - 8.4 Triton Inference Server
   - 8.5 Serving Framework Comparison Table
9. [Production Considerations](#9-production-considerations)
10. [References & Further Reading](#10-references--further-reading)

---

## 1. Quantization Fundamentals

**Quantization** reduces the numerical precision of model weights (and optionally activations / KV cache) from full-precision (FP32, BF16) to lower-bit representations (INT8, INT4, FP8, NF4). This shrinks memory footprint, reduces memory bandwidth pressure, and accelerates inference — especially on bandwidth-bound decode steps.

### Key concepts

| Concept | Description |
|---|---|
| **Weight quantization** | Map each weight tensor's values from a high-precision range into a low-bit grid. Reduces model size by 2-4x at common bit widths. |
| **Activation quantization** | Quantize intermediate activations at runtime. Harder than weight quantization due to dynamic range variation. Common in INT8 pipelines (e.g., TensorRT). |
| **KV cache quantization** | Quantize the key-value cache during autoregressive generation. Critical for long-context serving where KV cache dominates GPU memory. |
| **Symmetric vs. asymmetric** | Symmetric: zero point = 0, range is [-max, max]. Asymmetric: zero point can shift. Asymmetric generally preserves more info for non-normalized distributions. |
| **Per-tensor vs. per-channel (per-group)** | Finer granularity (per-channel or per-group) captures outlier distributions better at the cost of storing more scale/zero-point metadata. |
| **Post-training quantization (PTQ)** | Quantize after training. Dominant paradigm for LLMs given cost of training. |
| **Quantization-aware training (QAT)** | Simulate quantization during training (e.g., FakeQuant ops). More accurate but expensive. Used in some production pipelines (e.g., NVIDIA TensorRT). |

### Why quantize LLMs?

- **Memory:** A 70B model at FP16 requires ~140 GB VRAM. At INT4, ~35 GB — fitting on a single H100/A100-80GB.
- **Throughput:** Lower-precision weights reduce memory bandwidth consumption, which is the primary bottleneck for autoregressive token generation (memory-bound, not compute-bound).
- **Cost:** Enables deployment on cheaper/consumer hardware (RTX 4090, Apple Silicon unified memory, CPU-only).

---

## 2. Quantization Approaches Compared

### 2.1 GGUF

**Type:** File format + quantization scheme  
**Ecosystem:** llama.cpp, Ollama, LM Studio  
**Bit widths:** Q2_K through Q8_0, plus Q4_K_M, Q5_K_M, Q6_K, etc.

GGUF (GPT-Generated Unified Format) is the successor to GGML. It packages a model's weights, tokenizer, and metadata into a single file. The quantization variants use a **k-quant** scheme that assigns different bit widths to different layers based on their importance:

- **Q4_K_M** — recommended sweet spot. Mixture of 4-bit and 6-bit quantization across layers. ~4.5 bits/weight effective.
- **Q5_K_M** — higher quality, ~5.5 bits/weight effective.
- **Q8_0** — near-lossless 8-bit, ~8.5 GB for a 7B model.
- **Q2_K** — aggressive 2-3 bit mix, significant quality loss.

**Key features:**
- Supports CPU inference natively (no GPU required for modest models).
- Can offload layers to GPU via `--n-gpu-layers`.
- Single-file distribution simplifies deployment.
- Supports a wide range of architectures (Llama, Mistral, Falcon, Gemma, etc.).
- **Imatrix** (importance matrix) quantization: weights are quantized with per-layer importance scores, improving quality at a given bit rate.

**Strengths:** Universal format, best CPU/edge support, large pre-quantized Hub ecosystem (TheBloke, etc.), excellent for local/offline use.

**Weaknesses:** Not natively supported by HuggingFace `transformers` or vLLM (though vLLM added GGUF support in 2025). GPU performance trails AWQ/GPTQ on NVIDIA hardware.

---

### 2.2 GPTQ

**Type:** Post-training quantization (weight only)  
**Ecosystem:** AutoGPTQ, HuggingFace optimum, vLLM, TGI  
**Bit widths:** 2-8 bits (most common: 4-bit), supports group size 32/64/128

GPTQ (GPT Post-Training Quantization) uses approximate second-order optimization (Hessian-based) to find weight quantizations that minimize output error. It was the first widely adopted 4-bit LLM quantization method.

**How it works:**
1. Sample a calibration dataset (typically 128 sequences from the training distribution).
2. Compute the approximate Hessian (Fisher information) for each weight column.
3. Quantize weights column-by-column, using the Hessian to prioritize preserving important weights.
4. Update remaining unquantized weights to compensate for quantization error (optimal brain quantization / OBC family).

**Key parameters:**
- **Group size** (g128, g64, g32): Smaller groups = higher accuracy but more scale storage overhead. g128 is common. g64 preferred for quality.
- **Desc_act / act_order** (activation order): When True, reorders columns by activation magnitude. Increases accuracy significantly but reduces speed in older implementations. vLLM's Marlin kernel makes desc_act fast.

**Strengths:** Strong quality at 4-bit, mature ecosystem, widely supported in serving frameworks.

**Weaknesses:** Calibration dataset required (cannot quantize a model from scratch without data). Quantization is slower than AWQ or NF4 due to Hessian computation.

---

### 2.3 AWQ

**Type:** Post-training quantization (weight only)  
**Ecosystem:** AutoAWQ, vLLM, TGI, TensorRT-LLM  
**Bit widths:** 4-bit (most common), also 2-bit, 3-bit variants

AWQ (Activation-Aware Weight Quantization) observes that a small fraction (~1%) of weight channels are "salient" — they handle large activations and are disproportionately important. AWQ protects these channels by scaling them up before quantization, then scaling the output down.

**How it works:**
1. Run a few calibration samples to collect activation statistics.
2. Identify salient channels (those with large activation magnitudes).
3. Apply per-channel scaling factors to redistribute quantization error from salient → non-salient channels.
4. Quantize with simple round-to-nearest.

**Key advantages:**
- **No group size dependency:** AWQ INT4 often matches GPTQ g128 quality without requiring groups, simplifying kernel implementation.
- **Very fast quantization:** Minutes instead of hours. No Hessian computation needed.
- **Excellent GPU kernel support:** Marlin kernel (for GPTQ-compatible AWQ) and AWQ-specific kernels in vLLM achieve near-peak hardware utilization.
- **Good hardware compatibility:** Works well on NVIDIA, AMD, and Apple Silicon via MLX.

**Strengths:** Best quality-to-speed tradeoff at 4-bit, fastest quantize time, strong production support in vLLM.

**Weaknesses:** Primarily designed for 4-bit (less flexible than GGUF's range of bit widths). Requires activation statistics → needs calibration data.

---

### 2.4 bitsandbytes NF4

**Type:** Post-training quantization (weight only)  
**Ecosystem:** HuggingFace `bitsandbytes`, `transformers`, PEFT/LoRA  
**Bit widths:** 4-bit (NF4), 8-bit (INT8)

Bitsandbytes (BnB) is a library from Tim Dettmers that implements efficient GPU quantization kernels. Its 4-bit variant uses **NormalFloat (NF4)** — a non-uniform quantization grid that assumes normally distributed weights.

**Key concepts:**
- **NF4:** A 4-bit data type with 16 levels, non-uniformly spaced to match the cumulative distribution function (CDF) of a normal distribution. This gives higher resolution near zero where most weight values cluster.
- **Double quantization:** Quantizes the quantization constants (scale/offset) themselves to save additional memory. Reduces the 4-bit overhead from ~0.5 bits/weight to ~0.127 bits/weight.
- **QLoRA:** Fine-tune quantized models with LoRA adapters. The base model stays in NF4; only the LoRA parameters are updated in FP16.

**Key features:**
- Native `transformers` integration via `BitsAndBytesConfig` — load any model in 4-bit with a single config object.
- Best for fine-tuning (QLoRA) and rapid prototyping.
- No calibration data needed — quantization is "on the fly" at load time.

**Strengths:** Simplest API, no calibration required, excellent for fine-tuning, HuggingFace-native.

**Weaknesses:** Slower inference than AWQ/GPTQ (dequantization at every forward pass). Kernels are less optimized for serving throughput. Not suitable for high-throughput production serving on its own.

---

### 2.5 EXL2

**Type:** Post-training quantization  
**Ecosystem:** ExLlamaV2  
**Bit widths:** Mixed 2-8 bits per layer

EXL2 is the quantization format for ExLlamaV2, a high-throughput inference engine. It supports mixed-precision within a single model — different layers can use different bit widths.

**Key features:**
- Fine-grained per-layer bit allocation for optimal quality/size tradeoffs.
- Very fast GPU inference — ExLlamaV2 kernels are among the fastest for single-batch inference.
- Less widely supported than GGUF/GPTQ/AWQ but excellent for local GPU use.

**Strengths:** Fastest local GPU inference for many models, flexible per-layer bit allocation.

**Weaknesses:** Smaller ecosystem, primarily desktop/local use. Not widely supported in production serving frameworks.

---

### 2.6 HQQ

**Type:** Post-training quantization  
**Ecosystem:** HuggingFace, independent  
**Bit widths:** 1-8 bits

HQQ (Half-Quadratic Quantization) uses a half-quadratic splitting approach to compute optimal quantization. It's notable for being extremely fast to quantize (no calibration data, no Hessian) and supporting very low bit widths (2-bit, 3-bit).

**Strengths:** Fastest PTQ (no data needed), supports 1-3 bits, good quality at lower bits.

**Weaknesses:** Needs `torch.compile` for reasonable inference speed; otherwise dequantization overhead is high. Less mature ecosystem.

---

### 2.7 Quantization Method Comparison Table

| Property | GGUF (Q4_K_M) | GPTQ (g128) | AWQ | BnB NF4 | EXL2 | HQQ |
|---|---|---|---|---|---|---|
| **Effective bits/weight** | ~4.5 | ~4.125 | ~4.0 | ~4.127 | Variable | 1-8 |
| **File format** | Single .gguf | HF safetensors | HF safetensors | HF safetensors | Custom | HF safetensors |
| **Calibration data req.** | No* | Yes (128 seq) | Yes (128 seq) | No | Yes | No |
| **Quantize speed** | Fast** | Slow (hrs) | Fast (min) | Instant (load) | Moderate | Very fast |
| **GPU inference speed** | Moderate | Fast (Marlin) | Very fast (Marlin) | Slow | Very fast | Moderate |
| **CPU inference** | Native | No | No | No | No | No |
| **Apple Silicon** | Native (MLX) | Via MLX | Via MLX | No | No | Possible |
| **Serving support** | vLLM, llama.cpp | vLLM, TGI | vLLM, TGI, TRT-LLM | Limited | ExLlamaV2 | Limited |
| **Fine-tuning support** | No | No | No | Yes (QLoRA) | No | Possible |
| **Ecosystem maturity** | Very high | High | High | Very high | Moderate | Low |
| **Typical PPL increase (7B)** | +0.15 | +0.10 | +0.10 | +0.20 | +0.10 | +0.12 |

\* GGUF quantization typically does not use calibration data; imatrix quantization does.  
\** Quantization is an explicit step via `llama-quantize` or similar tools (not on-the-fly).

---

## 3. Calibration Data Strategies

Some quantization methods (GPTQ, AWQ, EXL2, imatrix) require a **calibration dataset** — a small set of representative text samples used to compute activation statistics or Hessian information.

### Recommended calibration datasets

| Dataset | Typical Size | Use Case |
|---|---|---|
| **Wikitext-2** | 128 seq × 2048 tokens | Standard benchmark, general text |
| **C4 (Colossal Clean Crawled Corpus)** | 128-256 seq | General web text, diverse |
| **Pile** | 128-256 seq | General, diverse (books, code, academic) |
| **Custom task-specific** | 128-512 seq | Domain adaptation (medical, legal, code) |
| **Random from training data** | 128 seq | Best if available (closest to training distribution) |

### Best practices

1. **Size:** 128-256 sequences of 2048 tokens is typically sufficient. More calibration data has diminishing returns and can even harm quality (overfitting the calibration set).
2. **Diversity:** Calibration data should broadly match the model's training distribution. A model trained on code + text benefits from a calibration mix of both.
3. **Sequences vs. random tokens:** Always use natural text sequences, not random tokens. Random tokens produce meaningless activation statistics.
4. **Avoid duplication:** Deduplicate calibration data. Repeated samples can skew Hessian estimates.
5. **Prompt-like structure:** For instruction-tuned models, including representative prompts in calibration data can improve downstream quality.
6. **Multiple calibration runs:** Some advanced pipelines run calibration on multiple small datasets and average the quantized parameters.

### When calibration data matters most

- **GPTQ** — critically important. Poor calibration data leads to significantly higher perplexity.
- **AWQ** — important but more robust than GPTQ. The scaling factor approach is less sensitive to calibration data quality.
- **GGUF imatrix** — uses importance matrices computed from calibration data. Worth the effort for best-quality GGUF quants.
- **NF4 / HQQ** — no calibration data needed.

---

## 4. Quality Impact Assessment

### Quantization degradation patterns

| Bit Width | Quality Impact |
|---|---|
| **FP16 / BF16** | Baseline (lossless reference) |
| **INT8 (Q8_0, BnB INT8)** | Near-lossless. Negligible PPL increase (<0.01). Output-level differences often undetectable. |
| **6-bit (Q6_K)** | Very minor PPL increase (~0.02). Safe for production. |
| **5-bit (Q5_K_M, Q5_0)** | Small PPL increase (~0.05-0.15). Generally safe. |
| **4-bit (Q4_K_M, GPTQ, AWQ, NF4)** | Moderate PPL increase (~0.10-0.35). Noticeable on complex reasoning tasks. AWS/GPTQ typically best, NF4 worst at same bit width. |
| **3-bit (Q3_K_S, HQQ int3)** | Significant degradation. PPL +0.5-1.5. Tasks requiring multi-step reasoning (CoT) degrade notably. |
| **2-bit (Q2_K)** | Heavy degradation. Only usable for very tolerant tasks. |

### Task-level sensitivity

Not all tasks degrade equally:

| Task Type | Sensitivity | Notes |
|---|---|---|
| **Perplexity / next-token prediction** | Low | Relatively robust to quantization. |
| **Single-token classification (MMLU)** | Low-Moderate | 4-bit typically loses 1-2% accuracy. |
| **Multi-step reasoning (CoT, MATH)** | High | 4-bit can lose 3-5%+ on math reasoning. 3-bit often fails entirely. |
| **Code generation** | Moderate | Functional correctness degrades at aggressive quantization. |
| **Creative writing** | Low | Quality differences are subtle at 4-bit; 3-bit may produce incoherence. |
| **Instruction following** | Moderate | Longer, multi-step instructions become harder at lower precision. |
| **Few-shot learning** | Moderate | Degrades faster than zero-shot performance. |

### Empirical data (Llama 3 8B, from LessWrong benchmarks)

| Method | MMLU (0-shot) | WMDP | The Pile PPL |
|---|---|---|---|
| BF16 (baseline) | 63.87% | 54.99% | 8.283 |
| BnB INT8 | 63.05% | 54.96% | 8.305 |
| HQQ INT8 | 63.87% | 54.66% | 8.298 |
| AWQ INT4 | 61.84% | 54.55% | 8.483 |
| HQQ INT4 | 62.29% | 54.23% | 8.482 |
| GPTQ INT4 | 61.58% | 53.30% | 8.575 |
| BnB NF4 | 61.44% | 54.42% | 8.499 |
| BnB INT4 | 60.80% | 52.73% | 8.633 |
| HQQ INT3 | 62.26% | 51.23% | 8.872 |

> **Key takeaway:** AWQ = GPTQ > HQQ > NF4 > BnB INT4 at 4-bit. At 8-bit, all methods are essentially lossless. Differential sensitivity across tasks means eval should always be task-specific.

### Recommended evaluation framework

1. **Perplexity** — quick sanity check. Compute on withheld validation split (100k tokens minimum).
2. **Task-specific accuracy** — MMLU, HumanEval, GSM8K, or domain-specific benchmarks.
3. **A/B comparison** — Run paired generations from FP16 and quantized model. Human eval or LLM-as-judge for quality differences.
4. **Downstream metric** — For RAG systems, measure retrieval precision. For chatbots, measure response acceptability.

---

## 5. Mixed Precision Patterns

Mixed precision assigns different numerical precisions to different parts of the model or computation graph. This is distinct from per-layer variable-width quantization.

### Common mixed precision patterns

#### 5.1 Weight quantization + high-precision compute

```
Weights: INT4 / NF4
Activations: FP16 / BF16
Gradients (training): FP32
```

- Most common pattern for LLM inference.
- Weights are dequantized on-the-fly to FP16 for computation.
- Offered by AWQ, GPTQ, BnB, GGUF.
- **Tradeoff:** Dequantization overhead. AWQ/GPTQ minimize this via fused kernels (Marlin).

#### 5.2 Low-precision compute (FP8 matmul)

```
Weights: FP8
Activations: FP8
Accumulation: FP16/FP32
```

- NVIDIA H100/H200 supports native FP8 tensor cores (2x throughput vs. FP16).
- TensorRT-LLM and vLLM (FP8 support) use this for high-throughput serving.
- Activation ranges are calibrated (per-tensor or per-channel) at export time.
- Quantization-aware scaling (QTS) ensures accuracy.

#### 5.3 INT8 compute with INT4 weights (W4A8)

```
Weights: INT4
Activations: INT8
Compute: INT8 tensor cores
```

- Emerging pattern for maximum throughput on hardware with INT8 tensor cores (all NVIDIA GPUs since Volta).
- Requires activation quantization at inference time — more complex.
- Used by TensorRT-LLM and some custom serving stacks.

#### 5.4 Per-layer variable precision

```
Layer 1: Q4_K
Layer 5: Q6_K
Layer 14: Q5_K
...
```

- GGUF k-quant and EXL2 use this pattern.
- Sensitive layers (e.g., embedding, lm_head, early/late transformer layers) get higher precision.
- Reduces average bit width without sacrificing critical layers.

#### 5.5 FP16 weights + INT8 KV cache

```
Weights: FP16
KV Cache: INT8/FP8
Compute: FP16/BF16
```

- KV cache is the memory bottleneck for long contexts.
- Quantizing only the KV cache (not weights) saves 50-75% of KV cache memory.
- Supported by vLLM and TensorRT-LLM.

### Precision selection decision tree

```
Is model size > GPU VRAM?
├── YES → Can we tolerate quality loss?
│   ├── YES → INT4 weight quantization (AWQ/GPTQ) + optional KV cache quant
│   └── NO  → FP8 weight quant (if H100) or BF16 with tensor parallelism
└── NO  → Is latency critical?
    ├── YES → INT4 weights + FP16 activations (Marlin kernel)
    └── NO  → BF16 baseline is fine
```

---

## 6. KV Cache Quantization

### The KV cache problem

During autoregressive generation, each transformer layer computes Key (K) and Value (V) tensors that are cached for all previous tokens. For a batch of size `b`, `n_layers` layers, `n_heads` attention heads, sequence length `s`, and dimension `d_per_head`:

```
KV cache size = 2 × b × n_layers × n_heads × s × d_per_head × precision_bytes
```

At FP16, a 32K-token sequence with Llama 3 70B (80 layers, 8 KV heads, d=128) requires ~320 GB for the KV cache alone. Quantizing to INT8 halves this; to INT4 quarters it.

### Attention architecture impact

| Architecture | KV Cache per token (FP16) | Notes |
|---|---|---|
| **MHA** (Multi-Head Attention) | 2 × n_layers × n_heads × d | Largest cache. Every layer has full key/value for all heads. |
| **MQA** (Multi-Query Attention) | 2 × n_layers × 1 × d | One KV head shared across all query heads. 8-32x smaller than MHA. |
| **GQA** (Grouped-Query Attention) | 2 × n_layers × n_kv_heads × d | Middle ground. Llama 2/3 uses 8 KV heads for 32+ query heads. |
| **MLA** (Multi-Head Latent Attention) | 2 × n_layers × d_latent | DeepSeek's approach. Compresses KV into a low-rank latent space. ~2-4x smaller than GQA. |

### KV cache quantization methods

| Method | Bit Width | Strategy |
|---|---|---|
| **KVTuner** | INT4/INT8 | Sensitivity-aware per-layer mixed-precision. Key layers get 8-bit, others 4-bit. |
| **KVQuant** | INT4/FP8 | Per-channel + per-token quantization with non-uniform grids. Targets 10M+ context. |
| **FP8 KV cache** (H100 native) | FP8 | Uses H100 FP8 tensor cores. Minimal quality loss. |
| **INT8 KV cache** (vLLM) | INT8 | Per-tensor symmetric quantization. Standard vLLM feature. |
| **INT4 KV cache** (experimental) | INT4 | Per-channel asymmetric. Quality loss noticeable at very long contexts. |

### Production guidelines

1. **Start with GQA/MLA architecture** — architecture-level KV cache reduction is more impactful than quantization.
2. **FP8 KV cache on H100** — essentially lossless, 2x memory reduction. Enable in vLLM with `--kv-cache-dtype fp8`.
3. **INT8 KV cache** — good tradeoff for A100/H100. Minimal quality impact for contexts under 32K tokens.
4. **INT4 KV cache** — quality impact grows with sequence length. Evaluate carefully for long-context applications.
5. **Layer-wise KV quantization** — tools like KVTuner offer better quality at same average bit width by allocating higher precision to critical layers.

---

## 7. Quantization Workflow

The canonical workflow for applying and validating quantization in production:

### Step 1: Establish Baseline

- Load the model in FP16/BF16.
- Run evaluation benchmark (MMLU, perplexity, domain-specific tasks).
- Record latency and throughput at relevant batch sizes.
- Record VRAM usage.
- This is the reference against which all quantized variants are compared.

### Step 2: Select Quantization Method

Use the comparison table (Section 2.7) to choose based on:
- **Target hardware** (CPU? GPU? Apple Silicon? Cloud instance type?)
- **Deployment framework** (vLLM? llama.cpp? TGI? TensorRT-LLM?)
- **Quality constraints** (must match FP16 within X%?)
- **Compute budget** (time available for quantization)

### Step 3: Quantize

```bash
# GGUF (via llama.cpp)
python3 convert.py --outtype f16 --model ./model --outpath model-f16.gguf
./llama-quantize model-f16.gguf model-q4km.gguf Q4_K_M

# GPTQ (via AutoGPTQ)
python3 -m auto_gptq --model ./model --quantize --bits 4 --group-size 128 --dataset c4

# AWQ (via AutoAWQ)
python3 -m awq.quantize --model_path ./model --quant_path ./awq-model --calib-data wikitext

# BnB NF4 (via transformers — on-the-fly)
# Simply load with BitsAndBytesConfig
```

### Step 4: Evaluate Quality

- **Primary metric:** Same evaluation benchmark as baseline (Step 1).
- **Secondary metric:** Perplexity on a held-out validation set (e.g., The Pile test split).
- **Tertiary metric:** A/B test with LLM-as-judge for generative tasks.
- **Threshold:** Define acceptable degradation (e.g., < 0.5% MMLU drop, < 0.3 PPL increase).

### Step 5: Benchmark Performance

- Measure tokens/second at batch size 1 (latency-sensitive).
- Measure throughput at max batch size (throughput-sensitive).
- Record peak VRAM usage.
- Compare to baseline and to alternative quantization methods.

### Step 6: Select the Winner

| Criteria | Decision |
|---|---|
| Quality within threshold, best throughput | Choose that method |
| Quality outside threshold | Try higher-precision variant (Q4_K_M → Q5_K_M; g128 → g64) |
| All methods fail threshold | Consider FP16 with tensor parallelism, or switch to a different architecture |
| Throughput insufficient | Consider FP8 (if H100) or lower-precision quant with faster kernel |

### Step 7: Production Deployment

- Store quantized model in model registry.
- Configure inference server with appropriate settings.
- Monitor quality metrics continuously (drift detection).
- Set up A/B test vs. previous version.

---

## 8. Inference Serving Architecture

### 8.1 vLLM

**Developed by:** UC Berkeley (Kwatra, Stoica)  
**Language:** Python/C++/CUDA  
**GitHub:** github.com/vllm-project/vllm  
**License:** Apache 2.0

vLLM is the most widely adopted open-source LLM serving framework, known for its combination of throughput and flexibility.

#### Core innovations

**PagedAttention**
- Inspired by OS virtual memory paging.
- KV cache is divided into fixed-size **blocks** (typically 16 or 32 tokens each).
- Blocks are stored in a non-contiguous page table, eliminating fragmentation.
- Enables near-zero memory waste vs. the 60-80% waste in traditional pre-allocated KV cache.
- Allows memory sharing across sequences for techniques like beam search and parallel sampling.

**Continuous Batching**
- Also called **in-flight batching** or **iteration-level batching**.
- Traditional servers wait for all sequences in a batch to finish before starting a new batch.
- Continuous batching adds/removes sequences from the batch **after every iteration** (every decoding step).
- Dramatically improves GPU utilization, especially when sequences have variable lengths.
- vLLM achieves up to **24x higher throughput** than HuggingFace Transformers on the same hardware.

**Chunked Prefill**
- Splits long prefill (prompt processing) into smaller chunks that can interleave with decode steps.
- Prevents long prompts from blocking decode-only sequences.
- Reduces time-to-first-token (TTFT) variability.

**Speculative Decoding**
- Uses a small draft model to propose multiple tokens, verified by the target model in one forward pass.
- 1.5-2.5x latency improvement on latency-sensitive workloads.

#### Key features

| Feature | Status | Notes |
|---|---|---|
| AWQ quantization | ✅ Native | Marlin kernel support |
| GPTQ quantization | ✅ Native | Marlin kernel support |
| GGUF quantization | ✅ Added 2025 | Via llama.cpp backend |
| FP8 (H100) | ✅ Native | Requires H100 |
| KV cache INT8/FP8 | ✅ Native | `--kv-cache-dtype` flag |
| Tensor parallelism | ✅ | Across GPU nodes |
| Pipeline parallelism | ✅ | Limited |
| Prefix caching | ✅ | Automatic KV cache reuse |
| OpenAI-compatible API | ✅ | Drop-in replacement |
| Multi-LoRA serving | ✅ | Efficient LoRA adapter switching |
| Guided decoding | ✅ | JSON schema, grammar |
| Disaggregated prefill/decode | ✅ | 2025 feature |

#### Typical deployment

```bash
# Start vLLM server
python3 -m vllm.entrypoints.openai.api_server \
    --model /path/to/model \
    --quantization awq \
    --dtype auto \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90 \
    --tensor-parallel-size 2 \
    --enable-prefix-caching
```

#### When to choose vLLM

- **High-throughput production serving** (chatbots, API endpoints).
- **Multi-model or multi-LoRA setups**.
- **OpenAI-compatible API needed**.
- **Heterogeneous GPU setups** (supports various NVIDIA GPUs, AMD ROCm).
- **Need for speculative decoding or prefix caching**.

---

### 8.2 llama.cpp Server

**Developed by:** Georgi Gerganov & community  
**Language:** C/C++  
**GitHub:** github.com/ggml-org/llama.cpp  
**License:** MIT

llama.cpp is a C/C++ inference engine focused on local/edge deployment with minimal dependencies. The `llama-server` component provides an HTTP API.

#### Architecture

- **No external dependencies** — pure C/C++ implementation with BLAS-optimized matrix operations.
- **ggml backend** — custom tensor library supporting CPU, CUDA, Metal, Vulkan, SYCL, and more.
- **Pure CPU inference** — unique among major serving frameworks. Can run 7B models at 10-20 tok/s on modern CPUs with AVX2.
- **GPU offloading** — `--n-gpu-layers N` offloads N transformer layers to GPU. The rest runs on CPU.
- **Quantization-native** — designed from the ground up for GGUF quantized models.

#### Key features

| Feature | Status | Notes |
|---|---|---|
| GGUF quantization | ✅ Native | Full k-quant suite |
| AWQ/GPTQ | ❌ Not native | Via conversions |
| CPU inference | ✅ Best-in-class | AVX2, AVX-512, NEON |
| GPU offloading | ✅ | CUDA, Metal, Vulkan |
| Batch inference | ✅ | Server mode with continuous batching |
| KV cache reuse | ✅ | Automatic |
| OpenAI-compatible API | ✅ | Built into `llama-server` |
| Grammar sampling | ✅ | GBNF grammar engine |
| Embedding endpoint | ✅ | Via `/v1/embeddings` |
| Vision (multimodal) | ✅ | Llava, etc. |
| Structured output | ✅ | JSON schema mode |

#### Typical deployment

```bash
./llama-server \
    --model /path/to/model.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --n-gpu-layers 32 \
    --threads $(nproc) \
    --ctx-size 8192 \
    --rope-scaling yarn \
    --cache-type-k q8_0 \
    --cache-type-v q8_0
```

#### When to choose llama.cpp

- **CPU-only or hybrid CPU/GPU deployments**.
- **Apple Silicon** (Metal backend is excellent).
- **Local/edge inference** (privacy-sensitive, offline).
- **Single-user or low-concurrency serving**.
- **Experimentation** (fastest iteration for trying different quant levels).
- **No dependency on Python or CUDA toolkit**.

---

### 8.3 Text Generation Inference (TGI)

**Developed by:** Hugging Face  
**Language:** Rust/Python  
**GitHub:** github.com/huggingface/text-generation-inference  
**License:** Apache 2.0

TGI is Hugging Face's production-grade inference server, used to power HuggingChat and the Hugging Face Inference API.

#### Architecture

- **Rust core** for HTTP routing and request management (high concurrency, low overhead).
- **Python/CUDA backend** for model execution.
- **Flash Attention 2** integration for efficient attention computation.
- **PagedAttention** added in v2.x (also called "Paged Attention in TGI").
- **Continuous batching** similar to vLLM.
- **Safetensors** and `transformers` integration — loads models directly from Hugging Face Hub.

#### Key features

| Feature | Status | Notes |
|---|---|---|
| AWQ quantization | ✅ | Via optimum |
| GPTQ quantization | ✅ | Via optimum |
| FP8 quantization | ✅ | H100 support |
| Bitsandbytes | ✅ | Via transformers |
| Tensor parallelism | ✅ | |
| Flash Attention 2 | ✅ | Default |
| PagedAttention (v2.x) | ✅ | Added after vLLM |
| Watermarking | ✅ | SynthID-Text |
| Message API | ✅ | Native chat templates |
| Streaming | ✅ | Server-Sent Events |
| Speculative decoding | ✅ | |

#### Typical deployment

```bash
docker run --gpus all \
    -p 8080:80 \
    -v /path/to/models:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id /data/model \
    --max-total-tokens 8192 \
    --quantize awq \
    --num-shard 2
```

#### When to choose TGI

- **Deep HuggingFace ecosystem integration** (Hub, optimum, tokenizers).
- **Production serving with AWS Inferentia** (TGI has native Inferentia2 support).
- **Message-based chat APIs** (native chat template handling).
- **When watermarking or model-level guardrails are needed**.

---

### 8.4 Triton Inference Server

**Developed by:** NVIDIA  
**Language:** C++/CUDA (backend), Python (frontend)  
**GitHub:** github.com/triton-inference-server/server  
**License:** BSD-3-Clause

Triton is NVIDIA's production inference server. It is model-framework-agnostic and designed for enterprise-grade deployments.

#### Architecture

- **Multi-framework backend:** Supports TensorRT, TensorRT-LLM, PyTorch, ONNX Runtime, vLLM, Python, and custom backends.
- **Concurrent model serving:** Multiple models (and multiple versions of the same model) served from a single instance.
- **Ensemble scheduler:** Chain multiple models together without custom code (e.g., embedding → re-rank → LLM).
- **Dynamic batching:** Client-side and server-side batching with configurable timeouts.
- **GPU/CPU/accelerator support:** Concurrent serving across heterogeneous hardware.
- **Prometheus metrics:** Native monitoring endpoint.

#### TensorRT-LLM backend

The TensorRT-LLM backend is the primary LLM serving path within Triton:

1. **Model optimization phase:** Convert model to TensorRT engine (FP16, INT8, INT4, FP8).
2. **Graph optimizations:** Kernel fusion, layer fusion, attention optimization.
3. **In-flight batching:** Equivalent to continuous batching.
4. **PagedAttention:** Adopted from vLLM's approach.
5. **Multi-node tensor parallelism:** Up to hundreds of GPUs.

#### Key features

| Feature | Status | Notes |
|---|---|---|
| Multi-framework | ✅ | Not just LLMs |
| TensorRT-LLM backend | ✅ | Highest throughput on H100 |
| INT4/FP8/INT8 quantization | ✅ | Through TensorRT |
| PagedAttention | ✅ | Via TensorRT-LLM |
| In-flight batching | ✅ | |
| Dynamic batching | ✅ | Server-side |
| Ensemble inference | ✅ | Pipeline multiple models |
| Concurrent model versions | ✅ | A/B test, gradual rollout |
| Model repository | ✅ | Pull models at startup |
| Prometheus monitoring | ✅ | |
| Custom metrics | ✅ | |
| Decoupled API | ✅ | Streaming responses |
| Request prioritization | ✅ | QoS support |

#### Typical deployment

```yaml
# Model repository structure
model_repository/
  ensemble_model/
    1/
      model.py (ensemble definition)
  tensorrt_llm/
    1/
      config.pbtxt
      model.engine
  embedding_model/
    1/
      config.pbtxt
      model.plan
```

```bash
docker run --gpus all --shm-size=4g \
    -p 8000:8000 -p 8001:8001 -p 8002:8002 \
    -v /path/to/model_repo:/models \
    nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3 \
    tritonserver --model-repository=/models
```

#### When to choose Triton

- **Enterprise production serving** (SLOs, multi-model, heterogeneous hardware).
- **Multi-model pipelines** (embed → re-rank → generate).
- **Multi-framework environments** (mixing TensorRT, PyTorch, ONNX).
- **High-performance LLM serving on H100/H200 clusters** (TensorRT-LLM path).
- **Need for request prioritization, A/B testing, multi-version serving**.
- **Kubernate-native deployments** (Triton has first-class K8s support).

---

### 8.5 Serving Framework Comparison

| Property | vLLM | llama.cpp Server | TGI | Triton + TRT-LLM |
|---|---|---|---|---|
| **Language** | Python/C++/CUDA | C/C++ | Rust/Python/CUDA | C++/CUDA |
| **Primary hardware** | NVIDIA GPU (+ AMD, Intel) | CPU, Apple, any GPU | NVIDIA GPU | NVIDIA GPU |
| **Best quantization** | AWQ, GPTQ, FP8 | GGUF (all k-quants) | AWQ, GPTQ, FP8 | INT4/FP8 via TRT |
| **Throughput (7B)** | Very high | Moderate | High | Highest (on H100) |
| **Latency (single request)** | Low | Low | Low | Very low |
| **CPU-only support** | No | Yes (best) | No | No |
| **Apple Silicon** | No | Yes (Metal) | No | No |
| **Multi-model serving** | Limited | No (one model) | No (one model) | Yes (full) |
| **Ensemble pipelines** | No | No | No | Yes |
| **OpenAI API compat** | ✅ Native | ✅ Built-in | ✅ Native | Requires NIM |
| **Ecosystem** | OSS community | OSS community | HuggingFace | NVIDIA |
| **License** | Apache 2.0 | MIT | Apache 2.0 | BSD-3 |

---

## 9. Production Considerations

### Model registry & versioning

- Store quantized models alongside their FP16 originals in a model registry (e.g., MLflow, HuggingFace Hub, S3).
- Tag each quantized model with: base model version, quantization method, bit width, calibration dataset, validation metrics.
- Never overwrite a quantized model — always create a new version.

### A/B testing in production

- Serve both FP16 and quantized variants simultaneously.
- Route a fraction of traffic to each variant.
- Compare quality (user feedback, downstream metrics), latency (p50, p95, p99), and throughput.
- Gradual rollout: 5% → 25% → 50% → 100%.

### Monitoring

| Metric | What to Watch | Alert Threshold |
|---|---|---|
| **p50/p99 TTFT** | Time to first token | +30% from baseline |
| **p50/p99 TPOT** | Time per output token | +20% from baseline |
| **Throughput** | Tokens/second | <80% of expected |
| **GPU memory utilization** | VRAM usage | >95% persistent |
| **KV cache utilization** | vs. allocated | >90% (good) |
| **Error rate** | 4xx/5xx responses | >1% |
| **Perplexity (eval)** | Quality drift | +0.5 from baseline |
| **Generation quality** | LLM-as-judge or human eval | Periodic |

### Hardware selection guide

| Deployment | Recommended Hardware | Recommended Setup |
|---|---|---|
| **Single user, local** | RTX 4090 (24 GB) | 7-13B, Q4_K_M GGUF, llama.cpp |
| **Low concurrency API** | A100-40GB or RTX 6000 | 7-13B, AWQ, vLLM |
| **Mid-scale production** | A100-80GB (x2-4) | 70B, AWQ/GPTQ, vLLM, TP=2-4 |
| **High-scale production** | H100-80GB (x8+) | 70B-405B, FP8/INT4, TRT-LLM, TP=8 |
| **Edge / CPU-only** | Modern x86 with AVX-512 | 7B, Q4_K_M, llama.cpp |
| **Apple Silicon** | M2 Ultra / M4 Ultra | 7-13B, GGUF, llama.cpp Metal |
| **Cost-sensitive** | L4 (24 GB) | 7-13B, AWQ, vLLM |

### Memory budget calculation

For a model with `P` parameters, quantized to `B` bits/weight:

```
Model weights:   P × B / 8 bytes
KV cache:        2 × n_layers × n_kv_heads × head_dim × max_seq_len × 2 (FP16) bytes
Activations:     ~20% of model weights (rough estimate)
Overhead:        CUDA context, framework, ~1-2 GB
```

Example — Llama 3 70B, AWQ INT4, seq_len 8192, batch_size 1:
```
Weights:   70B × 0.5 = ~35 GB
KV cache:  2 × 80 × 8 × 128 × 8192 × 2 = ~2.7 GB
Activations + overhead: ~8 GB
Total:     ~46 GB → fits on a single A100-80GB or H100
```

### Cold start / warm-up

- Quantized models may produce garbage tokens for the first few inference steps (cold-start artifacts).
- Always run a warm-up prompt (e.g., "Hello") before production traffic.
- For serverless deployments, keep a warm standby or use model repository pre-loading.

### Throughput vs. latency tradeoffs

| Configuration | TTFT | TPOT | Throughput | Use Case |
|---|---|---|---|---|
| Batch size 1 | Lowest | Moderate | Lowest | Real-time chat |
| Max batch, parallel | Higher | Higher | Highest | Offline batch |
| Chunked prefill | Moderate | Moderate | High | Mixed workloads |
| Speculative decoding | Low | Low | Moderate | Latency-sensitive |

### Security considerations

- **GGUF models are executable files** — only load from trusted sources. A malicious GGUF can execute arbitrary code.
- **Safetensors** (used by AWQ/GPTQ) are safer but not invulnerable.
- Validate model provenance:
  - Check SHA256 hashes against published values.
  - Only load from trusted registries (HuggingFace verified orgs, internal registry).
- Harden the inference server:
  - Run as non-root user.
  - Use network isolation (no external access for the server).
  - Rate-limit API endpoints.

---

## 10. References & Further Reading

### Foundational papers

- **GPTQ:** Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (2023) — [arXiv:2210.17323](https://arxiv.org/abs/2210.17323)
- **AWQ:** Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (2024) — [arXiv:2306.00978](https://arxiv.org/abs/2306.00978)
- **Bitsandbytes / NF4 / QLoRA:** Dettmers et al., "QLoRA: Efficient Finetuning of Quantized Language Models" (2023) — [arXiv:2305.14314](https://arxiv.org/abs/2305.14314)
- **PagedAttention:** Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) — [arXiv:2309.06180](https://arxiv.org/abs/2309.06180)
- **vLLM performance analysis:** "Anatomy of a High-Throughput LLM Inference System" (2025) — [vLLM Blog](https://vllm.ai/blog/2025-09-05-anatomy-of-vllm)
- **KV cache quantization (KVTuner):** Liu et al., "KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization" (2025) — [OpenReview](https://openreview.net/forum?id=zDwipF6h06)
- **MLA:** "TransMLA: Multi-head Latent Attention Is All You Need" (2025) — [arXiv:2502.07864](https://arxiv.org/abs/2502.07864)

### Guides & benchmarks

- "Which Quantization Method is Right for You (GPTQ vs. GGUF vs. AWQ)" — [Maarten Grootendorst](https://newsletter.maartengrootendorst.com/p/which-quantization-method-is-right)
- "Comparing Quantized Performance in Llama Models" (2024) — [LessWrong](https://www.lesswrong.com/posts/qmPXQbyYA66DuJbht/comparing-quantized-performance-in-llama-models)
- "The Complete Guide to LLM Quantization with vLLM" (2026) — [Jarvis Labs](https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks)
- "An Empirical Study of Qwen3 Quantization" (2025) — [arXiv:2505.02214](https://arxiv.org/abs/2505.02214)
- "LLM Inference at scale with TGI" (2024) — [HuggingFace Blog](https://huggingface.co/blog/martinigoyanes/llm-inference-at-scale-with-tgi)
- "Continuous Batching: The Single Biggest GPU Utilization Unlock" (2026) — [Tian Pan](https://tianpan.co/blog/2026-04-09-continuous-batching-llm-inference)

### Tools & repositories

- **vLLM** — [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
- **llama.cpp** — [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)
- **TGI** — [github.com/huggingface/text-generation-inference](https://github.com/huggingface/text-generation-inference)
- **TensorRT-LLM** — [github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)
- **Triton Inference Server** — [github.com/triton-inference-server/server](https://github.com/triton-inference-server/server)
- **AutoGPTQ** — [github.com/PanQiWei/AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)
- **AutoAWQ** — [github.com/casper-hansen/AutoAWQ](https://github.com/casper-hansen/AutoAWQ)
- **bitsandbytes** — [github.com/bitsandbytes-foundation/bitsandbytes](https://github.com/bitsandbytes-foundation/bitsandbytes)
- **ExLlamaV2** — [github.com/turboderp/exllamav2](https://github.com/turboderp/exllamav2)
- **Awesome LLM Quantization** — [github.com/pprp/awesome-llm-quantization](https://github.com/pprp/awesome-llm-quantization)

---

> **End of reference document.** This is a living document — update benchmarks and framework versions as the ecosystem evolves.

# Training Infrastructure

## GPU Selection

| GPU | VRAM | Best For | Notes |
|-----|------|----------|-------|
| RTX 4090 | 24 GB | LoRA/QLoRA 7B–13B, inference | Consumer, no NVLink |
| RTX 5090 | 32 GB | LoRA 13B–30B, QLoRA 70B | Consumer, no NVLink |
| A6000 | 48 GB | Full fine-tune 7B, LoRA 30B–70B | Prosumer, NVLink pair |
| A100 80GB | 80 GB | Full fine-tune 13B–30B, multi-GPU | Datacenter, NVLink |
| H100 80GB | 80 GB | Full fine-tune 30B–70B, RLHF | Datacenter, NVLink, FP8 |
| H200 141GB | 141 GB | Full fine-tune 70B+, long context | Datacenter, NVLink |

## VRAM Budgeting

Rule of thumb for training memory (mixed precision, AdamW):

```
VRAM ≈ params × (2 + 2 + 4 + 4) bytes  [weights + grads + optimizer states]
     + activation memory (batch-dependent)
```

| Technique | VRAM multiplier | Example: 7B model |
|-----------|----------------|-------------------|
| Full fine-tune (FP16 + Adam) | ~16× params | ~112 GB |
| LoRA (rank 64) | ~2.5× params | ~18 GB |
| QLoRA (4-bit + LoRA) | ~0.8× params | ~6 GB |
| Inference only (FP16) | ~2× params | ~14 GB |
| Inference only (4-bit) | ~0.5× params | ~4 GB |

## Multi-GPU Training

| Strategy | When | Framework |
|----------|------|-----------|
| DataParallel (DP) | Single node, quick experiments | PyTorch native |
| DistributedDataParallel (DDP) | Single node, production training | `torchrun --nproc_per_node=N` |
| FSDP / DeepSpeed ZeRO | Model doesn't fit one GPU | `accelerate`, DeepSpeed config |
| Pipeline parallelism | Very large models (>70B) | DeepSpeed, Megatron-LM |
| Tensor parallelism | Latency-critical inference | vLLM, TensorRT-LLM |

### DDP Launch Pattern

```bash
torchrun --nproc_per_node=4 --master_port=29500 train.py \
  --model_name meta-llama/Llama-3-8B \
  --per_device_train_batch_size 4 \
  --gradient_accumulation_steps 8 \
  --bf16 true
```

Effective batch size = `per_device × nproc × grad_accum` = 4 × 4 × 8 = 128.

### FSDP Config (accelerate)

```yaml
# accelerate_config.yaml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_sharding_strategy: FULL_SHARD  # ZeRO-3 equivalent
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_cpu_ram_efficient_loading: true
mixed_precision: bf16
num_processes: 4
```

## Cloud vs On-Prem Decision

| Factor | Cloud (Lambda, RunPod, Vast) | On-Prem |
|--------|------------------------------|---------|
| Utilization < 30% | ✅ Pay per hour | ❌ Idle hardware |
| Utilization > 60% | ❌ Expensive at scale | ✅ Amortizes in ~8 months |
| Data sensitivity | ❌ Data leaves premises | ✅ Stays local |
| Burst capacity | ✅ Scale to 8×H100 on demand | ❌ Fixed ceiling |
| Ops burden | ❌ Zero (managed) | ✅ You maintain cooling, power, drivers |
| Experiment velocity | ✅ Spin up, tear down | ⚠️ Queue contention on shared cluster |

### Cloud Cost Reference (spot/on-demand, 2025)

| GPU | On-demand $/hr | Spot $/hr |
|-----|----------------|-----------|
| A100 80GB | $1.80–$2.50 | $0.90–$1.40 |
| H100 80GB | $3.50–$5.00 | $2.00–$3.00 |
| RTX 4090 | $0.40–$0.70 | $0.25–$0.45 |

## Storage and Data Pipeline

- Training data on NVMe or tmpfs — network storage stalls GPUs
- Checkpoints to object storage (S3/GCS) or NAS — never only local disk
- Use `safetensors` format — faster load, no pickle security risk
- Pre-tokenize datasets for large corpora — tokenization at load time wastes GPU-hours

## Monitoring Training

| Metric | Healthy Range | Red Flag |
|--------|--------------|----------|
| GPU utilization | > 90% | < 70% = data pipeline bottleneck |
| GPU memory | Stable after warmup | Growing = leak (check grad accumulation) |
| Loss curve | Smooth decrease | Spikes = LR too high; plateau = converged or stuck |
| Grad norm | Stable or decreasing | Exploding = reduce LR or add clipping |
| Throughput (samples/sec) | Consistent | Degrading = thermal throttle or I/O |

#!/usr/bin/env python3
"""Eval-set overlap/leakage checker for ml-engineering.

Compares a training corpus against an evaluation corpus and reports how much of
each eval file shares contiguous n-grams with the training text. Overlap between
train and eval is the classic test-set contamination signature: when an eval
example (or a near-copy of one) appears in the training data, large spans of its
text show up verbatim in the training corpus, so the score on that example stops
measuring generalization.

The check is static and conservative. Character shingles of size 8 (the default)
flag duplicated prose, paraphrases, and generated examples; ``--token-ngram``
flags repeated token sequences instead. Small shared fragments from common words
are expected; a single eval file is flagged only when its overlap fraction
exceeds ``--max-overlap-fraction``.

Inputs: one or more files or directories per side. Directories are searched
recursively for regular files (hidden files are skipped). Text is normalized to
lowercase alphanumerics before shingling.

Exit codes:
  0  no eval file exceeds the overlap threshold
  1  one or more eval files exceed the overlap threshold (leakage found)
  2  usage or I/O error (missing path, unreadable file, empty corpus)
"""

import argparse
import json
import re
import sys
from pathlib import Path

NON_ALNUM = re.compile(r"[^a-z0-9]+")


def normalize(text):
    """Lowercase text and reduce it to alphanumeric runs separated by spaces."""
    return NON_ALNUM.sub(" ", text.lower()).strip()


def char_shingles(text, size):
    """All contiguous character windows of ``size`` over normalized text."""
    text = normalize(text)
    if len(text) < size:
        return []
    return [text[i : i + size] for i in range(len(text) - size + 1)]


def token_ngrams(text, size):
    """All contiguous windows of ``size`` tokens over normalized text."""
    tokens = normalize(text).split()
    if len(tokens) < size:
        return []
    return [" ".join(tokens[i : i + size]) for i in range(len(tokens) - size + 1)]


def shingle_document(text, shingle_size, token_size):
    if token_size:
        return token_ngrams(text, token_size)
    return char_shingles(text, shingle_size)


def collect_files(paths):
    """Return the sorted list of readable files under the given paths."""
    found = []
    for raw in paths:
        path = Path(raw)
        if not path.exists():
            raise ValueError(f"path does not exist: {raw}")
        if path.is_file():
            found.append(path)
            continue
        if not path.is_dir():
            raise ValueError(f"not a file or directory: {raw}")
        for child in sorted(path.rglob("*")):
            if child.is_file() and not child.name.startswith("."):
                found.append(child)
    return sorted(found, key=str)


def read_text(path):
    try:
        return path.read_text(encoding="utf-8", errors="replace")
    except OSError as exc:
        raise ValueError(f"cannot read {path}: {exc.strerror}") from exc


def build_report(train_paths, eval_paths, shingle_size, token_size, threshold, show_examples):
    """Compute the per-eval-file overlap report against the train corpus."""
    train_files = collect_files(train_paths)
    eval_files = collect_files(eval_paths)
    if not train_files:
        raise ValueError("training corpus contains no files")
    if not eval_files:
        raise ValueError("evaluation corpus contains no files")

    train_shingle_set = set()
    for path in train_files:
        train_shingle_set.update(shingle_document(read_text(path), shingle_size, token_size))
    if not train_shingle_set:
        raise ValueError(
            "training corpus yields no shingles (text too short or empty); "
            "lower --shingle-size or use --token-ngram"
        )

    files_report = []
    eval_shingle_count = 0
    shared_across_corpus = set()
    leaked = []
    for path in eval_files:
        doc_shingles = set(shingle_document(read_text(path), shingle_size, token_size))
        eval_shingle_count += len(doc_shingles)
        shared = doc_shingles & train_shingle_set
        total = len(doc_shingles)
        fraction = len(shared) / total if total else 0.0
        shared_across_corpus |= shared
        verdict = "LEAK" if fraction > threshold else "ok"
        entry = {
            "path": str(path),
            "shingles": total,
            "shared": len(shared),
            "fraction": round(fraction, 4),
            "verdict": verdict,
            "examples": sorted(shared)[:show_examples],
        }
        if verdict == "LEAK":
            leaked.append(str(path))
        files_report.append(entry)

    return {
        "tool": "check-eval-overlap.py",
        "shingle_mode": f"token-{token_size}" if token_size else f"char-{shingle_size}",
        "train_files": len(train_files),
        "train_shingles": len(train_shingle_set),
        "eval_files": len(eval_files),
        "eval_shingles": eval_shingle_count,
        "shared_shingles": len(shared_across_corpus),
        "max_overlap_fraction": threshold,
        "leaked": leaked,
        "files": files_report,
    }


def format_count(value):
    return f"{value:,}"


def print_human(report):
    threshold_pct = report["max_overlap_fraction"] * 100
    print(
        f"{report['shingle_mode']} overlap check: {report['train_files']} train file(s), "
        f"{format_count(report['train_shingles'])} distinct train shingles; "
        f"{report['eval_files']} eval file(s), {format_count(report['eval_shingles'])} distinct eval shingles"
    )
    for entry in report["files"]:
        label = "LEAK" if entry["verdict"] == "LEAK" else "ok"
        print(
            f"  {entry['path']}: {format_count(entry['shared'])}/{format_count(entry['shingles'])} "
            f"shingles ({entry['fraction'] * 100:.1f}%) overlap -> {label}"
        )
        for example in entry["examples"]:
            print(f"      shared n-gram: {example!r}")
    overall_pct = (
        report["shared_shingles"] / report["eval_shingles"] * 100 if report["eval_shingles"] else 0.0
    )
    if report["leaked"]:
        print(
            f"Leakage detected in {len(report['leaked'])} of {report['eval_files']} eval file(s) "
            f"(threshold {threshold_pct:.1f}%)."
        )
        return
    print(
        f"No leakage: {format_count(report['shared_shingles'])} shared shingle(s) across "
        f"{report['eval_files']} eval file(s) ({overall_pct:.2f}% of eval shingles; "
        f"threshold {threshold_pct:.1f}%)."
    )


def build_parser():
    parser = argparse.ArgumentParser(
        prog="check-eval-overlap.py",
        description=(
            "Detect eval-set leakage: report how much of each eval corpus file shares "
            "contiguous n-grams with the training corpus. Exit 0 when no eval file exceeds "
            "the overlap threshold, 1 when leakage is found, 2 on usage or I/O errors."
        ),
        epilog=(
            "Example: python3 check-eval-overlap.py --train data/train --eval data/eval --json"
        ),
    )
    parser.add_argument(
        "--train",
        nargs="+",
        required=True,
        metavar="PATH",
        help="training corpus: a file or directory (searched recursively); may be repeated",
    )
    parser.add_argument(
        "--eval",
        nargs="+",
        required=True,
        metavar="PATH",
        help="evaluation corpus: a file or directory (searched recursively); may be repeated",
    )
    parser.add_argument(
        "--shingle-size",
        type=int,
        default=8,
        metavar="N",
        help="character shingle size for overlap detection (default: 8)",
    )
    parser.add_argument(
        "--token-ngram",
        nargs="?",
        const=5,
        type=int,
        default=None,
        metavar="N",
        help="use token n-grams of size N instead of character shingles (default when bare: 5)",
    )
    parser.add_argument(
        "--max-overlap-fraction",
        type=float,
        default=0.10,
        metavar="F",
        help="leak threshold: an eval file whose overlap fraction exceeds F is flagged (default: 0.10)",
    )
    parser.add_argument(
        "--show-examples",
        type=int,
        default=3,
        metavar="N",
        help="number of example shared n-grams to print per leaked file (default: 3)",
    )
    parser.add_argument("--json", action="store_true", help="emit a machine-readable JSON report")
    return parser


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)
    if args.shingle_size < 2:
        parser.error("--shingle-size must be >= 2")
    if args.token_ngram is not None and args.token_ngram < 2:
        parser.error("--token-ngram must be >= 2")
    if not 0 < args.max_overlap_fraction <= 1:
        parser.error("--max-overlap-fraction must be in (0, 1]")
    if args.show_examples < 0:
        parser.error("--show-examples must be >= 0")

    try:
        report = build_report(
            args.train,
            args.eval,
            args.shingle_size,
            args.token_ngram,
            args.max_overlap_fraction,
            args.show_examples,
        )
    except ValueError as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2

    if args.json:
        print(json.dumps(report, indent=2))
    else:
        print_human(report)
    return 1 if report["leaked"] else 0


if __name__ == "__main__":
    sys.exit(main())

"""Tests for check-eval-overlap.py.

Covers: clean corpora (exit 0), exact-duplicate eval files (exit 1), partial
overlap above and below the threshold, JSON report shape, directory recursion,
token n-gram mode, the --shingle-size flag, missing-path errors, an empty eval
file, a training corpus too short to shingle, and --help.

Discoverable by both pytest and unittest (unittest.TestCase classes).
"""

import json
import os
import subprocess
import sys
import tempfile
import unittest

SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
CHECKER = os.path.join(SCRIPTS_DIR, "check-eval-overlap.py")

TRAIN_PARAGRAPH = (
    "The quick brown fox jumps over the lazy dog near the riverbank while the "
    "sun sets behind the mountains and the evening train crosses the old bridge."
)
EVAL_SENTENCE = (
    "The quick brown fox jumps over the lazy dog near the riverbank"
)
UNIQUE_FILLER = (
    "blueberry pancakes with maple syrup served at the corner cafe on tuesday "
    "morning while the radio played classical music and the barista practiced "
    "latte art with carefully steamed milk and a gentle hand."
)


def run_checker(args):
    proc = subprocess.run(
        [sys.executable, CHECKER, *args],
        capture_output=True,
        text=True,
        timeout=30,
    )
    return proc.returncode, proc.stdout, proc.stderr


class TempCorpora(unittest.TestCase):
    """Per-test temporary train/ and eval/ directories."""

    def setUp(self):
        self._tmp = tempfile.TemporaryDirectory()
        self.root = self._tmp.name
        self.train = os.path.join(self.root, "train")
        self.eval_dir = os.path.join(self.root, "eval")
        os.makedirs(self.train)
        os.makedirs(self.eval_dir)

    def tearDown(self):
        self._tmp.cleanup()

    def write_train(self, name, content):
        path = os.path.join(self.train, name)
        with open(path, "w", encoding="utf-8") as handle:
            handle.write(content)
        return path

    def write_eval(self, name, content):
        path = os.path.join(self.eval_dir, name)
        with open(path, "w", encoding="utf-8") as handle:
            handle.write(content)
        return path


class TestOverlapDetection(TempCorpora):
    def test_help_exits_zero(self):
        rc, stdout, _ = run_checker(["--help"])
        self.assertEqual(rc, 0)
        self.assertIn("overlap", stdout)

    def test_clean_corpora_exit_zero(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", UNIQUE_FILLER)
        rc, stdout, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 0)
        self.assertIn("No leakage", stdout)
        self.assertIn("ok", stdout)

    def test_exact_duplicate_flags_leak(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", TRAIN_PARAGRAPH)
        rc, stdout, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 1)
        self.assertIn("LEAK", stdout)
        self.assertIn("Leakage detected in 1", stdout)

    def test_partial_overlap_above_threshold_flags_leak(self):
        # Eval reuses the whole train sentence (53 shingles) plus ~210 chars of
        # unique filler, so overlap is roughly 20% and must be flagged.
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", EVAL_SENTENCE + " " + UNIQUE_FILLER[:210])
        rc, stdout, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 1)
        self.assertIn("LEAK", stdout)

    def test_overlap_below_threshold_clean(self):
        # Eval reuses one sentence (~54 shingles) plus unique filler (~210
        # shingles), for roughly 20-25% overlap. A raised threshold must accept
        # it; the default threshold flags the same input (see the partial-overlap
        # test above).
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", EVAL_SENTENCE + " " + UNIQUE_FILLER)
        rc, stdout, _ = run_checker(
            ["--train", self.train, "--eval", self.eval_dir, "--max-overlap-fraction", "0.30"]
        )
        self.assertEqual(rc, 0)
        self.assertIn("No leakage", stdout)


class TestOverlapCli(TempCorpora):
    def test_json_report_shape(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", TRAIN_PARAGRAPH)
        rc, stdout, _ = run_checker(
            ["--train", self.train, "--eval", self.eval_dir, "--json"]
        )
        self.assertEqual(rc, 1)
        report = json.loads(stdout)
        self.assertEqual(report["tool"], "check-eval-overlap.py")
        self.assertEqual(report["train_files"], 1)
        self.assertEqual(report["eval_files"], 1)
        self.assertEqual(len(report["leaked"]), 1)
        entry = report["files"][0]
        self.assertEqual(entry["verdict"], "LEAK")
        self.assertAlmostEqual(entry["fraction"], 1.0, places=4)
        self.assertGreater(entry["shared"], 0)

    def test_directory_recursion(self):
        nested = os.path.join(self.train, "nested")
        os.makedirs(nested)
        with open(os.path.join(nested, "corpus.txt"), "w", encoding="utf-8") as handle:
            handle.write(TRAIN_PARAGRAPH)
        self.write_eval("eval.txt", TRAIN_PARAGRAPH)
        rc, _, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 1)

    def test_multiple_eval_files_partial_leak(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("clean.txt", UNIQUE_FILLER)
        self.write_eval("duplicate.txt", TRAIN_PARAGRAPH)
        rc, stdout, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 1)
        self.assertIn("Leakage detected in 1 of 2", stdout)

    def test_token_ngram_mode(self):
        tokens = "alpha beta gamma delta epsilon"
        self.write_train("train.txt", tokens)
        self.write_eval("eval.txt", tokens)
        rc, stdout, _ = run_checker(
            ["--train", self.train, "--eval", self.eval_dir, "--token-ngram", "3"]
        )
        self.assertEqual(rc, 1)
        self.assertIn("token-3", stdout)

    def test_shingle_size_flag(self):
        # A short shared word is invisible at size 8 but flagged at size 4.
        self.write_train("train.txt", "zebra crossing")
        self.write_eval("eval.txt", "zebra")
        rc_default, _, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        rc_small, stdout, _ = run_checker(
            ["--train", self.train, "--eval", self.eval_dir, "--shingle-size", "4"]
        )
        self.assertEqual(rc_default, 0)
        self.assertEqual(rc_small, 1)
        self.assertIn("LEAK", stdout)


class TestOverlapErrors(TempCorpora):
    def test_missing_path_exit_two(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        rc, _, stderr = run_checker(
            ["--train", self.train, "--eval", os.path.join(self.root, "nope")]
        )
        self.assertEqual(rc, 2)
        self.assertIn("ERROR", stderr)

    def test_empty_eval_file_handled(self):
        self.write_train("train.txt", TRAIN_PARAGRAPH)
        self.write_eval("empty.txt", "")
        rc, _, _ = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 0)

    def test_train_corpus_too_short_exit_two(self):
        self.write_train("tiny.txt", "hi")
        self.write_eval("eval.txt", TRAIN_PARAGRAPH)
        rc, _, stderr = run_checker(["--train", self.train, "--eval", self.eval_dir])
        self.assertEqual(rc, 2)
        self.assertIn("no shingles", stderr)

    def test_missing_train_required_flag(self):
        rc, _, _ = run_checker(["--eval", self.eval_dir])
        self.assertEqual(rc, 2)


if __name__ == "__main__":
    unittest.main()

# Drift response record

- Model and deployed version:
- Signal type: system / input / feature / prediction / outcome
- Detection query or evaluator:
- Window and minimum sample:
- Threshold and first observed time:
- Alert owner and service owner:

## Diagnosis

- Schema/source change checked:
- Seasonality or segment mix checked:
- Label delay and metric integrity checked:
- Training-serving parity checked:
- Critical capability impact:

## Decision

- Action: observe / repair data / recalibrate / retrain / human review / roll back / retire
- Candidate and baseline lineage links:
- Quality, latency, capacity, and cost evidence:
- Approval and expiry (if temporary):
- Post-action validation and result:

# Eval Regression Table

Use this table to track model quality across runs and to make regression triage
auditable. One row per eval case or capability subset, one table per model change
being compared. Every number must name the eval set version that produced it.

## Change Under Evaluation

- Change / run IDs compared: `[fill: e.g. ft-support-lora-013 vs ft-support-lora-014]`
- Date: `[fill: YYYY-MM-DD]`
- Evaluator: `[fill: name or handle]`
- Eval set version: `[fill: eval set revision/hash — do not compare across versions]`
- Leakage check status: `[fill: overlap-check result between train and eval corpora]`

## Results

| Eval case / capability subset | Baseline score | New score | Delta | Noise estimate | Verdict | Notes |
|---|---|---|---|---|---|---|
| `[fill: case or subset name]` | `[fill: baseline value]` | `[fill: new value]` | `[fill: signed delta]` | `[fill: e.g. +/-0.02 from repeat runs]` | `[fill: regression / improvement / no change]` | `[fill: what changed and why]` |
| `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` |

## Analysis

- Aggregate verdict: `[fill: e.g. "overall +3%, but multi-turn subset regressed"]`
- Is the delta beyond noise? `[fill: yes/no and the evidence]`
- Per-example inspection of any regression: `[fill: link or notes for the regressed cases]`
- Root-cause hypothesis: `[fill: data mix drift, overfitting, trade-off, eval artifact, other]`
- Contamination check: `[fill: re-run the overlap checker if a subset dropped suspiciously]`

## Decision

- Decision: `[fill: ship / retrain / tune / roll back]`
- Rollback or follow-up action: `[fill: adapter version to revert to, retrain ticket, etc.]`
- Sign-off: `[fill: who approved and when]`

# Model lineage record

## Decision and ownership

- Model/service and use case:
- Responsible ML engineer:
- Data and platform owners:
- Promotion decision and date:

## Immutable links

| Stage | Revision/hash or URI | Owner | Timestamp | Notes |
|---|---|---|---|---|
| Source snapshot |  |  |  |  |
| Dataset revision |  |  |  |  |
| Feature definitions and temporal cutoff |  |  |  |  |
| Train/validation/test/eval split |  |  |  |  |
| Code and configuration |  |  |  |  |
| Run and artifacts |  |  |  |  |
| Evaluation report |  |  |  |  |
| Registry entry |  |  |  |  |
| Serving image/configuration |  |  |  |  |
| Deployment/endpoint |  |  |  |  |

## Parity and release checks

- Event-time and availability-time rules:
- Offline/online feature replay result:
- Critical capability thresholds:
- System and ML monitoring ready:
- Rollback target and test evidence:
- Missing links or accepted limitations:

# Quantization Decision Record

Fill this record before shipping a quantized model. Quantization is a trade-off,
not a default: the record captures what was measured, what was compared, and why
the chosen variant is safe for the workloads it serves.

## Context

- Model: `[fill: model id and version/commit]`
- Serving engine: `[fill: e.g. llama.cpp, vLLM, TGI, Triton]`
- Hardware: `[fill: GPU model(s) and count, VRAM per GPU]`
- Workload: `[fill: concurrency, max context, latency and throughput targets]`
- Decision date: `[fill: YYYY-MM-DD]`

## Baseline (Unquantized)

- Eval set version: `[fill: eval set revision/hash]`
- Baseline quality: `[fill: metric and value, per capability if applicable]`
- Baseline VRAM / throughput / latency: `[fill: measured numbers, not model-card arithmetic]`

## Candidates Compared

| Candidate | Calibration data | Quality delta | VRAM | Tokens/sec | Latency | Notes |
|---|---|---|---|---|---|---|
| `[fill: e.g. GGUF q8_0]` | `[fill: source and size of calibration set]` | `[fill: delta vs baseline]` | `[fill: GB]` | `[fill: value]` | `[fill: value]` | `[fill: notes]` |
| `[fill: e.g. GGUF q4_k_m]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` |
| `[fill: e.g. AWQ 4-bit]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` |
| `[fill: e.g. GPTQ 4-bit]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` | `[fill: ...]` |

All candidates measured on the same eval set version and the same hardware under
production-like concurrency.

## Quality Threshold

- Required threshold: `[fill: the minimum quality per critical capability that must hold]`
- Regression check: `[fill: which capabilities were re-checked and the results]`
- Capabilities where regression is acceptable: `[fill: list or none]`

## Decision

- Selected variant: `[fill: quantization scheme and size, e.g. GGUF q5_k_m]`
- Rationale: `[fill: why this variant meets the quality threshold within the VRAM/latency budget]`
- Rejected variants and reasons: `[fill: e.g. "q4_k_m dropped code-gen subset by 12%"]`
- Rollback path: `[fill: e.g. "serve the fp16/bf16 weights; keep serving config unchanged"]`

## Follow-Ups

- Re-evaluate when: `[fill: e.g. "new engine version or model release, or after 2 weeks of production traffic"]`
- Production monitoring: `[fill: how quality will be sampled in production]`
- Open questions: `[fill: anything unresolved]`

# Training Run Record

Fill this record for every training or fine-tuning run, and commit it next to the
training code. A run that cannot be reproduced is not evidence. If a field is not
applicable, write `n/a` — do not leave it blank.

## Run Identity

- Run ID: `[fill: unique id, e.g. ft-support-lora-014]`
- Date: `[fill: YYYY-MM-DD]`
- Engineer: `[fill: name or handle]`
- Linked issue / ticket: `[fill: issue number or URL]`

## Model & Data

- Base model: `[fill: model id and commit/hash, e.g. meta-llama/Llama-3.1-8B-Instruct @ 5c2c...]`
- Adapter/output model path: `[fill: artifact path and hash after training]`
- Training dataset version: `[fill: dataset revision/hash, not just the folder name]`
- Dataset size: `[fill: example count]`
- Train / validation split: `[fill: how the split was made and whether leakage was audited]`
- Leakage check: `[fill: result of the eval-set overlap check between train and eval corpora]`

## Configuration

| Setting | Value |
|---|---|
| Method (LoRA / QLoRA / full / other) | `[fill: method]` |
| Seed | `[fill: integer]` |
| Optimizer | `[fill: e.g. AdamW, adamw-torch]` |
| Learning rate | `[fill: value and schedule (cosine/linear/constant)]` |
| Warmup steps | `[fill: count or fraction]` |
| Batch size (per device) | `[fill: value]` |
| Gradient accumulation steps | `[fill: value]` |
| Epochs | `[fill: value or early-stop criterion]` |
| Precision / mixed precision | `[fill: e.g. bf16, fp16, fp32]` |
| LoRA rank / alpha / target modules | `[fill: values, or n/a]` |
| Max sequence length | `[fill: tokens]` |
| Other notable flags | `[fill: anything else that changes training behavior]` |

## Environment

- Framework versions (transformers, peft, torch, trl, ...): `[fill: exact versions]`
- Training hardware: `[fill: GPU model(s), count, VRAM per GPU]`
- CUDA / driver version: `[fill: version]`
- Container / OS image: `[fill: image tag or OS + versions]`

## Evaluation

- Eval set version: `[fill: eval set revision/hash]`
- Eval harness and scoring code: `[fill: script/repo and commit]`
- Baseline score (base model on same eval set): `[fill: metric and value]`
- Final score: `[fill: metric and value, per capability if applicable]`
- Eval notes: `[fill: anything unusual about how the numbers were produced]`

## Artifacts

- Weights / adapter location: `[fill: path or registry reference]`
- Logs / training run URL: `[fill: wandb/mlflow URL or log path]`
- Config file: `[fill: path to the exact config used]`

## Reproduction Check

- [ ] A fresh checkout plus this record reproduces the reported numbers
- [ ] If not reproducible, the gap is documented here: `[fill: what is missing]`

## Decisions & Follow-Ups

- Why this configuration: `[fill: the reasoning behind the key choices]`
- Known issues / next steps: `[fill: anything to try next, or leave blank]`