# 01 — ML systems architecture

A production ML system is mostly *not* the model. The
[Hidden Technical Debt in ML Systems](https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/)
paper's famous point: the ML code is a small box in the middle of a large
system of data collection, feature extraction, serving, and monitoring — and
that surrounding system is where debt accumulates. Design the system, not just
the model.

## 1. The model is the small part

- [Rules of ML](https://developers.google.com/machine-learning/guides/rules-of-ml)
  #1–#4: don't be afraid to ship without ML; design and implement **metrics**
  first; a **simple model with a solid pipeline** beats a sophisticated model on
  a fragile one. Get the end-to-end pipeline (data → features → train → eval →
  serve → monitor) working with a trivial model, then improve the model inside
  that frame.
- Decide the prediction architecture up front: **batch** (precompute, store),
  **online/real-time** (serve on request), or **streaming**. This drives the
  feature and serving design (`rules/05`).

## 2. Training vs serving paths

- The training path (offline, large batch, historical data) and the serving
  path (online, low-latency, current data) are different code paths over the
  same logical features. If they compute features differently, you get
  **train/serve skew** (`rules/02`) — the most common silent production failure.
- Eliminate skew structurally: a **feature store** (e.g. Feast-style) or shared
  transformation code/library used by both paths, so a feature is defined once.

## 3. Feature store

- A feature store centralizes feature definitions, computes and **versions**
  features, serves them consistently to training (offline store) and inference
  (online store), and enables reuse across models. Its core value is
  **training-serving consistency** and point-in-time-correct historical lookups
  (no future leakage in the training join).
- Not every project needs a dedicated feature store — but it needs *one*
  definition of each feature shared by both paths. Reimplementing features in
  the serving app is a skew bug waiting to happen.

## 4. Model registry & artifacts

- A **model registry** (MLflow-style) is the source of truth for trained models:
  versioned artifacts with their metrics, data/code/config lineage, deployment
  label (version **aliases** like `@champion`/`@challenger` — MLflow deprecated
  fixed staging/production/archived stages in favor of aliases and tags), and
  approver. Promotion is an explicit, gated transition (`rules/04`, `rules/05`),
  not a file copy.
- Store the model with everything needed to reproduce and explain it: training
  data reference + hash, feature versions, hyperparameters, code commit,
  environment, and eval report.

## 5. Reproducibility is architectural

- Any model in production must be **rebuildable**: pin data (versioned/hashed),
  code (commit), config, environment (container/lockfile), and seeds (`rules/03`).
  "We can't reproduce the prod model" is a HIGH finding — you can't debug,
  audit, or safely retrain it.

## 6. The Hidden-Technical-Debt anti-patterns

Audit for these (Sculley et al.) — each is real ML debt:

- **Entanglement / CACE** ("Changing Anything Changes Everything"): no input is
  truly independent; adding/removing a feature or changing data shifts the whole
  model. Mitigate with isolation, versioning, and monitoring of model behavior.
- **Undeclared consumers**: other systems silently depend on your model's output
  — changing the model breaks them invisibly. Declare and access-control
  consumers.
- **Feedback loops**: the model influences its own future training data (direct)
  or another model's (hidden). Detect and break them; they make offline metrics
  lie.
- **Data dependencies cost more than code dependencies**: unstable/underutilized
  input signals. Version data sources; drop unused features (`rules/02`).
- **Glue code & pipeline jungles**: most of the system becomes plumbing around a
  general-purpose package; scrappy ETL accreting into an unmaintainable jungle.
  Refactor toward clean, tested components.
- **Configuration debt**: ML systems sprawl config (features, thresholds,
  data selection). Treat config as code — reviewed, versioned, validated.

## Audit checklist

```bash
# Reproducibility — HIGH if a prod model can't be rebuilt
#   Is there a versioned link model → (data hash, code commit, config, env)?
grep -rniE 'mlflow|wandb|model.?registry|model.?card|lineage' . | head
ls -R | grep -iE 'requirements|environment.ya?ml|poetry.lock|uv.lock|Dockerfile|conda'   # env pinned?

# Train/serve consistency — CRITICAL if features computed two ways
grep -rniE 'feature.?store|feast|transform' --include='*.py' . | head
#   Compare training feature code vs serving feature code — same source?

# Glue code / pipeline jungle / config sprawl — MEDIUM
grep -rniE 'TODO|FIXME|HACK|temp|quick' --include='*.py' . | grep -iE 'pipeline|feature|etl' | head
find . -name '*.ipynb' | head        # notebook-only training/serving == debt

# Undeclared consumers / feedback loops — MEDIUM/HIGH (manual)
#   Who reads the model's outputs? Does the model's action affect its future training data?

# Unused features kept in infra — LOW (Rules of ML: drop them)
```

# 02 — Data and features: leakage, skew, versioning

Most ML production failures are data failures, not model failures. The two
deadliest are **data leakage** (offline metrics lie) and **train/serve skew**
(online behavior diverges from offline). Both are silent — the model looks
great and predicts badly.

## 1. Data leakage — the metric-inflating CRITICAL

Leakage is any information in training features that won't legitimately be
available at prediction time (or that encodes the target). It produces
spectacular offline metrics and a model that fails in production.

- **Target/label leakage**: a feature that is a proxy for, or derived after, the
  label (e.g. "account_closed_date" predicting churn; an aggregate computed over
  a window that includes the outcome).
- **Preprocessing leakage**: fitting scalers, encoders, imputers, feature
  selection, or resampling on the **full** dataset before the train/test split —
  the test set leaks into training. Fit transforms on **train only**, inside the
  CV fold (use a `Pipeline` so fit happens per-fold).
- **Temporal leakage**: using future data to predict the past. For time series,
  split **temporally** and ensure every feature is point-in-time correct (only
  data available at the prediction timestamp).
- **Group leakage**: the same entity (user, patient) in both train and test
  inflates metrics — use **grouped** splits.

```python
# BAD — scaler fit on all data before split: test leaks into train
X = StandardScaler().fit_transform(X_all); train, test = split(X)
# GOOD — fit inside the pipeline, per fold
pipe = Pipeline([("scale", StandardScaler()), ("clf", model)])
cross_val_score(pipe, X_train, y_train, cv=TimeSeriesSplit())
```

## 2. Train/serve skew

- Skew = features (or their distribution) differ between training and serving.
  Causes: features computed by different code in the two paths; different data
  sources; time-of-day/freshness differences; a transform applied in training
  but missing in serving.
- Fix structurally (`rules/01`): one feature definition (feature store / shared
  transform) used by both. Then **detect** residual skew by logging served
  feature values and comparing their distribution to training (Rules of ML #29:
  *the best way to make sure you train like you serve is to log features at
  serving time and use them to train*).

## 3. Splits and validation design

- Choose the split to match deployment reality: random for IID; **temporal** for
  anything time-ordered (forecasting, any "predict the future" task); **grouped**
  when rows share an entity. A wrong split silently leaks.
- Keep a held-out test set touched only at the end; use CV on train for model
  selection. Never tune on the test set.

## 4. Feature engineering discipline

- Prefer few, well-understood features; start with directly-observed/reported
  features before learned ones (Rules of ML). Document each feature's source,
  semantics, and freshness.
- **Drop unused/underperforming features** — they're data dependencies that cost
  maintenance and add skew surface (Hidden Technical Debt, `rules/01`).
- Handle missing values and categoricals deliberately and identically in both
  paths; don't let a serving-time unseen category crash or silently mis-encode.

## 5. Data & feature versioning

- Version training **data** (dataset snapshot/hash, DVC/lakeFS-style or a
  warehouse snapshot) and **feature definitions** so a model's inputs are
  reproducible (`rules/01`). "Which data trained this model?" must have an exact
  answer.
- Validate data **schema and distribution** at pipeline entry (types, ranges,
  nullability, expected categories) — catch a broken upstream feed before it
  trains a bad model. (TFX-DV / Great Expectations-style; cross-ref
  `sota-data-engineering` for pipeline contracts.)

## 6. Data governance & PII

- Minimize personal data in features; collect/keep only what has a lawful basis
  and document it (`rules/07`, cross-ref `sota-privacy-compliance`). Don't use
  protected attributes as features unless justified and lawful; beware proxies.
- Track data provenance/consent so you can honor deletion and explain what a
  model was trained on.

## Audit checklist

- [ ] **Train/serve skew**: is the transformation that produces a training feature the
      *same code path* as the one serving it — a shared library or a feature store — or
      two implementations that must be kept in step by hand? Two implementations is the
      finding, whether or not they currently agree.
- [ ] **Point-in-time correctness**: does every training label join features **as of** the
      label's timestamp? A join that picks up feature values computed after the event
      leaks the future into training and inflates offline metrics — it will not reproduce
      in serving, which is how it is usually discovered.
- [ ] If a **feature store** is in use: are offline and online stores written from one
      pipeline, is freshness monitored per feature, and is a feature's serving default
      (on a store miss) the same value training saw for a missing feature?
- [ ] If one is **not** in use: what enforces the two answers above instead? "We are
      careful" is not a mechanism (`rules/01` §3).

```bash
# Preprocessing leakage — CRITICAL
grep -rnE '\.fit(_transform)?\(' --include='*.py' . | grep -vE 'Pipeline|fit\(X_train|fit\(train'  # fit on full data?
grep -rnE 'SMOTE|resample|SelectKBest|StandardScaler|fit_transform' --include='*.py' . # before split?

# Split correctness — CRITICAL/HIGH
grep -rnE 'train_test_split\(' --include='*.py' . | grep -v 'stratify\|TimeSeries\|Group'  # temporal/group needed?
grep -rniE 'TimeSeriesSplit|GroupKFold|GroupShuffle' --include='*.py' . || echo "no temporal/group split — verify IID"

# Train/serve skew — CRITICAL
#   Diff training feature code vs serving feature code; are served features logged for training?
grep -rniE 'feature.?store|feast|log.*feature|skew' --include='*.py' . | head

# Data/feature versioning — HIGH
grep -rniE 'dvc|lakefs|dataset.*hash|snapshot|data.?version' . | head || echo "no data versioning found"

# Data validation at entry — HIGH
grep -rniE 'great_expectations|pandera|tfdv|schema.*valid|expect_' --include='*.py' . || echo "no data validation"

# PII in features — HIGH (cross-ref sota-privacy-compliance)
grep -rniE 'email|ssn|phone|dob|address|name|ip_addr' --include='*.py' . | grep -i feature | head
```

# 03 — Training & experimentation

Training must be **reproducible** and **tracked**, or you can't compare models,
rebuild a production model, or explain a result. The discipline here is the
difference between "we got 0.92 once in a notebook" and a model you can ship and
defend.

## 1. Experiment tracking

- Track every run: code version (commit), data version/hash, feature set,
  hyperparameters, environment, metrics, and artifacts — with a tool (MLflow,
  Weights & Biases, or equivalent), not a spreadsheet. Untracked experiments are
  unreproducible and uncomparable (MEDIUM debt).
- Log enough to answer "why did model B beat model A?" — same eval data, same
  metric definitions, recorded deltas.

## 2. Reproducibility

- Pin and record: random **seeds** (numpy/framework/CUDA where feasible),
  library versions (lockfile), the data snapshot, and config. Containerize the
  training environment.
- Note nondeterminism you can't remove (GPU kernels, parallelism) and bound it —
  report metric variance across seeds rather than a single lucky number.
- A training run should be a parameterized, version-controlled **pipeline**, not
  a hand-run notebook. Notebooks are fine for exploration; production training is
  code (`rules/01`).

## 3. Configuration management

- Treat experiment config as code: versioned, reviewed, validated (typed config
  — Hydra/pydantic-style). Avoid magic numbers scattered across scripts
  (configuration debt, `rules/01`).
- Separate config from code so a run is fully described by (code commit +
  config + data version).

## 4. Hyperparameter search

- Use a principled search (grid for small spaces, random/Bayesian/Optuna-style
  for larger) with a fixed validation protocol; never tune on the test set
  (`rules/02`). Budget it — log all trials to the tracker.
- Guard against overfitting the validation set through many trials: keep a final
  held-out test untouched until the end; consider nested CV for small data.

## 5. Distributed & large-scale training

- For multi-GPU/multi-node: checkpoint regularly (resume from failure), make
  data loading deterministic where it matters, and verify the effective batch
  size / LR scaling. Checkpoints are part of the reproducible artifact set.
- Cost-awareness: training is expensive — track GPU-hours; use spot/preemptible
  with checkpointing; don't retrain from scratch when a warm start or incremental
  update suffices (cross-ref `sota-performance`, `sota-cloud-infrastructure`).

## 6. Start simple, iterate inside the pipeline

- First model: simplest thing that beats the baseline, wired through the full
  pipeline with metrics and monitoring (`rules/01`, `rules/04`). Add complexity
  only when the eval (on slices, vs baseline) justifies it. Most gains come from
  better features and data, not fancier models (Rules of ML).

## Audit checklist

```bash
# Experiment tracking present? — MEDIUM if absent
grep -rniE 'mlflow|wandb|neptune|comet|sacred|tensorboard' . | head || echo "no experiment tracking"

# Seeds / determinism — MEDIUM (reproducibility)
grep -rniE 'seed|random_state|set_seed|manual_seed|deterministic' --include='*.py' . | head \
  || echo "no seeds set — runs not reproducible"

# Config as code — MEDIUM
grep -rniE 'hydra|omegaconf|pydantic|argparse|yaml.safe_load|config' --include='*.py' . | head
grep -rnE '= ?(0\.[0-9]+|[0-9]{2,})' --include='*.py' . | grep -iE 'lr|rate|epoch|batch|threshold' | head  # magic numbers

# Hyperparameter search hygiene — MEDIUM
grep -rniE 'GridSearch|RandomizedSearch|optuna|ray.tune|hyperopt' --include='*.py' . | head
#   verify search uses validation set, not test

# Notebook-only training — LOW/MEDIUM (debt)
find . -name '*.ipynb' | head    # is production training a notebook?

# Checkpointing for long/distributed runs — MEDIUM
grep -rniE 'checkpoint|save_model|state_dict|ModelCheckpoint' --include='*.py' . | head
```

# 04 — Evaluation & validation

A single offline accuracy number is not evidence a model is ready. Evaluate
against the **business objective**, on **slices**, versus a **baseline**, and
gate promotion on automated **validation**. The
[ML Test Score](https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/)
rubric (data / model / infra / monitoring tests) is the backbone.

## 1. Metrics that match the objective

- Pick metrics that reflect the real goal, not convenience. Accuracy is
  misleading on imbalanced data — use precision/recall/F1, PR-AUC, calibration,
  or a cost-weighted metric tied to the decision. For ranking: NDCG/MAP; for
  regression: MAE/RMSE/MAPE chosen for the loss that matters.
- Distinguish the **model metric** (offline) from the **business metric**
  (online) and state how they relate. Rules of ML: the offline metric is a proxy
  — validate it predicts the online outcome (`rules/06`).

## 2. Baselines and ablations

- Always compare to a baseline: a trivial predictor (majority class, last value,
  simple heuristic) and the **current production model**. "0.91 F1" is meaningless
  without "baseline 0.88, prod 0.90". A model that doesn't beat the baseline
  shouldn't ship.
- Ablate: does the new feature/complexity actually help on held-out data, or just
  fit noise?

## 3. Sliced evaluation and fairness

- Report metrics **per slice**, not just aggregate: by segment, geography,
  device, time, and protected/ sensitive groups where relevant. A model that
  wins on average can fail badly on a subgroup — aggregate metrics hide it
  (this is a core ML Test Score / fairness requirement). Set **minimum
  per-slice floors** for high-stakes models.
- Check calibration and error distribution, not just central tendency. Document
  known failure modes.

## 4. Validation gates before promotion

- Promotion to production is **gated** by automated checks (the ML Test Score
  "model development" + "infra" tests), run in CI/CD for ML:
  - metric ≥ threshold **and** no regression vs current production (within
    tolerance), on a fixed eval set;
  - per-slice floors met;
  - data/schema validation passed (`rules/02`);
  - the model is reproducible and registered with lineage (`rules/01`,`rules/03`);
  - a successful **shadow/canary** test where applicable (`rules/05`).
- A model that can't pass the gate doesn't get promoted — no manual override
  without sign-off. Missing a validation gate is HIGH.

## 5. Test the pipeline, not just the model

- The ML Test Score covers **infrastructure tests**: the training pipeline is
  reproducible, the full pipeline is integration-tested, model specs are
  unit-tested, the model can be rolled back, and serving matches training. Skew
  and serving correctness are tested, not assumed (`rules/02`, `rules/05`).
- Test for NaNs/inf, schema conformance, and that the model gives stable outputs
  on a canonical input (a "golden" prediction regression test).

## 6. Offline → online validation

- Offline wins don't guarantee online wins. Validate with an **online
  experiment** (A/B test / interleaving) measuring the business metric, with
  proper sample size and guardrail metrics, before full rollout (`rules/05`,
  `rules/06`). Beware feedback loops contaminating the comparison (`rules/01`).

## Audit checklist

```bash
# Single-metric / no-baseline evaluation — MEDIUM/HIGH
grep -rniE 'accuracy_score|f1_score|roc_auc|rmse|mae' --include='*.py' . | head
#   Is there a baseline + current-prod comparison? A single aggregate number is a finding.

# Sliced / fairness evaluation — HIGH if absent on a high-stakes model
grep -rniE 'group_?by|slice|segment|subgroup|fairness|by_cohort|disaggregat' --include='*.py' . \
  || echo "no sliced evaluation — aggregate metrics only"

# Validation gate before promotion — HIGH if missing
grep -rniE 'threshold|gate|promote|regression|assert.*metric|validate_model' --include='*.py' .ci* .github/ 2>/dev/null | head \
  || echo "no automated promotion gate found"

# Pipeline/model tests (ML Test Score infra) — HIGH
grep -rniE 'def test_|pytest|golden|integration' --include='*.py' . | grep -iE 'model|pipeline|predict|feature' | head

# Online experiment before full rollout — MEDIUM/HIGH
grep -rniE 'a/?b.?test|experiment|interleav|shadow|canary|holdout' . | head
```

# 05 — Deployment & serving

Shipping a model is a deployment, with the same discipline as any production
release — plus ML-specific concerns: serving/training parity, reversibility, and
progressive rollout validated on live traffic.

## 1. Serving pattern: batch vs online vs streaming

- **Batch** — precompute predictions on a schedule, store them, serve from a
  store. Simplest and cheapest when freshness tolerates it.
- **Online/real-time** — model behind a low-latency API; needs the online
  feature path and latency budgets.
- **Streaming** — predictions on an event stream.
- Choose by freshness/latency requirement; don't build a real-time service when
  nightly batch suffices. The choice dictates the feature architecture
  (`rules/01`).

## 2. Packaging & the serving environment

- Package the model with its inference dependencies for a **reproducible serving
  environment** (container; pinned libs). The serving-time framework/version
  must match what the model expects — a silent library mismatch changes outputs.
- Prefer portable, **safe** model formats: ONNX for cross-framework serving,
  `safetensors` over `pickle` (`rules/07`). Use a serving runtime (Triton,
  KServe, BentoML, Ray Serve, or a framework server) rather than ad-hoc Flask
  where scale/standardization matters. Do **not** adopt TorchServe — the repo
  was archived Aug 2025 (no updates or security patches); flag it in existing
  systems and migrate.
- The serving path must apply the **same feature transforms** as training
  (`rules/02`) — share code or a feature store, never reimplement.

## 3. Registry-gated promotion

- Deploy from the **model registry** (`rules/01`): a model is promoted
  staging→production only after the validation gate passes (`rules/04`). The
  deployed artifact is immutable and traceable to its lineage.
- Keep the **previous production model** available for instant rollback.

## 4. Progressive rollout & rollback

- Don't flip 100% of traffic to a new model. Use:
  - **Shadow** (dark launch): run the new model on real traffic without serving
    its predictions; compare outputs/latency to prod safely.
  - **Canary / A-B**: route a small % to the new model, watch guardrail and
    business metrics, ramp up.
- **Rollback must be fast and tested** — one action to revert to the prior
  model. A deployment with no rollback path is HIGH (ML Test Score requires it).

## 5. Operational concerns

- Latency/throughput: meet the budget (batching, hardware/accelerator choice,
  quantization/distillation if needed); load-test before launch (cross-ref
  `sota-performance`).
- Versioned, backward-compatible serving API; handle unseen categories/missing
  features gracefully (don't crash or silently mis-encode). Validate inputs at
  the serving boundary.
- Health checks, autoscaling, and resource limits like any service (cross-ref
  `sota-observability`, `sota-cloud-infrastructure`, `sota-kubernetes`).

## Audit checklist

```bash
# Serving/training parity — CRITICAL if features reimplemented in the server
grep -rniE 'predict|inference|serve' --include='*.py' . | head
#   Confirm the server calls the SAME feature transform code/store as training (rules/02)

# Safe model format & reproducible env — HIGH
grep -rniE 'pickle|joblib|torch.load|cloudpickle' --include='*.py' . | head    # unsafe load? (rules/07)
grep -rniE 'safetensors|onnx|torchscript' --include='*.py' . | head
grep -rniE 'torchserve|torch-model-archiver' . | head    # EOL runtime (archived Aug 2025, no security patches) — HIGH
ls Dockerfile* requirements*.txt poetry.lock uv.lock conda*.yml 2>/dev/null     # serving env pinned?

# Registry-gated deploy + rollback — HIGH
grep -rniE 'registry|stage|promote|production|rollback|previous.*model|champion|challenger' . | head \
  || echo "no registry/rollback path found"

# Progressive rollout — MEDIUM/HIGH
grep -rniE 'shadow|canary|a/?b|traffic.*split|gradual|ramp' . | head || echo "no progressive rollout"

# Serving input validation — MEDIUM
grep -rniE 'validate|schema|pydantic|unseen|unknown.*categor|fillna|missing' --include='*.py' . | head
```

# 06 — Monitoring & drift

A model is not "done" at deploy — it decays. The world shifts away from the
training distribution, and offline metrics can't see it. An **unmonitored model
in production is a HIGH finding**: it can degrade silently for months. Monitor
inputs, outputs, and (as labels arrive) live performance, and trigger
retraining.

## 1. What drifts

- **Data/feature drift (covariate shift)**: the input distribution P(X) moves
  away from training. Detect *before* performance visibly drops.
- **Concept drift**: the relationship P(y|X) changes — the same inputs now map
  to different outcomes (seasonality, behavior change, an external shock).
- **Label/prediction drift**: the output distribution shifts.
- **Performance decay**: the actual metric falls — the ground truth, but you only
  see it once labels arrive (often delayed).

## 2. Detecting drift

- Compare live feature/prediction distributions to a training/reference window
  with statistical tests: **PSI** (Population Stability Index — common rule of
  thumb: >0.1 moderate, >0.25 significant shift; verify thresholds for your
  data), **KS test** for continuous features, chi-square/JS-divergence for
  categoricals. Tools: Evidently, NannyML, WhyLogs, or built-in platform monitors.
- Monitor per-feature and on the prediction distribution; alert on sustained
  shift, not single-batch noise.

## 3. Monitoring performance (the real signal)

- When labels arrive (even delayed), compute the **live metric** and compare to
  the offline expectation and to deployment-time performance. Account for
  **label lag** — design how/when ground truth is collected.
- Where labels are very delayed, use proxy/leading indicators and drift as early
  warning. NannyML-style performance *estimation* can approximate metric decay
  before labels land — treat as estimate, confirm with labels.

## 4. Operational & data-quality monitoring

- Monitor the serving system like any service: latency, throughput, error rate,
  resource use (cross-ref `sota-observability`).
- Monitor **input data quality** at serving: schema conformance, null/NaN spikes,
  range violations, unexpected categories, feature freshness/staleness — a broken
  upstream feature is a common silent failure (ML Test Score monitoring tests).
- Watch for **training/serving skew** continuously by comparing logged served
  features to training (`rules/02`).

## 5. Retraining strategy

- Define the **retraining trigger** explicitly: scheduled (cadence matched to
  drift rate), or **drift/performance-triggered** (retrain when PSI or metric
  crosses a threshold). Don't retrain blindly on a timer if nothing changed, and
  don't wait for a complaint if drift is detected.
- Retraining runs the **same validated pipeline** with gates (`rules/03`,
  `rules/04`) — an automatically-retrained model still passes validation and
  progressive rollout before it serves. Beware feedback loops contaminating new
  training data (`rules/01`).
- Keep model/version history and the ability to roll back a bad retrain
  (`rules/05`).

## 6. Alerting & ownership

- Drift/decay/data-quality alerts route to an owner with a runbook (what to
  check, when to retrain, when to roll back) — cross-ref `sota-observability`
  and `sota-detection-engineering` for alerting discipline. An alert nobody owns
  is noise.

## Audit checklist

```bash
# Any production monitoring at all? — HIGH if none
grep -rniE 'evidently|nannyml|whylogs|drift|psi|kolmogorov|ks_2samp|monitor' --include='*.py' . | head \
  || echo "no drift/perf monitoring found — HIGH"

# Drift detection method present?
grep -rniE 'population_stability|psi|ks_2samp|chi2|js_diverg|wasserstein' --include='*.py' . | head

# Live performance tracking + label lag handling — HIGH
grep -rniE 'ground.?truth|label.*lag|actual|delayed|backfill.*label|live.*metric' --include='*.py' . | head

# Data-quality/freshness monitoring at serving — MEDIUM/HIGH
grep -rniE 'freshness|stale|null.*rate|schema.*serv|nan|range.*check' --include='*.py' . | head

# Retraining trigger defined? — MEDIUM/HIGH
grep -rniE 'retrain|schedule|cron|airflow|trigger|cadence' . | grep -iE 'train|drift|model' | head \
  || echo "no explicit retraining trigger"

# Alerts have an owner/runbook — MEDIUM (manual)
```

# 07 — ML security & governance

ML systems have an attack surface ordinary software doesn't (the data and the
model are attackable), plus regulatory obligations. Map threats with
[MITRE ATLAS](https://atlas.mitre.org/) (adversarial tactics/techniques against
AI systems) and govern with the
[NIST AI RMF](https://www.nist.gov/itl/ai-risk-management-framework). For
prompt-injection/agent threats specific to LLMs, see `sota-code-security`
rules/08 and `sota-llm-engineering`; this file covers classical-ML security.

## 1. Attacks on ML systems (MITRE ATLAS)

- **Training-data poisoning** — adversary corrupts training data (or labels) to
  degrade the model or implant a backdoor/trigger. Control data provenance and
  integrity; validate and monitor training data; restrict who/what can write to
  training sources (`rules/02`).
- **Evasion / adversarial examples** — crafted inputs at inference cause
  misclassification. Validate/bound inputs; consider adversarial training and
  detection for high-stakes models.
- **Model extraction/stealing** — querying the API to clone the model.
  Rate-limit, monitor query patterns, avoid returning raw confidence vectors
  where not needed.
- **Membership inference / model inversion** — inferring whether a record was in
  training, or reconstructing training data, from outputs/confidences. Minimize
  output granularity; consider differential privacy for sensitive training data.
- ATLAS is a living knowledge base (date-based `v2026.MM` releases since May
  2026; techniques now carry platform designations — Predictive AI, Generative
  AI, Agentic AI, Enterprise — check current); use it to enumerate threats
  during design, like ATT&CK for AI.

## 2. ML supply chain

- **Never load an untrusted model artifact.** `pickle`/`joblib`/`cloudpickle`
  execute arbitrary code on load — a malicious model file is RCE. `torch.load`
  defaults to `weights_only=True` (restricted unpickler) since PyTorch 2.6:
  `weights_only=False` or torch <2.6 is still arbitrary code execution, and even
  `weights_only=True` was bypassed to RCE on ≤2.5.1 (CVE-2025-32434, fixed in
  2.6.0) — treat it as hardening, not a trust boundary. Load models only from
  trusted, integrity-verified sources; prefer **`safetensors`**/ONNX (data, not
  code). `weights_only=False` on untrusted input is CRITICAL on sight (`rules/05`).
- Verify integrity/provenance of models and datasets (hashes, signing); pin and
  scan ML dependencies (the PyData/CUDA stack is large attack surface) — cross-ref
  `sota-devsecops`. Beware pre-trained weights/datasets from unvetted hubs — and
  don't treat a passing pickle scan as a trust boundary: blacklist-based scanners
  (picklescan-style, used by major model hubs) were repeatedly bypassed in 2025
  (multiple CVSS 9.3 CVEs: renamed extensions, corrupted ZIP flags, subclassed
  imports). Only trusted sources + integrity verification + safe formats count.
- Protect the model registry and feature store with authn/z; a tampered registry
  ships a tampered model.

## 3. Privacy in ML

- Training data often contains personal data — minimize it, document a lawful
  basis, and honor deletion/retention (cross-ref `sota-privacy-compliance`). A
  model can **memorize** and leak training data; treat models trained on
  sensitive data as sensitive artifacts.
- Consider anonymization/aggregation, and differential privacy where the threat
  model warrants. Don't log raw PII features in monitoring (`rules/06`).

## 4. Governance & documentation

- **Model card** for each production model: intended use, training data summary,
  metrics **including per-slice** (`rules/04`), limitations, ethical
  considerations, owner. It's the artifact auditors and downstream consumers
  read.
- **NIST AI RMF** (Govern / Map / Measure / Manage) for the organizational
  process: identify context and risks, measure them (metrics, fairness,
  robustness), and manage with controls and monitoring. Treat as governance
  scaffolding, not a checkbox.
- **Fairness/bias**: assess disparate performance across protected groups
  (`rules/04`); document findings and mitigations. Bias is both an ethical and,
  increasingly, a legal requirement.

## 5. Regulatory (EU AI Act and beyond)

- The **EU AI Act** imposes obligations by risk tier; **high-risk** systems
  (e.g. employment, credit, biometric, essential services) carry requirements:
  risk management, data governance, technical documentation, logging,
  transparency, human oversight, and accuracy/robustness/cybersecurity. Determine
  your system's tier early — it shapes the whole lifecycle. Verify current
  obligations and timelines against the official text (they phase in over time).
- Sector rules may also apply (credit, health, insurance). Cross-ref
  `sota-privacy-compliance`.

## Audit checklist

```bash
# Unsafe model deserialization — CRITICAL
grep -rnE '\b(pickle\.load|joblib\.load|cloudpickle|torch\.load)\b' --include='*.py' .
grep -rnE 'weights_only\s*=\s*False' --include='*.py' .    # arbitrary code execution on load
grep -rnE 'torch\s*[=<>~!]=+\s*[12]\.[0-5]\b' requirements*.txt pyproject.toml 2>/dev/null  # <2.6: CVE-2025-32434 weights_only bypass
grep -rniE 'safetensors|onnx' --include='*.py' . || echo "consider safetensors/ONNX over pickle"

# Model/data provenance & integrity — HIGH
grep -rniE 'hash|sha256|sign|verify|provenance|checksum' . | grep -iE 'model|dataset|weight' | head \
  || echo "no model/dataset integrity verification"

# Training-data write access / poisoning surface — HIGH (manual)
#   Who can write to training data sources? Is training data validated (rules/02)?

# Extraction/inference exposure — MEDIUM
grep -rniE 'predict_proba|confidence|logits|rate.?limit|throttle' --include='*.py' . | head  # raw scores exposed? rate-limited?

# Governance docs — MEDIUM/LOW
grep -rniE 'model.?card|MODEL_CARD|datasheet|intended.use|limitation' . | head || echo "no model card"
grep -rniE 'nist|ai.?rmf|risk.?assessment|fairness|bias' . | head

# EU AI Act / regulatory tier considered — HIGH for high-risk domains (manual)
grep -rniE 'ai.?act|high.?risk|gdpr|differential.privacy|anonymiz' . | head
```