---
description: SIGMA rule authoring, OpenSearch DSL translation, MITRE ATT&CK mapping, detector creation API, field normalization patterns, false positive suppression
---

# Detection Engineering

Authoring detection rules for OpenSearch Security Analytics: SIGMA format, DSL translation, MITRE mapping, anomaly detector setup, correlation rules, FP suppression. Vendor-neutral methodology with OpenSearch API specifics.

> **Scope**: Detection authoring and translation patterns. Incident escalation lives in `incident-escalation.md`. Detector failure modes live in `detection-safety-patterns.md`.
> **Version range**: OpenSearch 2.x Security Analytics plugin

---

## MITRE ATT&CK Quick Reference

Every detection includes technique ID + tactic + kill chain phase. Tactic alone is insufficient.

| Tactic | ID | Common Techniques (Cloud / Identity Focus) |
|--------|-----|-------------------------------------------|
| Reconnaissance | TA0043 | T1595 (Active Scan), T1596 (Search Open Datasets) |
| Initial Access | TA0001 | T1078 (Valid Accounts), T1190 (Exploit Public App) |
| Credential Access | TA0006 | T1110 (Brute Force), T1110.001 (Password Guessing), T1110.003 (Password Spray), T1110.004 (Credential Stuffing), T1528 (Steal App Access Token) |
| Lateral Movement | TA0008 | T1550 (Use Alternate Auth Material), T1550.001 (App Access Token), T1021 (Remote Services) |
| Privilege Escalation | TA0004 | T1078.004 (Cloud Accounts), T1548 (Abuse Elevation Control) |
| Defense Evasion | TA0005 | T1578 (Modify Cloud Compute Infra), T1070 (Indicator Removal) |
| Exfiltration | TA0010 | T1537 (Transfer Data to Cloud Account), T1530 (Data from Cloud Storage) |

---

## SIGMA Rule Structure

Author detections in SIGMA first (vendor-neutral), then translate to OpenSearch DSL.

```yaml
title: Authentication Password Spray
id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
status: experimental
description: Multiple failed authentication attempts from one source against many users
author: SOC
date: 2026-05-22
logsource:
  category: authentication
  product: {your-auth-product}
detection:
  selection:
    event.outcome: failure
    url.path|contains: '/auth/tokens'
  condition: selection | count(user.name) by source.ip > 5
falsepositives:
  - Misconfigured service accounts
  - Load balancer health checks
level: high
tags:
  - attack.credential_access
  - attack.t1110.003
```

---

## SIGMA → OpenSearch DSL Translation

### Rule-based detection (filter)

```json
POST /_plugins/_security_analytics/detectors
{
  "type": "detector",
  "detector_type": "OTHERS_APPLICATION",
  "name": "auth-failure-detector",
  "enabled": true,
  "schedule": { "period": { "interval": 5, "unit": "MINUTES" } },
  "inputs": [{
    "detector_input": {
      "description": "Authentication failures",
      "indices": ["auth-logs-*"],
      "queries": [{
        "id": "auth-fail-q1",
        "name": "auth_failure_query",
        "query": "event.outcome:failure AND url.path:*auth*tokens*",
        "tags": ["attack.credential_access", "attack.t1110"]
      }]
    }
  }],
  "triggers": [{
    "detector_trigger": {
      "name": "High Failure Rate",
      "severity": "3",
      "types": ["rules"],
      "sev_levels": ["high", "critical"],
      "tags": ["attack.t1110"]
    }
  }]
}
```

### Threshold-based detection (aggregation)

```json
POST /_plugins/_alerting/monitors
{
  "type": "monitor",
  "monitor_type": "bucket_level_monitor",
  "name": "password-spray-monitor",
  "enabled": true,
  "schedule": { "period": { "interval": 5, "unit": "MINUTES" } },
  "inputs": [{
    "search": {
      "indices": ["auth-logs-*"],
      "query": {
        "size": 0,
        "query": {
          "bool": {
            "filter": [
              { "term": { "event.outcome": "failure" } },
              { "wildcard": { "url.path": "*auth*tokens*" } },
              { "range": { "@timestamp": { "gte": "now-5m" } } }
            ]
          }
        },
        "aggs": {
          "source_ips": {
            "terms": { "field": "source.ip", "size": 100 },
            "aggs": {
              "unique_users": {
                "cardinality": { "field": "user.name" }
              }
            }
          }
        }
      }
    }
  }],
  "triggers": [{
    "bucket_level_trigger": {
      "name": "spray_threshold",
      "severity": "2",
      "condition": {
        "buckets_path": { "unique_users": "unique_users" },
        "parent_bucket_path": "source_ips",
        "script": {
          "source": "params.unique_users >= 5",
          "lang": "painless"
        }
      },
      "actions": []
    }
  }]
}
```

---

## Detector Creation API

```bash
# Create Security Analytics detector
POST /_plugins/_security_analytics/detectors

# List existing detectors
GET /_plugins/_security_analytics/detectors/_search
{ "query": { "match_all": {} } }

# Get detector findings
GET /_plugins/_security_analytics/findings/_search?detector_id={id}&startIndex=0&size=20

# Enable / disable detector
POST /_plugins/_security_analytics/detectors/{id}/_start
POST /_plugins/_security_analytics/detectors/{id}/_stop

# List custom rules
GET /_plugins/_security_analytics/rules/_search?pre_packaged=false
{ "query": { "match_all": {} } }
```

---

## Field Normalization (OpenTelemetry Semantic Conventions)

Detection rules reference normalized field names, not raw log fields. Map raw → OTel before authoring rules.

### Example: OpenStack Keystone field mapping

Keystone/WSGI logs ingested through a field-mapping pipeline use `attributes.*` for non-standard fields. Use the same pattern for any authentication source.

| Raw Keystone Field | OTel Mapped Field | Type |
|--------------------|-------------------|------|
| `REMOTE_ADDR` | `source.ip` | `ip` |
| `HTTP_USER_AGENT` | `user_agent.original` | `text` |
| `REQUEST_METHOD` | `http.request.method` | `keyword` |
| `PATH_INFO` | `url.path` | `keyword` |
| `HTTP_X_AUTH_TOKEN` | `attributes.token_id` | `keyword` |
| `wsgi.user_id` | `user.id` | `keyword` |
| `wsgi.project_id` | `cloud.account.id` | `keyword` |
| HTTP response status | `http.response.status_code` | `integer` |

### Cardinality check before adding to aggregation

```bash
POST /auth-logs-*/_search
{
  "size": 0,
  "aggs": {
    "field_cardinality": {
      "cardinality": { "field": "source.ip" }
    }
  }
}
```

High-cardinality fields (>100k unique values) in `terms` buckets cause heap pressure. Use `filter` + `cardinality` instead of `terms` for high-cardinality keys.

---

## Detection Methodology Selection

| Scenario | Methodology | Why |
|----------|-------------|-----|
| Known bad pattern (specific URL abuse, exact CVE signature) | Rule-based | Low latency, deterministic, low FP |
| Abnormal rate (50 failures / 5 min) | Threshold-based | Tunable, fast, explainable |
| Subtle behavioral shift (time-of-day anomaly) | Anomaly-based | Catches slow-burn attacks; higher FP rate |
| Multi-source correlation (auth + network + IAM) | Correlation rule | Required for lateral movement detection |

---

## Anomaly Detector Setup

```bash
POST /_plugins/_anomaly_detection/detectors
{
  "name": "auth-volume-anomaly",
  "description": "Anomalous authentication volume per source IP",
  "time_field": "@timestamp",
  "indices": ["auth-logs-*"],
  "feature_attributes": [{
    "feature_name": "failed_auth_count",
    "feature_enabled": true,
    "aggregation_query": {
      "failed_auths": {
        "filter": { "term": { "event.outcome": "failure" } }
      }
    }
  }],
  "detection_interval": { "period": { "interval": 5, "unit": "Minutes" } },
  "window_delay": { "period": { "interval": 1, "unit": "Minutes" } },
  "category_field": ["source.ip"],
  "shingle_size": 8
}
```

**Cold start**: Model requires `shingle_size * 2` intervals (16 intervals = 80 min at 5-min detection) before producing results. Run historical analysis first on production rollouts:

```bash
POST /_plugins/_anomaly_detection/detectors/{id}/_start
{
  "start_time": 1700000000000,
  "end_time": 1700086400000
}
```

---

## False Positive Suppression

| Pattern | Suppression Approach | Where |
|---------|---------------------|-------|
| Known IP CIDR (internal LBs, monitoring) | `bool.must_not: [{ "cidr": { "field": "source.ip", "value": "10.0.0.0/8" } }]` | Filter in monitor query |
| Service account prefix (e.g., `svc-*`) | `bool.must_not: [{ "wildcard": { "user.name": "svc-*" } }]` | Filter in monitor query |
| Known-good user-agent (health checks) | `bool.must_not: [{ "match": { "user_agent.original": "healthcheck" } }]` | Filter in monitor query |
| Time-window suppression (maintenance) | `range: @timestamp: { gte/lte: ... }` | Add to trigger condition |

### Threshold calibration process

1. Run monitor in dry-run mode for 5 business days
2. Export findings: `GET /_plugins/_security_analytics/findings/_search`
3. Label TPs and FPs manually
4. Adjust threshold (e.g., cardinality count) until FP rate ≤ 10%
5. Document threshold decision in use case lifecycle doc

---

## Correlation Rule Example

```bash
POST /_plugins/_security_analytics/correlation/rules
{
  "name": "auth-then-lateral-movement",
  "correlate": [
    {
      "index": "auth-logs-*",
      "query": "event.outcome:failure AND url.path:*auth*tokens*",
      "category": "credential_access",
      "tags": ["attack.t1110.003"]
    },
    {
      "index": "network-logs-*",
      "query": "destination.port:(22 OR 3389 OR 5985) AND event.action:connection",
      "category": "lateral_movement",
      "tags": ["attack.t1021"]
    }
  ],
  "time_window": 600
}
```

---

## Example: Detection Authoring Walkthrough (Keystone Password Spray)

End-to-end example — substitute your own auth source for the same flow.

1. **Scope**: T1110.003 (Password Spray), Credential Access tactic, severity High.
2. **Field check**: `GET /keystone-logs-*/_mapping/field/source.ip,user.name,event.outcome,url.path` — confirm types are `ip`, `keyword`, `keyword`, `keyword`.
3. **SIGMA rule**: written as in the SIGMA Rule Structure section above.
4. **Translate** to bucket-level monitor (Threshold methodology).
5. **FP suppression**: exclude internal LB CIDR `10.0.0.0/8` and service-account prefix `svc-*`.
6. **Calibrate**: 5-day dry run, label findings, tune `unique_users >= 5` if FP rate exceeds 10%.
7. **Document**: 6-section use case (see `incident-escalation.md` template).

---

## Error → Fix Mapping

| Error | Root Cause | Fix |
|-------|------------|-----|
| `field [X] not found` on detector create | Field absent from index mapping | `GET {index}/_mapping`; add field or adjust rule |
| Detector in FAILED state after creation | Bootstrap conflict on shared index | See `detection-safety-patterns.md` |
| Anomaly detector no results after 2 hours | Cold start not complete | Check `shingle_size × detection_interval`; run historical mode |
| Monitor trigger never fires | Wrong time range in query | Verify `range.@timestamp.gte` matches monitor schedule |
| High FP rate on threshold monitor | Threshold too low or missing IP allowlist | Add CIDR exclusion filter; recalibrate |

---

## See Also

- `detection-safety-patterns.md`: Field alias bootstrap conflicts, chained findings index flood
- `incident-escalation.md`: Severity tiers, escalation packages, KPIs

---
description: OpenSearch Security Analytics failure modes — chained findings index flood, field alias bootstrap conflicts, alias-vs-text conflicts, type coercion, missing nested paths, with concrete API diagnosis and fix commands
---

# Detection Safety Patterns

OpenSearch-specific failure modes that block or destabilize detector creation. Production-observed bugs with concrete diagnosis commands and remediation paths.

> **Scope**: Mapping failures and runtime safety patterns specific to Security Analytics detectors and alerting. General index management (ILM, reindexing) lives in `opensearch-elasticsearch-engineer`.
> **Version range**: OpenSearch 2.x Security Analytics plugin

---

## Failure Mode 1: Chained Findings Index Flood

`chained_findings` monitors create a new query index on every run and delete it after. At high schedule frequency, this floods cluster index count.

### Detection

```bash
# Check index count trend
GET /_cat/indices?v&h=index,creation.date.string,status | grep "chained_findings"

# Count chained_findings query indices
GET /_cat/indices?v | grep -c "chained_findings_queries"

# Identify chained-findings-style monitors
GET /_plugins/_alerting/monitors?size=50 | python3 -c "
import json, sys
data = json.load(sys.stdin)
for m in data.get('monitors', []):
    if m.get('monitor_type') == 'query_level_monitor':
        print(f'CHAINED: {m[\"name\"]}')
"
```

**Symptom**: Index count grows by 1 per monitor run; indices named `chained_findings_queries_{uuid}` appear and disappear. At a 1-minute schedule, this can generate 23k+ index create/delete operations per hour.

### Fix: Static Query Indices

Replace per-run query index creation with a static, reused index.

```bash
# 1. Create static query index with explicit mapping
PUT /siem-chained-findings-queries
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "finding_id": { "type": "keyword" },
      "detector_id": { "type": "keyword" },
      "queries": { "type": "object" }
    }
  }
# In production: also disable dynamic mapping ("auto-map-new-fields": false)
# to prevent unintended field creation under load
}

# 2. Update chained findings monitor to use static index
PUT /_plugins/_alerting/monitors/{monitor_id}
{
  ...existing monitor config...,
  "inputs": [{
    "search": {
      "indices": ["siem-chained-findings-queries"],
      ...
    }
  }]
}

# 3. Verify: index count stabilizes
GET /_cat/count/siem-chained-findings-queries?v
```

**Root cause**: The Security Analytics plugin's chained findings implementation creates a new backing index for each evaluation to hold intermediate query results. With no TTL or reuse, each run adds a new index permanently (or until manual cleanup). Static indices are not the default — configure them explicitly.

---

## Failure Mode 2: Field Alias Bootstrap Conflict

Security Analytics field alias bootstrap is destructive. When a detector is created targeting a shared datastream, the bootstrap process writes field aliases that can conflict with existing alias mappings from previous detector runs.

### Detection

```bash
# Check existing aliases on the target index
GET /auth-logs-*/_mapping | python3 -c "
import json, sys
data = json.load(sys.stdin)
for idx, mapping in data.items():
    props = mapping.get('mappings', {}).get('properties', {})
    for field, fdef in props.items():
        if fdef.get('type') == 'alias':
            print(f'{idx}: {field} -> {fdef.get(\"path\")}')
"

# Check for alias conflicts on a specific field
GET /auth-logs-*/_mapping/field/source.ip

# Attempt to add an alias (fails with a conflict if one exists)
PUT /auth-logs-*/_mapping
{
  "properties": {
    "src_ip_alias": {
      "type": "alias",
      "path": "source.ip"
    }
  }
}
```

**Error message**: `mapper_parsing_exception: failed to parse mapping [_doc]: Cannot update to alias mapping, a non-alias mapping exists at [field_name]`

### Fix Option A: Detection-Owned Index (Preferred)

Create a dedicated index for the detector. Security Analytics bootstraps aliases on this index without conflicting with the ingestion datastream.

```bash
# 1. Create detection-owned index with explicit mapping
PUT /siem-detection-auth
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "siem-short-retention"
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "source.ip": { "type": "ip" },
      "user.name": { "type": "keyword" },
      "event.outcome": { "type": "keyword" },
      "url.path": { "type": "keyword" },
      "http.response.status_code": { "type": "integer" }
    }
  }
# In production: also disable dynamic mapping
}

# 2. Create reindex pipeline copying relevant fields from the ingestion index
PUT /_ingest/pipeline/siem-detection-auth-copy
{
  "processors": [
    { "set": { "field": "siem_processed", "value": true } }
  ]
}

# 3. Update detector to use detection-owned index
POST /_plugins/_security_analytics/detectors/{detector_id}
{
  ...
  "inputs": [{ "detector_input": { "indices": ["siem-detection-auth"] }}]
}
```

### Fix Option B: Reindex to Clean Index

Use when the shared datastream already has conflicting aliases and cannot be changed in place.

```bash
# 1. Create new clean index
PUT /auth-logs-clean-v2
{ ...explicit mapping without alias conflicts... }

# 2. Reindex (async)
POST _reindex?wait_for_completion=false
{
  "source": { "index": "auth-logs-*" },
  "dest": { "index": "auth-logs-clean-v2" }
}

# 3. Monitor reindex progress
GET _tasks/{task_id}

# 4. Verify no failures (response.task.status.failures must be empty)

# 5. Atomic alias swap
POST _aliases
{
  "actions": [
    { "remove": { "index": "auth-logs-*", "alias": "auth-logs" } },
    { "add": { "index": "auth-logs-clean-v2", "alias": "auth-logs" } }
  ]
}
```

**Why `PUT _mapping` cannot fix this**: The field alias bootstrap writes a mapping entry of type `alias`. Once an index has a field mapped as `alias`, it cannot be changed to any other type — not even to another `alias` pointing to a different path. Reindex is the only resolution path.

---

## Failure Mode 3: Alias-vs-Text Conflict

Alias fields and text fields cannot coexist at the same path.

### Detection

```bash
# Check field type
GET /auth-logs-*/_mapping/field/source.ip

# Response showing conflict:
# { "auth-logs-000001": { "mappings": { "source.ip": { "mapping": { "source.ip": { "type": "text" } } } } } }
# Expected: "type": "alias" or "type": "ip"
```

**Error when creating detector**: `Validation Failed: ... field [source.ip] of type [alias] cannot be used in aggregations`

### Diagnosis Table

| Symptom | Type | Fix |
|---------|------|-----|
| Field is `text` where `keyword` or `ip` expected | Type mismatch | Reindex with corrected mapping |
| Field is `alias` where `text` expected | Alias-vs-text conflict | Reindex; alias cannot be removed via `PUT _mapping` |
| Field path resolves to `null` | Missing nested path | Check parent object exists in mapping |
| Aggregation on `text` field throws exception | Missing `keyword` sub-field | Add `.keyword` sub-field; reindex if data exists |
| `ip` field stores non-IP string | Type coercion failure | Add `ignore_malformed: true` or fix ingestion pipeline |

### Fix: Type Coercion on IP Field

```bash
# Diagnose malformed IP values
POST /auth-logs-*/_search
{
  "query": {
    "bool": {
      "must_not": [
        { "exists": { "field": "source.ip" } }
      ],
      "filter": [
        { "exists": { "field": "REMOTE_ADDR" } }
      ]
    }
  },
  "size": 10,
  "_source": ["REMOTE_ADDR"]
}

# Tolerate malformed IPs temporarily
PUT /auth-logs-clean-v2/_mapping
{
  "properties": {
    "source.ip": {
      "type": "ip",
      "ignore_malformed": true
    }
  }
}
```

---

## Failure Mode 4: Missing Path in Nested Object

Security Analytics rules referencing nested paths (e.g., `attributes.user.name`) fail when the parent object is unmapped.

### Detection

```bash
# Check if path resolves in mapping
GET /auth-logs-*/_mapping/field/attributes.user.name

# Empty response means path is not explicitly mapped
# Verify data exists at this path
POST /auth-logs-*/_search
{
  "query": { "exists": { "field": "attributes.user.name" } },
  "size": 1
}
```

### Fix

```bash
PUT /auth-logs-*/_mapping
{
  "properties": {
    "attributes": {
      "properties": {
        "user": {
          "properties": {
            "name": { "type": "keyword" }
          }
        }
      }
    }
  }
}
```

This succeeds only on indices without conflicting mappings at `attributes`. If `attributes` is already mapped as `flattened` or `object` with dynamic mapping disabled, the PUT will fail. Reindex is the resolution.

---

## Error → Fix Mapping

| Error / Symptom | Root Cause | Diagnosis Command | Fix |
|-----------------|------------|-------------------|-----|
| `chained_findings_queries_*` indices accumulating | Chained findings monitor creates index per run | `GET _cat/indices \| grep chained_findings` | Use static query index |
| Detector stuck in FAILED state after creation | Field alias bootstrap conflict on shared datastream | `GET {index}/_mapping \| python3 ...` (check alias type) | Detection-owned index or reindex |
| `Cannot update to alias mapping` on `PUT _mapping` | Existing non-alias mapping at field path | `GET {index}/_mapping/field/{name}` | Reindex to clean index |
| `mapper_parsing_exception` on indexing | Type mismatch (IP field receiving hostname string) | Query `ignore_malformed` docs | Add `ignore_malformed: true`; fix upstream |
| Aggregation exception on `text` field | Missing `keyword` sub-field | `GET {index}/_mapping/field/{name}` | Add `.keyword` sub-field; reindex if data exists |
| `field_not_found` in Security Analytics rule | Field absent from index mapping | `GET {index}/_mapping` | Add field to mapping or adjust rule |
| Index count grows without bound | Chained findings index flood | `GET _cat/indices \| wc -l` (trend) | Static query index (see above) |

---

## Detection Commands Reference

```bash
# Check index count (run repeatedly to detect growth)
GET /_cat/indices?v&s=creation.date:desc | head -20

# Find all alias-type fields across SIEM indices
curl -s "$OS_HOST/siem-*/_mapping" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for idx, m in data.items():
    for field, fdef in m.get('mappings', {}).get('properties', {}).items():
        if fdef.get('type') == 'alias':
            print(f'{idx}: {field} -> {fdef.get(\"path\")}')
"

# Verify field types before creating a detector rule
GET /auth-logs-*/_mapping/field/source.ip,user.name,event.outcome

# Check for chained-findings-style monitor configuration
GET /_plugins/_alerting/monitors/_search
{
  "query": {
    "match": { "monitor.monitor_type": "query_level_monitor" }
  }
}

# Validate reindex had no failures
GET _tasks/{task_id}
# Confirm: response.task.status.failures == 0
```

---

## See Also

- `detection-engineering.md`: Detector creation API, SIGMA translation, field normalization
- `incident-escalation.md`: Escalation content requirements, KPIs

---
description: Severity tier framework, escalation criteria, 9-field escalation package, handoff protocol, after-hours procedure, 6-section use case lifecycle template, KPI definitions
---

# Incident Escalation

SOC incident response framework: severity tiers and SLAs, escalation criteria, 9-field package gate, handoff RACI, use case lifecycle, KPI definitions. Vendor-neutral; SLA defaults are configurable per organization.

> **Scope**: SOC escalation procedures, use case documentation, KPI measurement. Source basis: SOC incident response best practices.
> **Binding**: SLAs are organizational commitments tracked as KPIs.

---

## Severity Tiers and SLAs

Default tier framework. Configure SLA values per organizational commitments; the four-tier structure and escalation logic are stable.

| Severity | Description | Default Initial Response | Default Max Processing |
|----------|-------------|--------------------------|------------------------|
| Very High | Catastrophic / immediate / long-term damage. Suspected TP. Immediate IR notification. | 15 minutes | 1 hour |
| High | High-profile, immediate + mid-term damage to multiple services. Coordinate with security manager. | 30 minutes | 2 hours |
| Medium | Immediate damage to single service; potential to spread. Managed by service owner. | 1 hour | 8 hours |
| Low | Lower risk / impact. Follow runbooks at analyst discretion. | 2 hours | 24 hours |

**SLA breach is itself an escalation trigger**: delayed triage time or sustained high FP rate both meet escalation criteria.

---

## Escalation Criteria

Escalate when ANY of these conditions is met:

| Trigger | Description |
|---------|-------------|
| High or Critical severity | Business impact, data exfiltration, ransomware behavior, or known threat-actor TTPs |
| Confirmed/suspected CIA impact | Confidentiality, Integrity, or Availability threatened or compromised |
| Correlated multi-vector alerts | Lateral movement + privilege escalation indicators from multiple sources |
| SLA / KPI breach | Delayed triage, sustained high FP rate, failed containment within SLA window |
| Unresolvable / ambiguous scope | Analyst cannot determine impact, scope, or attribution without internal context |
| Forensic or legal requirement | HR, legal, or regulatory disclosure may be required |

**Alert becomes incident** when: confirmed CIA impact OR multiple correlated indicators suggest coordinated attack.

---

## 9-Field Escalation Package (Required)

All 9 fields are mandatory. Missing fields fail the QA gate and reduce escalation quality score.

| # | Field | Format / Notes |
|---|-------|----------------|
| 1 | Ticket ID + link | Ticket system URL |
| 2 | Alert summary + link | SIEM alert ID and OpenSearch Dashboards link |
| 3 | MITRE ATT&CK mapping | Technique ID (T####.###) + tactic + kill chain phase |
| 4 | Timeline of events | Chronological; first event → detection → current state |
| 5 | Investigation actions taken | Commands run, systems queried, remediation attempted |
| 6 | Initial impact analysis | Services affected, data potentially exposed, blast radius |
| 7 | Evidence artifacts | Log snippets, IPs, hashes, screenshots, query results |
| 8 | Containment recommendation | Specific action (block IP, revoke credential, isolate host) |
| 9 | 5 Ws | Who (actor), What (action), When (timestamp), Where (target), How (vector) |

### Validation checklist

```
[ ] Ticket ID present and linked
[ ] Alert ID + dashboard link present
[ ] MITRE technique ID (T####.###) specified
[ ] MITRE tactic category specified
[ ] Timeline spans first event → now
[ ] At least 2 investigation actions documented
[ ] Impact: affected service(s) named
[ ] At least 1 evidence artifact (log line, IP, hash)
[ ] Containment action specified or "no action recommended" stated
[ ] Who / What / When / Where / How all answered
```

Run this checklist mechanically before escalation submission — every field, every time.

---

## Handoff Protocol

Once escalation is accepted by internal stakeholders:

1. Internal incident manager **assumes ownership** of the case
2. Source SOC **continues supporting investigation** unless explicitly released
3. All updates documented in agreed system (ticketing platform)
4. Closure requires **formal confirmation from internal cybersecurity teams** — source SOC remains in support role

**Communication channels** are organization-specific. Document the on-call channel, the alert channel, and the case-management system in your runbook.

### RACI for escalation

| Activity | SOC Analyst | SOC IM | Internal Cybersecurity | Service Owner |
|----------|-------------|--------|------------------------|---------------|
| Alert Triage | R | S | A/C | I |
| Escalation | R | A | S/I | C |
| IR Coordination | I | R/A | C | C |
| KPI Reporting | S | R | A/C | I |

---

## After-Hours Escalation Procedure

For escalations outside standard hours:

1. Escalate via designated on-call channel (phone, ticketing system, chat)
2. Log the escalation attempt with timestamp
3. If no acknowledgement: retry until confirmation received
4. Owner of on-call list keeps contacts current

---

## Use Case Lifecycle Template (6 Sections)

Produce use case documentation in this structure for every new detection.

### Section 1: General Info

| Field | Value |
|-------|-------|
| Title | `{Attack Category} - {Specific Scenario}` |
| Unique ID | `{CATEGORY}-{SUBCATEGORY}-{TARGET}-{NNN}` (e.g., BRUTE-PWD-AUTH-001) |
| Version | Semver (1.0, 1.1, etc.) |
| Owner | Name + team |
| Status | Active / Under Review / Deprecated / Planned |
| Severity | Very High / High / Medium / Low |
| Tags | MITRE tactic, attack vector keywords |

### Section 2: Context

| Field | Value |
|-------|-------|
| Business Relevance | Why this detection matters to operations |
| MITRE Mapping | Tactic name (link to ATT&CK) |
| MITRE Technique | T-ID (link to specific technique) |
| Risk Mapping | Internal risk register item |
| Compliance Mapping | NIST CSF / ISO 27001 / regulatory control |

### Section 3: Outcomes

| Field | Value |
|-------|-------|
| Goal | What the detection prevents or detects |
| Link to Playbook | Runbook link (analysis + IR steps for Tier-1) |

### Section 4: Detection Logic

| Field | Value |
|-------|-------|
| Necessary Logs | Log sources required |
| Detection Methodology | Rule-based / Threshold-based / Anomaly-based / Correlation |
| Frequency | Monitor interval (e.g., every 5 minutes) |
| Link to Detection Rule | SIEM rule ID or repository link |
| Link to Filters | Allowlist / suppression filter links |

### Section 5: Continuous Improvement

| Field | Value |
|-------|-------|
| Expected KPIs | FP rate target, response-time link to SLA tier, re-run threshold |
| Last Review Date | ISO date |
| Next Review Date | ISO date (max 1 year out) |
| Retirement Criteria | Conditions under which this use case is deprecated |

### Section 6: Analyst Support

| Field | Value |
|-------|-------|
| Timeline Link | OpenSearch Dashboards link showing historical alerts |
| Case Links | Links to past incidents triggered by this detection |
| Vendor Documentation | Links to source / OS docs for affected log fields |

---

## Tier-1 vs Tier-2 SOC Workflow

| Activity | Tier-1 | Tier-2 |
|----------|--------|--------|
| Approach | Runbook-driven, structured | Deep-dive, unstructured |
| Scope | Triage, enrichment, initial categorization | Investigation, attribution, scoping |
| Output | Triaged alert (escalate / close / monitor) | Containment plan, root-cause analysis |
| Tools | Dashboards, runbook commands | Raw queries, forensic tools, threat intel |

A use case fails if its runbook does not give Tier-1 a clear escalate-or-close decision in <30 minutes.

---

## KPI Definitions and Measurement

| KPI | Definition | Measurement | Default Target |
|-----|-----------|-------------|----------------|
| Time to Detect (TTD) | Event occurrence → alert detection | `alert.triggered_at - event.first_seen` | Tier-dependent |
| Time to Respond (TTR) | Alert detection → first mitigation action | `first_action.timestamp - alert.triggered_at` | Within SLA window |
| Mean Time to Resolution (MTTR) | Detection → case closure | Mean of `(case.closed_at - alert.triggered_at)` | Track trend, reduce |
| False Positive Rate | % alerts closed as FP | `FP_count / total_alerts` per period | ≤ 10% per use case |
| Escalation Quality Score | % escalations meeting all 9-field QA standard | `complete_escalations / total_escalations` | ≥ 90% |
| Log Source Onboarding Rate | New log sources added per period | Count per sprint/month | Track vs. coverage roadmap |
| Use Case Improvement Rate | Rules created/improved per period | Count from rule change log | Track vs. gap analysis |

### KPI dashboard query (MTTR example)

```json
POST /siem-cases-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-30d" } }
  },
  "aggs": {
    "mttr_stats": {
      "stats": {
        "script": {
          "source": "(doc['closed_at'].value.toInstant().toEpochMilli() - doc['detected_at'].value.toInstant().toEpochMilli()) / 60000",
          "lang": "painless"
        }
      }
    }
  }
}
```

---

## See Also

- `detection-engineering.md`: SIGMA authoring, detector creation, MITRE mapping
- `detection-safety-patterns.md`: Mapping errors and detector failure modes

# Phase 2: DISPATCH — Agent Dispatch Prompts

> **Load when**: Phase 2 (DISPATCH) begins, after Phase 1 segmentation is complete.
> **Purpose**: Per-domain agent dispatch prompts for parallel package review.

---

## Dispatch Principles

- **Read the actual code.** Agents MUST use the Read tool to read every .go file in their assigned packages. Read every file directly rather than guessing from names or grep output.
- **Use gopls MCP tools when available**: `go_workspace` to detect workspace structure, `go_file_context` after reading each .go file for intra-package dependency understanding, `go_symbol_references` to verify type usage across packages (critical for export decisions), `go_package_api` to inspect package APIs, `go_diagnostics` to verify any fixes.
- **Real review, not checklists.** The primary question for every function is: "Would this pass review?" not "does it follow a checklist." A real reviewer reads code holistically and reacts to architectural issues, not just mechanical patterns.
- **Segment by package, not by concern.** Dispatch agents by package groups, NOT by concern area. Each agent reviews its packages holistically (errors + architecture + patterns + tests together), exactly like a real PR review.
- **Code-level findings only.** Every finding MUST include the actual code snippet and a concrete fix showing what it should become.

---

## Standard Dispatch Prompt

Use this prompt verbatim for each dispatched agent, substituting `[list of packages with full paths]`:

```
You are reviewing code in an SAP Converged Cloud Go project against established
review standards. Your job is to find things that would actually be commented on
or rejected in a PR.

PACKAGES TO REVIEW: [list of packages with full paths]

Read EVERY .go file in these packages using the Read tool. For each file:

1. **Over-engineering** (Lead Reviewer's #1 Concern)
   - Interfaces with only one implementation? → "Just use the concrete type." Project convention: only create interfaces when there are 2+ real implementations.
   - Wrapper function that adds nothing? → "Delete this, call the real function"
   - Struct for one-time JSON? → "Use fmt.Sprintf + json.Marshal" (per project convention)
   - Option struct for constructor? → "Just use positional params." Project convention uses 7-8 positional params, always positional params.
   - Config file/viper? → "Use osext.MustGetenv." Project convention uses environment variables exclusively. Pure env vars only.

2. **Dead code**
   - Exported functions with no callers outside the package? Use Grep to check: `grep -r "FunctionName" --include="*.go"`. If no callers exist, flag it.
   - Interface methods unused
   - Fields set but unread
   - Entire packages imported but barely used
   - "TODO: remove" comments on code that should already be gone

3. **Error messages** (Secondary Reviewer's #1 Concern)
   - `http.Error(w, "internal error", 500)` — useless to the caller
   - Error wrapping: uses %w when caller needs errors.Is/As, %s with .Error() to intentionally break chain
   - Message format: "cannot <operation>: %w" or "while <operation>: %w" with relevant identifiers
   - Would a user/operator reading this know what to do?
   - "internal error" with no context = CRITICAL
   - Return the primary error; log secondary/cleanup errors. Primary error returned, secondary/cleanup errors logged.

4. **Constructor patterns**
   - Constructor should be `NewX(deps...) *X` — returns infallibly (no error) (construction is infallible)
   - Uses positional struct literal init: `&API{cfg, ad, fd, sd, ...}` (no field names)
   - Injects default functions for test doubles: `time.Now`, etc.
   - Override pattern for test doubles: fluent `OverrideTimeNow(fn) *T` methods

5. **Interface contracts**
   - If the package implements an interface from another package: Read the interface definition
   - Check if the implementation actually satisfies the contract
   - Does it return correct error types? Default values where errors are expected?
   - Interfaces should be defined in the consumer package, not the implementation package

6. **Copy-paste structs**
   - Two structs with the same fields (one for internal, one for API response)?
   - Handler functions that are 90% identical (extract the common pattern)
   - Duplicated validation logic

7. **HTTP handler patterns** (Must match keppel patterns)
   - Handlers: methods on *API with `handleVerbResource(w, r)` signature
   - Auth: called inline at top of handler, NOT middleware
   - JSON decode: `json.NewDecoder` + `DisallowUnknownFields()`
   - Responses: `respondwith.JSON(w, status, map[string]any{"key": val})`
   - Internal errors: `respondwith.ObfuscatedErrorText(w, err)` — hides 500s from clients
   - Route registration: one `AddTo(*mux.Router)` per API domain, composed via `httpapi.Compose`

8. **Database patterns**
   - SQL queries as package-level `var` with `sqlext.SimplifyWhitespace()`
   - PostgreSQL `$1, $2` params (always `$1, $2` (PostgreSQL syntax))
   - gorp for simple CRUD, raw SQL for complex queries
   - Transactions: `db.Begin()` + `defer sqlext.RollbackUnlessCommitted(tx)`
   - NULL: `Option[T]` (from majewsky/gg/option), not `*T` pointers

9. **Type patterns**
   - Named string types for domain concepts: `type AccountName string`
   - String enums with typed constants (NOT iota): `const CleanSeverity VulnerabilityStatus = "Clean"`
   - Model types use `db:"column"` tags; API types use `json:"field"` tags — separate types
   - Pointer receivers for all struct methods (value receivers only for tiny data-only types)

10. **Logging patterns**
    - `logg.Fatal` ONLY in cmd/ packages for startup failures
    - `logg.Error` for secondary/cleanup errors (only for secondary/cleanup errors)
    - `logg.Info` for operational events
    - Use logg package for all logging
    - Panics only for impossible states, annotated with "why was this not caught by Validate!?"

11. **Mixed approaches** (Pattern consistency)
    - Some handlers return JSON errors, others return text/plain
    - Some constructors panic on nil args, others return errors
    - Some packages use logg, others use log

For EACH finding, output:

### [MUST-FIX / SHOULD-FIX / NIT]: [One-line summary]
**File**: `path/to/file.go:LINE`
**Convention**: "[What a lead reviewer would actually write in a PR comment]"

**Current code**:
```go
[actual code from the file, 3-10 lines]
```

**Should be**:
```go
[what the code should look like after fixing]
```

**Why**: [One sentence explaining the principle]

---

SEVERITY GUIDE:
- MUST-FIX: Would block the PR (data loss, interface violation, wrong behavior)
- SHOULD-FIX: Would get a strong review comment (dead code, copy-paste, bad errors)
- NIT: Would get a comment but not block (style, naming, minor simplification)

Skip:
- Generic Go best practices (t.Parallel, DisallowUnknownFields, context.Context first)
- Things that are actually fine but could theoretically be "better"
- Suggestions that add complexity without clear benefit

Focus on:
- Real over-engineering (lead reviewer's #1 concern)
- Actually useless error messages (secondary reviewer's #1 concern)
- Dead code that should be deleted
- Interface contract bugs
- Constructor/config patterns that diverge from keppel patterns
- Inconsistent patterns within the same repo
```

---

## Dispatch Instruction

**Dispatch all agents in a single message using the Task tool with `subagent_type=golang-general-engineer`.**

Gate: All agents dispatched. Proceed to Phase 3.

# Agent Dispatch Prompts

This file contains the shared preamble and all 10 domain-specialist agent specifications used during Phase 2 (DISPATCH).

---

## Shared Preamble

Include this block in every agent prompt:

```
REFERENCE FILES TO READ FIRST (mandatory):
1. Read ~/.claude/skills/engineering/go-patterns/references/sapcc-conventions/sapcc-code-patterns.md
   (Focus on sections listed below, but skim all for context)
2. Read [domain-specific reference file]

REPO TO REVIEW: [current working directory]

SCAN METHOD:
- Use Glob to find all .go files: **/*.go (excluding vendor/)
- Use Read to examine each file
- Use Grep to search for specific patterns across all files

OUTPUT FORMAT for each finding:
### [CRITICAL|HIGH|MEDIUM|LOW]: [One-line summary]
**File**: `path/to/file.go:LINE`
**Rule**: §[section].[subsection]: [rule name]
**Convention**: "[What the lead reviewer would write in a PR comment]"

REJECTED (current code):
```go
[actual code, 3-10 lines]
```

CORRECT (what it should be):
```go
[fixed code]
```

**Why**: [One sentence]
---

Write ALL findings to: [output file path]
```

---

## Agent 1: Function Signatures, Constructors, Configuration

**Sections**: §1 (Function Signatures), §2 (Configuration), §3 (Constructor Patterns)
**Extra Reference**: `review-standards-lead.md`

**What to check across ALL packages:**
- Constructor taking option struct or functional options instead of positional params
- Functions with >8 params (should they be split?)
- context.Context in wrong position (should be first only for external calls)
- Config loaded from files/viper instead of env vars via osext.MustGetenv
- Constructors that return errors (should be infallible)
- Missing Override methods for test doubles
- Missing time.Now / ID-generator injection in constructors

---

## Agent 2: Interfaces, Types, Option[T]

**Sections**: §4 (Interface Patterns), §8 (Type Definitions), §32 (Option[T] Complete Guide), §36 (Contract Cohesion)
**Extra Reference**: `architecture-patterns.md`

**What to check across ALL packages:**
- Interfaces with only one implementation (should be concrete type)
- Interfaces defined in implementation package instead of consumer package
- `*T` used for optional fields instead of `Option[T]` from majewsky/gg/option
- Missing dot-import for option package (`import . "github.com/majewsky/gg/option"`)
- `iota` used for enums instead of typed string constants
- Named types for domain concepts missing (raw `string` where `AccountName` type should exist)
- Pointer receivers where value receiver is appropriate (or vice versa)
- Type exported when only constructor needs to be public
- **Contract cohesion (§36)**: Constants, error sentinels, or validation functions in a different file from the interface/type they belong to. If `ErrFoo` is returned by `FooDriver` methods, both must live in `foo_driver.go`. MEDIUM for new violations, LOW for pre-existing.
- **Interface consumer audit**: When a sentinel value or special parameter is introduced on an interface method, grep for ALL implementations AND all callers of that interface method across the entire repo. Use gopls `go_symbol_references` when available. Verify every caller validates the sentinel before passing it. Do not rely on the PR description's claim about authorization — verify the call chain independently.

---

## Agent 3: HTTP/API Design

**Sections**: §5 (HTTP/API Patterns), §34 (Architectural Opinions Feb 2026)
**Extra Reference**: `api-design-detailed.md`

**What to check across ALL packages:**
- Auth done as middleware instead of inline at top of handler
- JSON responses not using respondwith.JSON
- Error responses using JSON instead of text/plain (http.Error for 4xx, respondwith.ErrorText for 500)
- Missing DisallowUnknownFields on json.NewDecoder
- Route registration not using httpapi.Compose pattern
- Handler not a method on *API struct
- Handler signature not `handleVerbResource(w, r)`
- Request structs not parsed into purpose-specific types
- API docs in Markdown instead of Go types on pkg.go.dev

---

## Agent 4: Error Handling

**Sections**: §6 (Error Handling), §26.13 (Error message naming)
**Extra Reference**: `error-handling-detailed.md`

**What to check across ALL packages:**
- Error messages not following "cannot <verb>: %w" format
- "failed to" instead of "cannot" in error messages
- Logging AND returning the same error (must choose one)
- Missing error wrapping (bare `return err` without context)
- Generic error messages ("internal error", "something went wrong")
- logg.Fatal used outside cmd/ packages
- Swallowed errors (err checked but not returned or logged)
- fmt.Errorf with %s when %w is needed (or vice versa)
- Validation functions returning (bool, string) instead of error

---

## Agent 5: Database and SQL

**Sections**: §7 (Database Patterns), §27 (Database Deep Dive)
**Extra Reference**: (use sapcc-code-patterns.md §7 + §27)

**What to check across ALL packages:**
- SQL queries not declared as package-level `var` with `sqlext.SimplifyWhitespace()`
- Using `?` placeholders instead of `$1, $2` (PostgreSQL style)
- Missing `defer sqlext.RollbackUnlessCommitted(tx)` after `db.Begin()`
- TIMESTAMP used instead of TIMESTAMPTZ
- NULL columns that should be NOT NULL
- Migrations that modify existing migrations (immutable rule)
- Missing down migrations
- App-level validation that should be DB constraints
- Using explicit transaction for single statements
- Not using `sqlext.ForeachRow` for row iteration

---

## Agent 6: Testing Patterns

**Sections**: §9 (Testing Patterns), §30 (go-bits Testing API Evolution)
**Extra Reference**: `testing-patterns-detailed.md`

**What to check across ALL *_test.go files AND test helpers:**
- Using removed `assert.HTTPRequest` (gone since go-bits commit 8b79638) instead of `httptest.Handler.RespondTo()`
- Using `assert.DeepEqual` where generic `assert.Equal` works
- Table-driven tests (project convention prefers sequential scenario-driven narrative)
- Missing `t.Helper()` in test helper functions
- Using reflect.DeepEqual instead of assert.DeepEqual
- Test fixtures as large JSON files instead of programmatic builders
- Duplicated test setup across test functions (should extract)
- Using `require` package instead of `must` from go-bits
- Not using `must.SucceedT` / `must.ReturnT` for error-checked returns
- Not using `assert.ErrEqual` for flexible error matching
- **Assertion depth check**: For security-sensitive code (auth, filtering, tenant isolation), presence-only assertions (`NotEmpty`, `NotNil`, `assert.True(t, ok)`) are INSUFFICIENT. Tests must verify the actual VALUE matches the expected input (e.g., `assert.Equal(t, expectedID, filters[0]["term"]["tenant_ids"])`)

---

## Agent 7: Package Organization, Imports, Comments

**Sections**: §10 (Package Org), §11 (Import Org), §13 (Comment Style), §28 (CLI Patterns), §36 (Contract Cohesion)
**Extra Reference**: `architecture-patterns.md`

**What to check across ALL packages:**
- Import groups not in stdlib / external / internal order (3 groups)
- Dot-import used for anything other than `majewsky/gg/option`
- Missing SPDX license header
- Comments using `/* */` instead of `//` for doc comments
- Missing 80-slash separator comments (`////////////////...`) between type groups
- `//NOTE:` markers missing for non-obvious logic
- Exported symbols without godoc comments
- cmd/ packages using wrong CLI patterns (if CLI repo)
- Package names not reading as English ("package utils" instead of meaningful name)
- **Contract cohesion (§36)**: Files named generically (`interface.go`, `types.go`, `constants.go`) when they should be named for the domain concept (`storage_driver.go`, `rbac_policy.go`). Constants/sentinels in `util.go` that belong to a specific interface's file. The test: if you can name the owning interface, the artifact must live in that interface's file.

---

## Agent 8: Modern Go, Standard Library, Concurrency

**Sections**: §14 (Concurrency), §15 (Startup/Shutdown), §29 (Modern Go Stdlib)
**Extra Reference**: (use sapcc-code-patterns.md §14, §15, §29)

**What to check across ALL packages:**
- Using `sort.Slice` instead of `slices.SortFunc` (Go 1.21+)
- Using manual `keys := make([]K, 0, len(m)); for k := range m { ... }` instead of `slices.Sorted(maps.Keys(m))`
- Using `strings.HasPrefix + strings.TrimPrefix` instead of `strings.CutPrefix` (Go 1.20+)
- Using manual `if a < b { return a }` instead of `min(a, b)` (Go 1.21+)
- Loop variable capture workaround (`v := v`) in Go 1.22+ code
- Goroutines without proper context cancellation
- Missing SIGINT context handling in main()
- `os.Exit` used instead of proper shutdown sequence
- `sync.Mutex` on struct value instead of per-resource
- Missing `for range N` syntax where applicable (Go 1.22+)

---

## Agent 9: Observability, Metrics, Background Jobs

**Sections**: §16 (Background Jobs), §17 (HTTP Client), §18 (String Formatting), §20 (Observability)
**Extra Reference**: (use sapcc-code-patterns.md §16-18, §20)

**What to check across ALL packages:**
- Prometheus metrics missing application prefix (e.g., `keppel_` or `logrouter_`)
- Counter metrics not initialized to zero
- Counter metric names not plural
- Gauge used where Counter is appropriate (or vice versa)
- Background jobs not using `jobloop.ProducerConsumerJob` pattern
- HTTP client creating new `http.Client` per request instead of using `http.DefaultClient`
- Custom HTTP transport instead of `http.DefaultTransport`
- Missing jitter in polling/retry loops
- `fmt.Sprintf` for simple string concatenation (use `+`)
- `+` for complex multi-part string building (use `fmt.Sprintf`)

---

## Agent 10: Divergences, LLM Tells, and Community Gaps

**Sections**: §22 (Divergences), §24 (Failure Modes), §25 (LLM Code Feedback), §33 (Portunus Architecture), §35 (Reinforcement Table)
**Extra Reference**: `preferred-patterns.md`

**This is the highest-value agent.** It checks for patterns that LLMs generate by default but the project explicitly rejects:

- Functional options pattern (project convention: positional params)
- Table-driven tests (project convention: sequential scenario narrative)
- Interface segregation / many small interfaces (project convention: 1-2 interfaces max per domain)
- Middleware-based auth (project convention: inline at handler top)
- Config validation layer (project convention: no separate validation)
- `*T` for optional fields (project convention: `Option[T]`)
- Config files / viper (project convention: pure env vars)
- Error messages starting with capital letter
- Error messages using "failed to" (project convention: "cannot")
- Helper functions extracted for cyclomatic complexity (project convention: "contrived edit to satisfy silly metrics")
- Exported types when only constructor is public
- Plugin creating its own DB connection (project convention: receive dependencies)
- `errors.New` + `fmt.Sprintf` instead of `fmt.Errorf`
- Manual row scanning instead of `sqlext.ForeachRow`
- Test setup in `TestMain` instead of per-test
- Verbose error checking instead of `assert.ErrEqual` / `must.SucceedT`
- **Extraction without guard transfer**: When inline code is extracted into a named helper, ALL defensive checks that relied on "the caller handles it" must be re-evaluated. A missing guard rated LOW as inline code becomes MEDIUM as a reusable function. Flag extracted helpers that lack self-contained validation.

---

## Cross-Agent Rule: Abstraction Boundary Violations (All Agents)

Flag these patterns regardless of which domain agent encounters them. Full examples: `go-patterns/references/preferred-patterns/code-examples.md` AP-8.

| Pattern | Detection | Severity | Fix |
|---------|-----------|----------|-----|
| Shotgun surgery / always-follows | `grep -rn "must be called after\|must always follow\|INVARIANT.*call" --include="*.go"` | MEDIUM (new) / LOW (existing) | Merge helper into prerequisite function |
| Same-pair call pattern | `helperB(result)` after every `result := functionA()` | MEDIUM (new) / LOW (existing) | Move `helperB` inside `functionA` |
| Defensive copy on fresh slice | `make([]T, len(x)+N)` + `copy` where x is freshly allocated | LOW | Replace with `append(x, element)` |
| Over-tested trivial function | Test LOC > 10x function LOC, complexity <= 2 | LOW | Integration tests likely suffice |

**Origin**: PR #220 — Stefan Majewsky.

# Index Management Reference

Deep reference for schema design, analyzer chains, mapping optimization, reindex strategies, alias management, and ILM policies. Loaded by INDEX mode.

---

## Schema Design by Use Case

### Field Type Selection

| Data | Field Type | Searchable | Aggregatable | Sortable | Notes |
|------|-----------|------------|-------------|----------|-------|
| Full-text content | `text` | Yes | No (use `.keyword` sub-field) | No | Analyzed, tokenized |
| Identifiers, enums | `keyword` | Exact only | Yes | Yes | Not analyzed, max 256 chars default |
| Both search + aggregate | `text` + `keyword` sub-field | Yes (text), exact (keyword) | Yes (keyword) | Yes (keyword) | Most common pattern for string fields |
| Numbers | `integer`, `long`, `float`, `double` | Range queries | Yes | Yes | Choose smallest type that fits |
| Dates | `date` | Range queries | Yes | Yes | Specify format explicitly |
| Booleans | `boolean` | Filter | Yes | Yes | |
| Geo coordinates | `geo_point` | Geo queries | Geo agg | Geo sort | lat/lon pair |
| Nested objects | `nested` | Independent scoring | Nested agg | No | Cross-field correlation preserved |
| Flattened objects | `flattened` | Exact only | Limited | No | Dynamic keys, low overhead |
| Dense vectors | `knn_vector` (OpenSearch) / `dense_vector` (ES) | kNN search | No | No | Embedding storage |
| Rank features | `rank_feature` | rank_feature query | No | No | Numeric signals for scoring only |

### Schema Patterns by Domain

#### E-Commerce Product Search

```json
// OpenSearch 2.x
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "product_analyzer",
        "fields": { "keyword": { "type": "keyword" } }
      },
      "description": { "type": "text", "analyzer": "product_analyzer" },
      "brand": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" } }
      },
      "categories": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "rating": { "type": "half_float" },
      "review_count": { "type": "integer" },
      "sku": { "type": "keyword" },
      "attributes": {
        "type": "nested",
        "properties": {
          "name": { "type": "keyword" },
          "value": { "type": "keyword" }
        }
      },
      "created_at": { "type": "date", "format": "strict_date_optional_time" },
      "popularity_score": { "type": "rank_feature" }
    }
  }
}
```

**Key decisions**:
- `nested` for attributes: preserves color:red + size:10 correlation (prevents "red size-5" matching "red" + "size 10")
- `rank_feature` for popularity: optimized for scoring, not storage
- `scaled_float` for price: efficient storage, avoids floating-point issues

#### Document/Knowledge Base Search

```json
// OpenSearch 2.x
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "document_analyzer",
        "fields": { "keyword": { "type": "keyword" }, "suggest": { "type": "completion" } }
      },
      "body": { "type": "text", "analyzer": "document_analyzer", "term_vector": "with_positions_offsets" },
      "author": { "type": "keyword" },
      "tags": { "type": "keyword" },
      "path": { "type": "keyword" },
      "last_modified": { "type": "date" },
      "content_type": { "type": "keyword" },
      "access_groups": { "type": "keyword" },
      "embedding": { "type": "knn_vector", "dimension": 768, "method": { "name": "hnsw", "space_type": "cosinesimil", "engine": "nmslib" } }
    }
  }
}
```

**Key decisions**:
- `term_vector` on body: enables highlighting without re-analysis at query time
- `completion` sub-field on title: powers type-ahead suggest
- `knn_vector` for embedding: hybrid search (BM25 + semantic)
- `access_groups` for document-level security filtering

#### Log/Event Search

```json
// OpenSearch 2.x
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "message": { "type": "text" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "host": { "type": "keyword" },
      "trace_id": { "type": "keyword" },
      "span_id": { "type": "keyword" },
      "status_code": { "type": "short" },
      "duration_ms": { "type": "integer" },
      "labels": { "type": "flattened" }
    }
  }
}
```

**Key decisions**:
- `flattened` for labels: handles dynamic key-value pairs without mapping explosion
- `short` for status_code: smallest type that fits
- No sub-fields on message: logs rarely need both text search and exact match

---

## Analyzer Chain Design

An analyzer = char_filters -> tokenizer -> token_filters. Each component transforms text in sequence.

### Component Reference

| Stage | Component | What It Does | When to Use |
|-------|-----------|-------------|-------------|
| **Char Filter** | `html_strip` | Removes HTML tags | Content from web crawlers |
| | `mapping` | Character replacement (`& -> and`) | Normalize special characters |
| | `pattern_replace` | Regex-based replacement | Clean up structured noise (IDs, hashes) |
| **Tokenizer** | `standard` | Unicode-aware word tokenizer | General text (default) |
| | `whitespace` | Split on whitespace only | Preserve special tokens (error codes, identifiers) |
| | `keyword` | No splitting — entire input is one token | When field should not be tokenized |
| | `pattern` | Regex-based splitting | Custom delimiters |
| | `path_hierarchy` | `/a/b/c` -> `/a`, `/a/b`, `/a/b/c` | File paths, URLs, categories |
| **Token Filter** | `lowercase` | Lowercases all tokens | Almost always |
| | `stemmer` | Reduces to word stem | Recall improvement (searching -> search) |
| | `stop` | Removes stop words | Usually skip — modern BM25 handles them well |
| | `synonym_graph` | Expands or maps synonyms | Domain vocabulary |
| | `edge_ngram` | Prefix tokens (`search` -> `s`, `se`, `sea`...) | Autocomplete / type-ahead |
| | `shingle` | Creates token n-grams (`search engine` -> `search engine`) | Phrase-like matching without match_phrase |
| | `word_delimiter_graph` | Splits on case change, delimiters | camelCase, under_score, hyphen-ated |
| | `asciifolding` | Folds unicode to ASCII (`café` -> `cafe`) | Multi-language, accent-insensitive search |
| | `truncate` | Truncate tokens to max length | Prevent oversized tokens |

### Analyzer Recipes

#### Product Search Analyzer

```json
{
  "analysis": {
    "char_filter": {
      "normalize_special": {
        "type": "mapping",
        "mappings": ["& => and", "+ => plus", "# => sharp"]
      }
    },
    "filter": {
      "product_synonyms": {
        "type": "synonym_graph",
        "synonyms_path": "analysis/product_synonyms.txt"
      },
      "product_stemmer": {
        "type": "stemmer",
        "language": "light_english"
      }
    },
    "analyzer": {
      "product_analyzer": {
        "type": "custom",
        "char_filter": ["normalize_special"],
        "tokenizer": "standard",
        "filter": ["lowercase", "asciifolding", "product_synonyms", "product_stemmer"]
      }
    }
  }
}
```

**Light stemming**: Use `light_english` instead of `english` for product search. Aggressive stemming conflates terms that should remain distinct in commerce ("running" vs "run" mean different things for shoes).

#### Autocomplete Analyzer (Index + Search Pair)

```json
{
  "analysis": {
    "analyzer": {
      "autocomplete_index": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "autocomplete_edge"]
      },
      "autocomplete_search": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase"]
      }
    },
    "filter": {
      "autocomplete_edge": {
        "type": "edge_ngram",
        "min_gram": 2,
        "max_gram": 15
      }
    }
  }
}
```

**Index/search analyzer split**: Index-time analyzer generates edge ngrams (`kub`, `kube`, `kuber`...). Search-time analyzer uses standard tokenization (the full query term). This way `kube` matches documents with `kubernetes` without generating edge ngrams of the query itself.

---

## Mapping Optimization

### Dynamic Mapping Control

```json
// OpenSearch 2.x — strict dynamic mapping
{
  "mappings": {
    "dynamic": "strict",
    "properties": { }
  }
}
```

| Setting | Behavior | Use When |
|---------|----------|----------|
| `true` (default) | Auto-detect and add new fields | Prototyping only. Mapping explosion risk. |
| `runtime` | New fields as runtime fields | Want flexibility without index bloat |
| `strict` | Reject documents with unmapped fields | Production schemas. Forces explicit mapping. |
| `false` | Store but do not index unknown fields | Preserve data without indexing overhead |

### Mapping Explosion Prevention

Mapping explosion happens when dynamic mapping creates thousands of fields from dynamic data (e.g., user-defined labels, arbitrary JSON).

| Problem | Solution |
|---------|----------|
| Dynamic key-value pairs | Use `flattened` field type |
| Nested objects with variable keys | Map known keys explicitly, set `dynamic: false` for the rest |
| Too many fields total | Set `index.mapping.total_fields.limit` (default 1000). If you need more, reconsider the schema. |
| Deep nesting | Set `index.mapping.depth.limit`. Flatten where possible. |

### Multi-Field Patterns

```json
{
  "title": {
    "type": "text",
    "analyzer": "standard",
    "fields": {
      "keyword": { "type": "keyword", "ignore_above": 256 },
      "exact": { "type": "text", "analyzer": "whitespace" },
      "autocomplete": { "type": "text", "analyzer": "autocomplete_index", "search_analyzer": "autocomplete_search" }
    }
  }
}
```

Use sub-fields when the same data needs different treatment: full-text search on `title`, aggregation on `title.keyword`, autocomplete on `title.autocomplete`.

---

## Reindex Strategies

Schema changes that alter field types, analyzers, or mappings require reindexing. Design for it.

### Zero-Downtime Reindex Pattern

```
1. Create new index (products_v2) with updated mappings
2. Reindex: POST _reindex { "source": {"index": "products_v1"}, "dest": {"index": "products_v2"} }
3. Verify document count: products_v2 count == products_v1 count
4. Switch alias: POST _aliases { "actions": [
     { "remove": { "index": "products_v1", "alias": "products" } },
     { "add": { "index": "products_v2", "alias": "products" } }
   ]}
5. Delete old index when confirmed
```

**Alias**: Applications always query the alias (`products`), not the index directly (`products_v1`). Alias swaps are atomic and zero-downtime.

### Reindex Performance

| Setting | Default | Tuning |
|---------|---------|--------|
| `_reindex` batch size | 1000 | Increase to 5000–10000 for large reindexes |
| `slices` | 1 | Set to `auto` (= number of shards) for parallelism |
| `refresh_interval` | 1s | Set to `-1` during reindex, restore after |
| `number_of_replicas` | N | Set to 0 during reindex, restore after |
| `requests_per_second` | unlimited | Throttle if reindex competes with production traffic |

---

## Alias Management

### Alias Patterns

| Pattern | Use Case | Setup |
|---------|----------|-------|
| **Read alias** | Application reads from `products` | Points to current active index |
| **Write alias** | Application writes to `products-write` | Points to current write target |
| **Filtered alias** | Tenant isolation, subset views | Alias with filter: `"filter": {"term": {"tenant_id": "abc"}}` |
| **Rollover alias** | Time-series data | Auto-creates new index when conditions met |

### Rollover for Time-Series

```json
// OpenSearch 2.x — rollover conditions
POST /logs-write/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_docs": 10000000,
    "max_primary_shard_size": "50gb"
  }
}
```

---

## Index Lifecycle Management (ILM / ISM)

### Phase Definitions

| Phase | Purpose | Typical Actions |
|-------|---------|----------------|
| **Hot** | Active indexing + querying | Full resources, high refresh rate |
| **Warm** | Query-only, recent data | Reduce replicas, merge segments, move to warm nodes |
| **Cold** | Infrequent access | Freeze index, move to cold storage |
| **Delete** | Data expired | Delete index |

### ISM Policy Example (OpenSearch)

```json
// OpenSearch 2.x — Index State Management policy
{
  "policy": {
    "description": "Log retention: hot 7d, warm 30d, cold 90d, delete 365d",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [{ "rollover": { "min_doc_count": 5000000, "min_index_age": "7d" } }],
        "transitions": [{ "state_name": "warm", "conditions": { "min_index_age": "7d" } }]
      },
      {
        "name": "warm",
        "actions": [
          { "replica_count": { "number_of_replicas": 1 } },
          { "force_merge": { "max_num_segments": 1 } }
        ],
        "transitions": [{ "state_name": "cold", "conditions": { "min_index_age": "30d" } }]
      },
      {
        "name": "cold",
        "actions": [{ "read_only": {} }],
        "transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "365d" } }]
      },
      {
        "name": "delete",
        "actions": [{ "delete": {} }],
        "transitions": []
      }
    ]
  }
}
```

---

## Index Template Patterns

### Composable Index Templates (OpenSearch 2.x / Elasticsearch 8.x)

```json
// Component template for common settings
PUT _component_template/common_settings
{
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "5s",
      "codec": "best_compression"
    }
  }
}

// Component template for common mappings
PUT _component_template/common_mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "created_at": { "type": "date" },
        "updated_at": { "type": "date" }
      }
    }
  }
}

// Index template composing components
PUT _index_template/products
{
  "index_patterns": ["products-*"],
  "composed_of": ["common_settings", "common_mappings"],
  "template": {
    "mappings": {
      "properties": {
        "title": { "type": "text" },
        "price": { "type": "scaled_float", "scaling_factor": 100 }
      }
    }
  },
  "priority": 200
}
```

---

## Common Pitfalls and Positive Alternatives

| Pitfall | What to Do Instead |
|---------|-------------------|
| Using `dynamic: true` in production | Set `dynamic: strict`. Map all fields explicitly. Catch schema drift at index time. |
| Applying aggressive stemming to product names | Use `light_english` or no stemming for product/brand names. "Running shoes" and "run shoes" are different queries. |
| Storing everything as `text` | Use `keyword` for identifiers, enums, and structured data. Text analysis has storage and query cost. |
| Querying index names directly from applications | Use aliases. Direct index names couple applications to index lifecycle. |
| Using `standard` analyzer for all text fields | Design analyzers per field role: titles need different analysis than body content, tags need `keyword`, autocomplete needs edge_ngram. |
| Setting shard count without considering data volume | Target 10-50 GB per shard. See performance-optimization.md for shard sizing guidance. |

# LLM Failure Modes in Search Engineering

Where LLMs systematically fail at search engineering tasks. Loaded across all modes as a guardrail reference.

---

## Why This File Exists

LLMs generate plausible-looking search configurations, query DSL, and relevance advice that passes a casual read but fails in production. Search engineering is particularly vulnerable because: query DSL is complex JSON with strict validation, platform versions have diverged significantly, and "improve relevance" sounds correct while being completely non-actionable.

This reference catalogs specific failure modes, their signatures, and defenses.

---

## Failure Mode 1: Hallucinated Query DSL

### What Happens

The LLM generates query DSL that looks syntactically correct but uses non-existent parameters, deprecated features, or impossible combinations. The JSON is well-formed, so it passes a visual check. It fails at runtime with a parse exception.

### Signatures

| Signal | Example |
|--------|---------|
| Non-existent parameters | `"match": { "field": { "query": "x", "boost_mode": "multiply" } }` — `boost_mode` is a `function_score` parameter, not a `match` parameter |
| Mixed platform syntax | Using Elasticsearch `knn` section syntax in an OpenSearch query |
| Invented query types | `"semantic_match"` — does not exist in any platform |
| Wrong nesting | `function_score` inside `bool.filter` (function_score is a top-level wrapper) |
| Deprecated structure | Type-level mappings (`_doc`), `_optimize` endpoint, `common` query |

### Defenses

| Defense | Implementation |
|---------|---------------|
| State platform + version | Begin every query generation with "Platform: OpenSearch 2.x" or "Elasticsearch 8.x" |
| Validate against docs | Check each query clause against the official API documentation for the stated version |
| Test before recommending | Run the query against a test cluster or provide the validation command |
| Annotate uncertainty | If uncertain whether a parameter exists in the target version, say so explicitly |
| Prefer simple constructs | `match`, `term`, `bool`, `range` are stable across versions. Exotic queries drift. |

---

## Failure Mode 2: Version Confusion

### What Happens

The LLM mixes features from different platform versions or confuses Elasticsearch with OpenSearch. After the fork (ES 7.10), these platforms diverged significantly on security, ML, alerting, and vector search APIs.

### Key Divergence Points

| Feature | Elasticsearch 8.x | OpenSearch 2.x |
|---------|-------------------|----------------|
| Security | On by default, built-in | Security plugin (separate) |
| Vector search | `knn` query type, `dense_vector` field | `knn_vector` field, k-NN plugin, different query syntax |
| ML | Elastic ML (proprietary) | ML Commons (open source, different API) |
| Alerting | Watcher / Kibana alerting | Alerting plugin (different API) |
| SQL | `_sql` endpoint | SQL plugin (different endpoint) |
| ILM / ISM | Index Lifecycle Management (ILM) | Index State Management (ISM) — different policy format |
| Aggregation pipeline | `pipeline` agg | Same syntax but some agg types differ |
| License | Elastic License 2.0 / AGPL | Apache 2.0 |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Confirm platform before generating | Ask: "Which platform and version?" before writing any config or query |
| Label all code blocks | `// OpenSearch 2.x` or `// Elasticsearch 8.14` on every code block |
| Check fork point | Features added after ES 7.10 are likely ES-only. Verify OpenSearch equivalents separately. |
| Separate mental models | OpenSearch 2.x security ≠ ES 8.x security. Different APIs, different defaults. |

---

## Failure Mode 3: Generic Relevance Advice

### What Happens

The LLM provides advice that sounds expert but is too vague to implement. "Boost important fields" does not help without knowing which fields, by how much, and measured against what baseline.

### Signatures

| Signal | Example |
|--------|---------|
| No concrete values | "Adjust your BM25 parameters for better relevance" — what values? |
| No measurement | "Add a freshness boost" — how to verify it helped? |
| Undefined terms | "Improve your relevance pipeline" — what specific component? |
| Contradictory advice | "Use BM25 for precision" + "Use vector search for precision" in the same response |
| Missing tradeoffs | "Add synonym expansion" — without noting the precision cost |
| One-size-fits-all | "Set k1=1.2, b=0.75" — these are defaults, not tuning |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Require specifics | Every tuning recommendation includes: parameter name, value, field, expected effect |
| Require measurement | Every change has a before/after metric comparison plan |
| Require context | What content type? What query patterns? What current metrics? |
| Surface tradeoffs | Every optimization has a cost. Name it: precision vs recall, latency vs quality, complexity vs maintainability. |
| Ban "improve relevance" as advice | This is like saying "make it better." Specify what aspect of relevance and how to measure improvement. |

---

## Failure Mode 4: Vector Search as Default Recommendation

### What Happens

The LLM recommends vector search / embeddings as the solution to every search problem. Vector search is powerful but adds significant complexity: embedding model selection, vector storage, approximate nearest neighbor trade-offs, hybrid retrieval fusion, and model retraining.

### When BM25 Solves the Problem

| Scenario | BM25 Sufficient? | Why |
|----------|-----------------|-----|
| Keyword-rich queries ("NullPointerException auth service") | Yes | Exact term matching is what the user wants |
| Known-item search ("OpenSearch documentation") | Yes | Navigational intent, title/URL matching |
| Faceted search ("red shoes size 10") | Yes | Structured filters + text, no semantic gap |
| Log search | Yes | Exact patterns, no semantic interpretation needed |
| Well-maintained synonym dictionaries | Often yes | Synonyms bridge the vocabulary gap BM25 misses |

### When Vector Search Adds Value

| Scenario | Why BM25 Falls Short |
|----------|---------------------|
| Conceptual queries ("how to handle errors gracefully") | Vocabulary gap — user's words differ from document terms |
| Cross-language search | BM25 is language-bound, embeddings can be multilingual |
| Image/multimodal search | No text to match on |
| Semantic similarity for recommendations | "More like this" needs semantic understanding |
| Very short queries against long documents | BM25 struggles with single-word queries on long docs |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Start with BM25 | Tune BM25 + analyzers + synonyms first. Measure the gap. |
| Justify the complexity | Vector search adds: embedding model, index overhead, fusion logic, model maintenance |
| Hybrid, not replacement | Vector search complements BM25. It rarely replaces it entirely. |
| Measure the delta | A/B test hybrid vs BM25-only. Quantify the relevance gain against the complexity cost. |

---

## Failure Mode 5: Ignoring Measurement

### What Happens

The LLM suggests relevance changes without establishing a measurement framework. Without baselines and metrics, there is no way to know whether a change helped, hurt, or had no effect.

### Signatures

| Signal | Example |
|--------|---------|
| No baseline request | "Add this boost" without asking for current metrics |
| Assumed improvement | "This will improve results" without specifying how to verify |
| Subjective evaluation | "Try some queries and see if it looks better" |
| Multiple simultaneous changes | "Update BM25 params, add boosts, and change the analyzer" — which one helped? |
| No regression check | Change improves one query class, silently breaks another |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Baseline first | Capture nDCG@10, P@5, MRR before any change |
| One variable at a time | Change one thing, measure, then change the next |
| Regression check | Evaluate across all query classes, not just the target class |
| Statistical significance | With <200 queries, a 0.02 nDCG change is noise |
| Track the history | Maintain a changelog of what was changed, when, and what the metric impact was |

---

## Failure Mode 6: Deprecated Feature Suggestions

### What Happens

The LLM suggests features that have been removed or deprecated in the target platform version. Training data includes old blog posts, Stack Overflow answers, and documentation from earlier versions.

### Common Deprecated Features

| Feature | Deprecated In | Replacement |
|---------|--------------|-------------|
| Type mappings (`_type`) | ES 7.0+ | Single type per index, `_doc` default |
| `_optimize` API | ES 2.1+ | `_forcemerge` |
| `common` query | ES 7.0+ | `match` query handles stop words natively |
| `indices.optimize` | ES 2.1+ | `_forcemerge` |
| `filtered` query | ES 5.0+ | `bool` with `filter` clause |
| `or`/`and` queries | ES 5.0+ | `bool` with `should`/`must` |
| String field type | ES 5.0+ | `text` or `keyword` |
| `fielddata: true` for aggregation | Discouraged since ES 5.0 | `keyword` sub-field or `doc_values` |
| `_all` field | ES 6.0+ | `copy_to` or `multi_match` |
| `parent-child` joins | ES 5.6+ | `join` field type |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Check release notes | Verify features against the target version's breaking changes documentation |
| Prefer modern constructs | Use `bool`+`filter` over `filtered`, `text`/`keyword` over `string` |
| Flag uncertainty | "This may be deprecated in your version — verify against the docs" when unsure |
| Test with version info | Include version in test setup to catch compatibility issues |

---

## Failure Mode 7: Over-Engineered Schemas

### What Happens

The LLM creates mappings with 50+ fields, sub-fields for every possible analysis, nested objects where flat structures work, and vector fields "just in case." The resulting schema has high storage overhead, slow indexing, and complex query requirements.

### Schema Complexity Assessment

| Signal | Indicates Over-Engineering |
|--------|---------------------------|
| Fields that no query uses | Schema mirrors data model, not query requirements |
| Every string field has 3+ sub-fields | Speculative analysis chains |
| Nested type where arrays of keywords work | Unnecessary join overhead |
| Vector field with no embedding pipeline | Added "for future use" |
| Custom analyzers that duplicate standard behavior | Reinventing existing analyzers |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Start from queries | List the queries the application runs. Map only the fields those queries need. |
| Add fields when needed | Ship minimal schema, add fields when query requirements emerge |
| Measure index cost | Track index size, indexing speed, segment count as schema grows |
| Question every nested type | "Do I need cross-field correlation?" If not, use keyword arrays. |
| Review sub-fields | Each sub-field has storage and indexing cost. Justify each one against a real query need. |

---

## Failure Mode 8: Cargo-Cult Configuration

### What Happens

The LLM copies cluster configuration from blog posts, conference talks, or generic templates without understanding the specific workload. "Best practices" settings harm performance when applied to the wrong workload.

### Common Cargo-Cult Settings

| Setting | Blog Recommendation | Reality |
|---------|-------------------|---------|
| `number_of_shards: 5` | "Always use 5 shards" | Depends on data volume. 50 MB index with 5 shards is wasteful. |
| `refresh_interval: 30s` | "Set to 30s for performance" | Depends on freshness requirements. Search applications need faster refresh. |
| `index.max_result_window: 100000` | "Increase for deep pagination" | Use `search_after` instead. This setting exists as a safety limit. |
| JVM heap at 31 GB | "Always set to 31 GB" | Depends on workload. Over-allocating heap steals from page cache. |
| `thread_pool.search.size: 100` | "Increase for more throughput" | Excessive threads cause context switching. Default is usually correct. |
| `translog.durability: async` | "Set to async for speed" | Trades data durability for indexing speed. Appropriate only during bulk ingest. |

### Defenses

| Defense | Implementation |
|---------|---------------|
| Justify every setting | "Why this value for this workload?" If you cannot answer, use the default. |
| Benchmark with your data | Test settings against representative data and query patterns |
| Monitor the effect | Change one setting, observe the metrics impact, then decide to keep or revert |
| Understand the tradeoff | Every non-default setting trades something. Name what you are trading away. |
| Defaults are usually good | Platform teams optimize defaults for common workloads. Deviate with evidence, not blog posts. |

---

## Cross-Cutting Defense: The Search Engineering Verification Habit

Before delivering any search engineering recommendation:

1. **Platform check**: Is this valid for the stated platform and version?
2. **Specificity check**: Does this include concrete values, field names, and expected outcomes?
3. **Measurement check**: Is there a plan to measure the impact?
4. **Tradeoff check**: Are costs and tradeoffs explicitly stated?
5. **Complexity check**: Is this the simplest approach that solves the problem?
6. **Regression check**: Could this improve one thing while breaking another?

If any check fails, fix it before delivering. "Improve your relevance" is not a recommendation — it is a wish.

# Performance Optimization Reference

Deep reference for query optimization, caching strategies, shard sizing, pagination, circuit breakers, and slow query diagnosis. Loaded by PERFORMANCE mode.

---

## Query Optimization Patterns

### Expensive Query Identification

| Query Pattern | Why Expensive | Mitigation |
|--------------|--------------|------------|
| Leading wildcard (`*search`) | Cannot use inverted index, scans all terms | Use `reverse` token filter + prefix query. Or use ngram sub-field. |
| Deep regex | Scans term dictionary | Restrict regex complexity. Use `keyword` field with `wildcard` type. |
| Large `terms` query (1000+ terms) | Each term is a lookup | Use `terms` lookup from an index. Or pre-filter with a `bool` filter. |
| Unbounded aggregations | Bucket explosion | Set `size` on `terms` agg. Use `composite` agg for paginated aggregation. |
| Nested queries | Per-document join cost | Denormalize where possible. Use `has_child`/`has_parent` only when nesting is truly needed. |
| Script scoring | Evaluated per document | Cache computed values as indexed fields. Use `rank_feature` for common patterns. |
| Deep pagination (from + size > 10000) | Coordinator collects from * size * shards | Use `search_after` or `point in time` + `search_after`. |
| Highlight on large fields | Re-analyzes text or reads term vectors | Use `term_vector: with_positions_offsets` at index time. |
| `match_all` with sort | Touches every shard, returns everything | Add a filter. Even broad filters improve performance. |

### Bool Query Optimization

The `filter` context skips scoring and uses bitset caching. Structure queries to maximize filter usage:

```json
// OpenSearch 2.x — optimized bool structure
{
  "query": {
    "bool": {
      "must": [
        { "match": { "body": "kubernetes deployment" } }
      ],
      "filter": [
        { "term": { "status": "published" } },
        { "range": { "date": { "gte": "2024-01-01" } } },
        { "terms": { "category": ["tutorial", "guide"] } }
      ],
      "should": [
        { "match_phrase": { "title": "kubernetes deployment" } }
      ],
      "minimum_should_match": 0
    }
  }
}
```

**Optimization rules**:
- Hard constraints (status, date range, access control) go in `filter` — cached, no scoring
- Text relevance goes in `must` — scored, drives ranking
- Soft boosts go in `should` — scored, additive, optional
- Put the most selective filter first — Lucene evaluates filters in order of selectivity when possible, but explicit ordering helps readability and debugging

### Profile API for Query Diagnosis

```json
// OpenSearch 2.x — profile a slow query
{
  "profile": true,
  "query": {
    "match": { "body": "kubernetes" }
  }
}
```

**Reading profile output**:

| Field | What It Tells You |
|-------|-------------------|
| `time_in_nanos` | Total time for this query component |
| `breakdown.build_scorer` | Time building scorer (high = complex scoring) |
| `breakdown.advance` | Time iterating posting list (high = low selectivity) |
| `breakdown.next_doc` | Time advancing to next matching doc |
| `breakdown.score` | Time computing score (high = expensive scoring) |
| `collector.reason` | Why results were collected (top_docs, aggregation, etc.) |

---

## Caching Strategies

### Cache Types (OpenSearch/Elasticsearch)

| Cache | What It Caches | Scope | Invalidation | When It Helps |
|-------|---------------|-------|-------------|---------------|
| **Node query cache** | Filter clause results (bitsets) | Node-level | Segment merge/change | Repeated filters (status, category, access control) |
| **Shard request cache** | Entire search response per shard | Shard-level | Index refresh | Repeated identical queries (aggregation dashboards) |
| **Field data cache** | Field values for aggregation/sorting | Node-level | Index change | Sorting/aggregating on text fields (use doc_values instead) |
| **OS page cache** | Index segments in memory | OS-level | LRU | All queries. Most important cache. |

### Cache Optimization

| Strategy | How | Impact |
|----------|-----|--------|
| Use `filter` context | Filters are cached in the query cache | Reduces scoring overhead for repeated constraints |
| Round timestamps | `"gte": "2024-01-01"` caches. `"gte": "now-1h"` generates unique cache keys. | Round to nearest hour/day for time-range filters |
| Use doc_values | Pre-computed columnar storage for sort/aggregation | Eliminates field data cache overhead |
| Preference routing | `_preference=custom_string` routes to same shard | Improves cache hit rate at cost of uneven load |
| Pre-warm queries | Search on index open/refresh with known query patterns | Warm caches after restart or reindex |

### Timestamp Rounding for Cache Hits

```json
// BAD — unique cache key every second
{ "range": { "date": { "gte": "now-24h" } } }

// BETTER — cache key changes hourly
{ "range": { "date": { "gte": "now-24h/h" } } }

// BEST for daily reports — cache key changes daily
{ "range": { "date": { "gte": "now-1d/d" } } }
```

The `/h` and `/d` suffixes round to the hour/day boundary, making the cache key stable for that period.

---

## Shard Sizing

### Sizing Guidelines

| Factor | Guideline | Reasoning |
|--------|-----------|-----------|
| Shard size | 10–50 GB per shard | <10 GB: overhead dominates. >50 GB: recovery/rebalance takes too long. |
| Shards per node | < 20 shards per GB of heap | Each shard consumes memory for metadata, segment files, caches |
| Max shards per index | Divide total data size by target shard size | |
| Minimum shards | 1 for <50 GB, scale linearly | Start with 1 shard for small indices |
| Write throughput | 1 shard handles ~10K docs/sec (depends on doc size) | Scale write shards to match ingest rate |

### Shard Count Formula

```
number_of_shards = ceil(expected_data_GB / target_shard_size_GB)

Example: 200 GB of data, 30 GB target shard size
→ ceil(200 / 30) = 7 shards
```

### Over-Sharding Symptoms

| Symptom | Diagnosis | Fix |
|---------|-----------|-----|
| Cluster state > 500 MB | Too many indices/shards | Merge small indices, use rollover with larger periods |
| Search latency high despite low data volume | Scatter-gather overhead across too many shards | Reduce shard count, shrink API |
| Master node instability | Cluster state updates overwhelm master | Reduce total shard count cluster-wide |
| Memory pressure on data nodes | Too many open segments | Reduce shards, force_merge old indices |

### Shard Allocation Awareness

```json
// OpenSearch 2.x — zone-aware allocation
{
  "cluster.routing.allocation.awareness.attributes": "zone",
  "cluster.routing.allocation.awareness.force.zone.values": "zone-a,zone-b,zone-c"
}
```

Ensures replicas are placed in different availability zones for fault tolerance.

---

## Pagination

### Pagination Methods

| Method | Max Depth | Consistency | Use Case |
|--------|-----------|-------------|----------|
| `from` + `size` | 10,000 (default limit) | Snapshot | UI pagination, small result sets |
| `search_after` | Unlimited | Snapshot (with PIT) | Deep pagination, infinite scroll |
| `scroll` | Unlimited | Frozen snapshot | Batch processing, export (deprecated for search) |
| `point_in_time` + `search_after` | Unlimited | Frozen snapshot | Deep pagination with consistency |

### search_after Pattern

```json
// OpenSearch 2.x — page 1
POST /products/_search
{
  "size": 20,
  "query": { "match": { "title": "shoes" } },
  "sort": [
    { "_score": "desc" },
    { "_id": "asc" }
  ]
}

// Page 2: use sort values from last result of page 1
POST /products/_search
{
  "size": 20,
  "query": { "match": { "title": "shoes" } },
  "sort": [
    { "_score": "desc" },
    { "_id": "asc" }
  ],
  "search_after": [0.87, "product_4521"]
}
```

**Tiebreaker**: Always include a unique field (`_id` or `_shard_doc`) as the last sort criterion. Without a tiebreaker, documents with equal scores may be skipped or duplicated across pages.

### Point in Time for Consistent Pagination

```json
// Step 1: Create PIT
POST /products/_search/point_in_time?keep_alive=5m

// Step 2: Search with PIT
POST /_search
{
  "pit": { "id": "PIT_ID", "keep_alive": "5m" },
  "size": 20,
  "query": { "match": { "title": "shoes" } },
  "sort": [{ "_score": "desc" }, { "_shard_doc": "asc" }]
}
```

---

## Circuit Breakers

### Default Circuit Breakers

| Breaker | Default Limit | What It Protects |
|---------|--------------|------------------|
| `indices.breaker.total.limit` | 95% of heap | Total memory across all breakers |
| `indices.breaker.request.limit` | 60% of heap | Per-request data structures (aggregations, sorting) |
| `indices.breaker.fielddata.limit` | 40% of heap | Field data cache |
| `network.breaker.inflight_requests.limit` | 100% of heap | In-flight network requests |

### Tripped Breaker Diagnosis

| Breaker | Common Cause | Remediation |
|---------|-------------|-------------|
| Request breaker | Large aggregation (high cardinality terms agg) | Reduce agg size, use composite agg, increase heap |
| Field data breaker | Sorting/aggregating on analyzed text field | Use `keyword` sub-field or `doc_values` |
| Parent breaker | Combined memory pressure | Reduce concurrent queries, increase heap, add nodes |

---

## Slow Query Diagnosis

### Enable Slow Query Logging

```json
// OpenSearch 2.x — dynamic setting
PUT /products/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.query.debug": "1s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "500ms"
}
```

### Slow Query Investigation Checklist

| Check | How | What to Look For |
|-------|-----|-----------------|
| Query complexity | Profile API | Deep nesting, expensive clauses, script scoring |
| Shard count | `GET /_cat/shards/products` | Over-sharded index |
| Segment count | `GET /_cat/segments/products` | Too many segments (needs force_merge) |
| Field data usage | `GET /_nodes/stats/indices/fielddata` | High field data = sorting on wrong field type |
| Cache hit rate | `GET /_nodes/stats/indices/query_cache` | Low hit rate = filters not cacheable |
| GC pauses | Node stats, GC logs | Long GC pauses cause query spikes |
| Hot threads | `GET /_nodes/hot_threads` | Shows what threads are doing during slow periods |
| Disk I/O | OS monitoring | Page cache thrashing = not enough memory for data |

### Common Slow Query Patterns and Fixes

| Pattern | Typical Latency | Fix |
|---------|----------------|-----|
| Leading wildcard on large field | 5-30s | Reverse token filter, ngram sub-field |
| `terms` with 10K+ values | 2-10s | Terms lookup, pre-filter, redesign data model |
| Deep pagination (from: 50000) | 5-60s | Switch to search_after |
| High-cardinality terms agg | 2-20s | Set explicit size, use composite agg, pre-aggregate |
| Regex on text field | 5-60s | Keyword sub-field with wildcard type |
| Script score touching all docs | 2-30s | Pre-compute and index as field, use rank_feature |
| match_all on large index | 1-10s | Add at least one filter to narrow candidates |
| Nested query on deeply nested docs | 2-15s | Denormalize, flatten structure |

---

## Bulk Indexing Optimization

| Setting | Default | During Bulk Ingest | Restore After |
|---------|---------|-------------------|---------------|
| `refresh_interval` | `1s` | `-1` (disable) | `1s` or `5s` |
| `number_of_replicas` | 1-2 | `0` | Original value |
| `translog.durability` | `request` | `async` | `request` |
| `translog.flush_threshold_size` | `512mb` | `1gb` | `512mb` |
| Bulk batch size | — | 5-15 MB per batch | — |

### Bulk Request Sizing

| Document Size | Recommended Batch | Docs per Batch |
|--------------|-------------------|----------------|
| < 1 KB | 5-10 MB | 5,000-10,000 |
| 1-10 KB | 5-15 MB | 1,000-5,000 |
| 10-100 KB | 5-15 MB | 100-500 |
| > 100 KB | 5-15 MB | 50-100 |

Target 5-15 MB per bulk request. Larger batches increase memory pressure. Smaller batches increase HTTP overhead.

---

## Refresh Interval Tuning

| Use Case | Refresh Interval | Tradeoff |
|----------|-----------------|----------|
| Real-time search | `1s` (default) | Higher indexing overhead, freshest results |
| Near-real-time | `5s`–`30s` | Good balance for most use cases |
| Batch / analytics | `60s`–`120s` | Higher throughput, stale results acceptable |
| Bulk ingest | `-1` (disable) | Maximum throughput, no search until manual refresh |

**Per-index setting**: Set refresh interval per index based on its use case. Search indices need faster refresh than analytics indices.

---

## Common Pitfalls and Positive Alternatives

| Pitfall | What to Do Instead |
|---------|-------------------|
| Adding nodes to fix slow queries | Profile the query first. Most slow queries are caused by query pattern, not cluster size. |
| Using `from: 10000, size: 20` for deep pagination | Use `search_after` with a PIT. Deep `from` requires coordinator to fetch and discard `from * shards` results. |
| Setting `number_of_shards` to match node count | Size shards by data volume (10-50 GB each). Node count is for replicas and capacity. |
| Sorting on `text` fields | Sort on `keyword` sub-field or `doc_values`-enabled field. Text field sorting loads field data into heap. |
| Disabling caching to "ensure fresh results" | Use appropriate refresh intervals instead. Caching is critical for performance. |
| Tuning JVM heap without data | Profile actual memory usage patterns. Start at 50% of available RAM (max 30-31 GB for compressed oops). |

# Query Understanding Reference

Deep reference for intent classification, entity extraction, query expansion, spelling correction, and query relaxation. Loaded by QUERY mode.

---

## Intent Classification Taxonomy

Classify every query before constructing the search request. Intent determines field selection, boosting strategy, and result presentation.

### Core Intent Types

| Intent | User Goal | Query Characteristics | Strategy |
|--------|-----------|----------------------|----------|
| **Navigational** | Find a specific known item | Short, contains proper nouns, product names, URLs | Title/name exact match, URL match, high precision |
| **Informational** | Learn about a topic | Question words, "how to", longer phrases | Full-text body search, snippet extraction, diversified results |
| **Transactional** | Take an action (buy, download, sign up) | Action verbs, product terms, pricing keywords | Product fields, availability, CTA matching |
| **Faceted** | Filter by attributes | Attribute-value pairs ("red shoes size 10") | Parse attributes into filters, text remainder into query |
| **Exploratory** | Browse/discover a topic area | Vague terms, broad categories | Related terms, faceted navigation, "did you mean" suggestions |
| **Exact** | Find an exact phrase or identifier | Quoted strings, codes, IDs, error messages | Exact match, no stemming, keyword fields |

### Intent Detection Signals

| Signal | Detection Method | Maps To |
|--------|-----------------|---------|
| Quoted strings | Regex: `"[^"]+"` | Exact intent — use match_phrase |
| Question words | Starts with who/what/when/where/why/how | Informational intent |
| Product identifiers | Regex: SKU patterns, model numbers | Navigational intent |
| Attribute patterns | `color:red`, `size:10`, compound adjective+noun | Faceted intent |
| Action verbs | buy, download, install, configure, fix | Transactional intent |
| Single word / broad term | Short query, no qualifiers | Exploratory intent |

---

## Entity Extraction

Extract structured entities from the query to apply as filters, boosts, or routing decisions.

### Entity Types for Search

| Entity Type | Examples | Extraction Method | Usage |
|------------|---------|-------------------|-------|
| Product names | "MacBook Pro", "OpenSearch" | Dictionary matching, NER | Navigate to product page, boost product fields |
| Categories | "shoes", "documentation", "APIs" | Taxonomy lookup | Category filter or boost |
| Attributes | "red", "large", "v2.x" | Attribute vocabulary matching | Structured filters |
| People | "John Smith", "author:jane" | Name patterns, prefix syntax | Author/creator filter |
| Dates/Ranges | "last week", "2024", "before March" | Date parsing | Date range filter |
| Locations | "San Francisco", "us-east-1" | Geo dictionary, region patterns | Geo filter or boost |
| Error codes | "HTTP 503", "NullPointerException" | Regex patterns | Exact match in error fields |
| Versions | "v2.3", "ES 8.x", "Python 3.11" | Semver regex | Version filter |

### Dictionary-Based Extraction

Maintain domain dictionaries for your search corpus:

```
# products.dict
opensearch -> product:OpenSearch
elasticsearch -> product:Elasticsearch
es -> product:Elasticsearch
solr -> product:Solr
vespa -> product:Vespa

# categories.dict
tutorial -> category:tutorials
guide -> category:guides
api reference -> category:api-docs
troubleshoot* -> category:troubleshooting
```

**Dictionary maintenance**: Dictionaries drift. Review monthly against actual query logs. Add terms that users search for but the dictionary misses. Remove terms that cause false positive extractions.

---

## Query Expansion

Expand the original query to improve recall without sacrificing precision.

### Synonym Expansion

| Strategy | How | When | Risk |
|----------|-----|------|------|
| **Index-time synonyms** | Synonyms applied during indexing (analyzer filter) | Synonyms are stable, full reindex acceptable | Cannot update without reindex. Over-expansion. |
| **Query-time synonyms** | Synonyms applied at search time (search_analyzer) | Synonyms change frequently, cannot reindex | Performance cost per query. |
| **Explicit synonyms** | `k8s => kubernetes`, `js => javascript` | Known abbreviations, brand names | Maintain the list manually or mine from logs |
| **Equivalent synonyms** | `notebook, laptop` | Interchangeable terms | Can cause precision loss on ambiguous terms |
| **One-way expansion** | `js => javascript` (but not reverse) | Abbreviations should expand, full terms should not contract | Need directional synonym rules |

### Synonym Configuration Example

```json
// OpenSearch 2.x — synonym filter
{
  "settings": {
    "analysis": {
      "filter": {
        "domain_synonyms": {
          "type": "synonym_graph",
          "synonyms": [
            "k8s, kubernetes",
            "js => javascript",
            "es => elasticsearch",
            "ml, machine learning",
            "db, database",
            "auth, authentication, authn",
            "authz => authorization"
          ]
        }
      },
      "analyzer": {
        "search_with_synonyms": {
          "tokenizer": "standard",
          "filter": ["lowercase", "domain_synonyms", "stemmer"]
        }
      }
    }
  }
}
```

**Synonym ordering**: Apply synonyms before stemming. `js => javascript` then stem `javascript` -> `javascript`. Reversing the order produces incorrect expansions.

### Embedding-Based Expansion

When the synonym dictionary does not cover a term, use embedding similarity to find expansion candidates.

| Approach | How | When |
|----------|-----|------|
| Nearest neighbors | Find top-k similar terms by embedding distance | Rare terms not in synonym dictionary |
| Query embedding | Embed the full query, find similar queries from logs | Query reformulation |
| Contextual expansion | Use the query context to disambiguate expansion | Polysemous terms ("java" = language or island?) |

**Threshold**: similarity > 0.85 for automatic expansion, 0.7–0.85 for "did you mean" suggestions. Below 0.7, skip — the expansion is likely noise.

---

## Spelling Correction

### Correction Strategies

| Strategy | How | Best For |
|----------|-----|----------|
| **Did you mean** | Suggest correction, do not auto-apply | Ambiguous corrections, low confidence |
| **Auto-correct** | Silently correct and search | High-confidence corrections, common misspellings |
| **Search both** | Search original AND corrected query, merge results | Maximize recall, user may have meant what they typed |

### Implementation Approaches

| Approach | Mechanism | Pros | Cons |
|----------|-----------|------|------|
| Index-based suggest | `_suggest` API with term/phrase suggesters | Uses your actual corpus vocabulary | Limited to indexed terms |
| Fuzzy matching | `fuzziness: "AUTO"` on match queries | Zero configuration | Performance cost, false positives |
| Custom dictionary | Pre-built correction dictionary from query logs | High precision for your domain | Maintenance overhead |
| Phonetic matching | `phonetic` token filter (soundex, metaphone) | Catches homophones | Language-specific, false positives |

### Fuzzy Query Configuration

```json
// OpenSearch 2.x — fuzzy match with controlled edit distance
{
  "query": {
    "match": {
      "title": {
        "query": "kuberntes",
        "fuzziness": "AUTO",
        "prefix_length": 2,
        "max_expansions": 50
      }
    }
  }
}
```

**fuzziness: AUTO** behavior:
- 0–2 characters: exact match only
- 3–5 characters: 1 edit allowed
- 6+ characters: 2 edits allowed

**prefix_length**: Characters that must match exactly at the start. Set to 2+ to avoid "cat" matching "bat". Higher values = faster but stricter.

---

## Query Relaxation

When initial queries return too few results, systematically relax constraints to broaden the search.

### Relaxation Hierarchy

Apply in order, stopping when result count is sufficient:

| Step | Action | Example |
|------|--------|---------|
| 1 | Remove date/time filters | `after:2024-01-01` -> no date filter |
| 2 | Remove location/source filters | `in:engineering` -> all sources |
| 3 | Reduce `minimum_should_match` | `100%` -> `75%` -> `50%` |
| 4 | Broaden specific terms | `"PostgreSQL migration"` -> `"database migration"` |
| 5 | Drop least important query terms | `kubernetes pod scheduling failure` -> `kubernetes pod failure` |
| 6 | Apply stemming if not already | `configurations` -> `configur*` |
| 7 | Add synonyms/expansions | `kubernetes` -> `kubernetes OR k8s OR container orchestration` |
| 8 | Fuzzy matching | `fuzziness: AUTO` on remaining terms |

### minimum_should_match Patterns

```json
// OpenSearch 2.x — adaptive minimum_should_match
{
  "query": {
    "bool": {
      "should": [
        { "match": { "body": "distributed" } },
        { "match": { "body": "search" } },
        { "match": { "body": "engine" } },
        { "match": { "body": "architecture" } }
      ],
      "minimum_should_match": "75%"
    }
  }
}
```

| Value | Meaning | When |
|-------|---------|------|
| `100%` or `all` | All terms must match | High-precision mode, short queries |
| `75%` | 3 of 4 terms (rounds down) | Default for informational queries |
| `2<75%` | First 2 required, 75% of remainder | Long queries where core terms matter |
| `1` | At least one term | Maximum recall, use with re-ranking |

---

## Query Transformation Examples

Real-world query rewrites showing the full pipeline:

| User Query | Intent | Entities | Expansion | Final Query |
|-----------|--------|----------|-----------|-------------|
| `kuberntes deploy error` | Informational | product:Kubernetes | spelling: kubernetes, synonym: k8s | `(kubernetes OR k8s) AND deploy* AND error` with body boost |
| `"NullPointerException" in auth service` | Exact + Navigational | error:NPE, service:auth | None (exact match) | `phrase_match("NullPointerException") AND service:auth*` |
| `how to configure opensearch dashboards` | Informational | product:OpenSearch Dashboards | synonym: kibana (for ES users) | Multi-match on title^3, body^1 for "configure opensearch dashboards" |
| `red shoes size 10 under $50` | Faceted | color:red, size:10, price:<50 | None | Filters: color=red, size=10, price<=50. Text: "shoes" |
| `john's PR from last week` | Navigational | person:john, time:last_week | None | Author:john*, date:last_7d, type:pull_request |

---

## Query Pipeline Architecture

```
User Query
    ↓
[1. Tokenize + Normalize]  — lowercase, unicode normalization
    ↓
[2. Spell Check]           — correct obvious misspellings
    ↓
[3. Entity Extract]        — pull out structured entities (people, dates, products)
    ↓
[4. Intent Classify]       — determine query type
    ↓
[5. Synonym Expand]        — add equivalent terms
    ↓
[6. Query Construct]       — build platform-specific query DSL
    ↓
[7. Boost/Filter Apply]    — intent-specific field weights and filters
    ↓
Platform Query
```

Each step is independently testable. Log the query at each stage for debugging relevance issues.

---

## Common Pitfalls and Positive Alternatives

| Pitfall | What to Do Instead |
|---------|-------------------|
| Applying synonyms bidirectionally when only one direction is correct | Use directional synonyms: `js => javascript` keeps precision. |
| Auto-correcting queries without showing the user | Use "did you mean" for ambiguous corrections. Auto-correct only when confidence is very high. |
| Expanding every query with embeddings | Reserve embedding expansion for tail queries with zero results. Head queries have enough signal. |
| Treating all query terms as equally important | Use IDF or query-term weights. Rare terms carry more information than common ones. |
| Building query understanding without logging | Log every stage of the query pipeline. Debugging without query logs is guesswork. |

# Relevance Tuning Reference

Deep reference for BM25 tuning, learned ranking, boost strategies, function scoring, and field weighting. Loaded by RELEVANCE mode.

---

## BM25 Parameter Tuning

BM25 has two parameters that control term frequency saturation (k1) and document length normalization (b). Default values (k1=1.2, b=0.75) are a reasonable starting point. Tuning them for your content type yields measurable gains.

### Parameter Behavior

| Parameter | Controls | Low Value Effect | High Value Effect |
|-----------|----------|-----------------|-------------------|
| **k1** | Term frequency saturation | Quickly saturates (one mention ≈ many mentions) | More mentions = more relevant, linear-ish |
| **b** | Length normalization | Length barely matters | Long documents penalized heavily |

### Tuning Recipes by Content Type

| Content Type | k1 | b | Rationale |
|-------------|-----|-----|-----------|
| Product titles / short text | 0.3–0.6 | 0.1–0.3 | Short fields. One mention is sufficient. Length variation is noise. |
| Product descriptions | 1.0–1.4 | 0.5–0.7 | Medium text. Repetition somewhat informative. Moderate length normalization. |
| Long-form articles / docs | 1.2–2.0 | 0.75–0.9 | Long text. Repetition matters. Strong length normalization prevents long-doc bias. |
| Log messages | 0.5–0.8 | 0.0–0.2 | Structured, consistent length. Minimal normalization needed. |
| Code / technical content | 0.8–1.2 | 0.3–0.5 | Term frequency informative but saturates. Length varies by file, partial normalization. |
| User reviews / comments | 1.0–1.5 | 0.6–0.8 | Variable length, repetition can indicate emphasis. Normalize for length. |

### How to Tune

1. Start with defaults (k1=1.2, b=0.75)
2. Run evaluation on judgment set, capture nDCG@10
3. Sweep k1 in [0.2, 0.5, 0.8, 1.2, 1.6, 2.0] with b fixed
4. Fix best k1, sweep b in [0.0, 0.25, 0.5, 0.75, 1.0]
5. Fine-tune around the best pair in smaller increments
6. Validate on held-out query set to confirm generalization

### Per-Field BM25 (OpenSearch/Elasticsearch)

```json
// OpenSearch 2.x — per-field similarity override
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "similarity": "title_bm25"
      },
      "body": {
        "type": "text",
        "similarity": "body_bm25"
      }
    }
  },
  "settings": {
    "index": {
      "similarity": {
        "title_bm25": {
          "type": "BM25",
          "k1": 0.5,
          "b": 0.2
        },
        "body_bm25": {
          "type": "BM25",
          "k1": 1.4,
          "b": 0.8
        }
      }
    }
  }
}
```

---

## Field Boosting Strategies

### Multi-Match with Field Weights

```json
// OpenSearch 2.x
{
  "query": {
    "multi_match": {
      "query": "kubernetes deployment",
      "type": "cross_fields",
      "fields": ["title^3", "summary^2", "body^1", "tags^2.5"],
      "tie_breaker": 0.3
    }
  }
}
```

### Multi-Match Types and When to Use Each

| Type | Behavior | Best For |
|------|----------|----------|
| `best_fields` | Score from best-matching field | Queries where one field should dominate |
| `most_fields` | Sum scores across fields | Same content analyzed differently (stemmed + exact) |
| `cross_fields` | Treats fields as one big field | Person names, addresses split across fields |
| `phrase` | Phrase match per field, take best | Exact phrase importance |
| `phrase_prefix` | Phrase prefix per field | Autocomplete / type-ahead |

### Boost Value Calibration

Boost values are relative multipliers. Start with these ranges, then measure:

| Field Role | Boost Range | Example |
|-----------|-------------|---------|
| Primary identifier (title, name) | 2.0–5.0 | `title^3` |
| Secondary text (summary, description) | 1.5–2.5 | `summary^2` |
| Body / content | 1.0 (baseline) | `body^1` |
| Structured metadata (tags, categories) | 1.5–3.0 | `tags^2.5` |
| Weak signals (comments, metadata) | 0.5–1.0 | `comments^0.5` |

Boost values above 5 rarely help and often indicate a structural problem. If you need title^10, consider whether a `bool` query with `should` clauses gives better control.

---

## Function Scoring

Use function_score when relevance depends on non-textual signals: popularity, freshness, authority, geographic proximity.

### Decay Functions for Freshness

```json
// OpenSearch 2.x — exponential decay on date
{
  "query": {
    "function_score": {
      "query": { "match": { "body": "kubernetes" } },
      "functions": [
        {
          "exp": {
            "publish_date": {
              "origin": "now",
              "scale": "30d",
              "offset": "7d",
              "decay": 0.5
            }
          }
        }
      ],
      "boost_mode": "multiply",
      "score_mode": "multiply"
    }
  }
}
```

### Decay Parameter Guide

| Parameter | Meaning | Tuning Guidance |
|-----------|---------|----------------|
| `origin` | "Ideal" value (usually `now` for dates) | Set to the optimal value for scoring |
| `scale` | Distance from origin where score = `decay` | Content shelf life: 7d for news, 90d for docs, 365d for reference |
| `offset` | No decay within this range | Grace period: 0 for time-sensitive, 7-30d for general |
| `decay` | Score at `scale` distance (0-1) | 0.5 is standard. Lower = steeper drop. |

### Decay Function Types

| Function | Curve | When |
|----------|-------|------|
| `exp` | Exponential | Strong freshness signal, news/social |
| `linear` | Linear | Steady decline, general content |
| `gauss` | Bell curve | Optimal range (geographic distance, price) |

### Field Value Factor for Popularity

```json
// OpenSearch 2.x — log-dampened popularity boost
{
  "query": {
    "function_score": {
      "query": { "match": { "body": "search tutorial" } },
      "functions": [
        {
          "field_value_factor": {
            "field": "view_count",
            "factor": 1.2,
            "modifier": "log1p",
            "missing": 1
          }
        }
      ],
      "boost_mode": "sum"
    }
  }
}
```

### Modifier Selection

| Modifier | Formula | When |
|----------|---------|------|
| `none` | value * factor | Linear boost. Use for small, bounded values. |
| `log1p` | log(1 + value * factor) | Dampened. Prevents runaway from high values. Most common. |
| `log2p` | log(2 + value * factor) | Slightly more dampened than log1p |
| `sqrt` | sqrt(value * factor) | Moderate dampening |
| `square` | (value * factor)^2 | Amplifies differences. Use with caution. |
| `reciprocal` | 1 / (value * factor) | Inverse. Lower values score higher. |

### boost_mode vs score_mode

| Setting | Controls | Options |
|---------|----------|---------|
| `score_mode` | How multiple functions combine | `multiply`, `sum`, `avg`, `first`, `max`, `min` |
| `boost_mode` | How function result combines with query score | `multiply`, `replace`, `sum`, `avg`, `max`, `min` |

**Common patterns**:
- `score_mode: multiply` + `boost_mode: multiply` — functions modulate text relevance
- `score_mode: sum` + `boost_mode: sum` — functions add independent signals
- `boost_mode: replace` — ignore text score, rank by function output only

---

## Learned Ranking (LTR)

When BM25 + hand-tuned boosts plateau, learned ranking (Learning to Rank) trains a model on relevance judgments to combine features optimally.

### Feature Engineering

Features that make LTR models effective:

| Feature Category | Examples | Implementation |
|-----------------|----------|----------------|
| Text relevance | BM25 score per field, TF-IDF, match count | Query-dependent, from search engine |
| Query features | Query length, query type, has quotes, has filters | Query-dependent, computed at query time |
| Document features | Document length, age, popularity, authority score | Query-independent, indexed as fields |
| Interaction features | Click-through rate, dwell time, bounce rate | Query-document pair, from click logs |
| Freshness | Days since publish, days since update | Query-independent |
| Coverage | Fraction of query terms matched | Query-dependent |
| Exact match | Title exact match, URL path match | Query-dependent, binary features |

### Feature Store Pattern (OpenSearch LTR Plugin)

```json
// OpenSearch 2.x — feature set definition
{
  "featureset": {
    "name": "product_search_features",
    "features": [
      {
        "name": "title_bm25",
        "params": ["keywords"],
        "template_language": "mustache",
        "template": {
          "match": { "title": "{{keywords}}" }
        }
      },
      {
        "name": "description_bm25",
        "params": ["keywords"],
        "template_language": "mustache",
        "template": {
          "match": { "description": "{{keywords}}" }
        }
      },
      {
        "name": "popularity",
        "params": [],
        "template_language": "mustache",
        "template": {
          "function_score": {
            "functions": [
              { "field_value_factor": { "field": "sales_rank", "modifier": "log1p" } }
            ],
            "query": { "match_all": {} }
          }
        }
      }
    ]
  }
}
```

### Model Selection

| Model | Approach | Pros | Cons |
|-------|----------|------|------|
| LambdaMART (XGBoost) | Gradient-boosted trees optimizing nDCG | Strong accuracy, interpretable features | Requires feature engineering |
| RankNet | Neural pairwise loss | Handles raw features | Needs more data, less interpretable |
| Linear | Weighted feature combination | Simple, fast, explainable | Limited expressiveness |

### LTR Workflow

1. Define feature set covering text, query, document, and interaction signals
2. Collect judgments: 4-point scale (Perfect, Good, Fair, Bad) on query-document pairs
3. Log features for judged pairs using `_ltr/_log` endpoint
4. Train model offline (XGBoost/LambdaMART typical)
5. Upload model to the search platform
6. A/B test against BM25 baseline
7. Monitor feature importance drift over time

### When to Use LTR vs Simpler Approaches

| Signal | Stick with BM25 + Boosts | Move to LTR |
|--------|--------------------------|-------------|
| Query volume | < 10K queries/day | > 10K queries/day (enough click data) |
| Relevance gap | Tuning BM25 params still improving | BM25 plateau — same nDCG regardless of tuning |
| Ranking signals | Text relevance dominates | Multiple non-text signals matter (popularity, freshness, personalization) |
| Judgment availability | < 500 judged queries | > 1000 judged queries with consistent labels |
| Engineering capacity | Limited ML infrastructure | Can maintain feature pipelines and model retraining |

---

## Rescoring

Rescoring applies an expensive second query to the top N results from the initial query. Useful for applying complex scoring without paying the cost on every document.

```json
// OpenSearch 2.x — rescore with phrase proximity
{
  "query": {
    "match": { "body": "distributed search engine" }
  },
  "rescore": {
    "window_size": 100,
    "query": {
      "rescore_query": {
        "match_phrase": {
          "body": {
            "query": "distributed search engine",
            "slop": 2
          }
        }
      },
      "query_weight": 0.7,
      "rescore_query_weight": 1.2
    }
  }
}
```

### Rescore Use Cases

| Use Case | First Pass | Rescore |
|----------|-----------|---------|
| Phrase proximity | match query | match_phrase with slop |
| LTR | BM25 | sltr model |
| Vector similarity | BM25 keyword match | knn on top candidates |
| Complex scripting | Standard query | script_score with expensive logic |

**Window size guideline**: Start with 100–200. Larger windows improve quality but cost latency. Measure the tradeoff for your workload.

---

## Common Pitfalls and Positive Alternatives

| Pitfall | What to Do Instead |
|---------|-------------------|
| Tuning boost values by intuition | Measure nDCG before and after each change. Let metrics guide. |
| Applying the same BM25 params to all fields | Per-field similarity. Short fields and long fields have different saturation curves. |
| Adding popularity boost without dampening | Use `log1p` modifier. Raw popularity scores create runaway effects. |
| Stacking multiple function_score functions | Start with one function, measure, add the next. Interaction effects are unpredictable. |
| Copying relevance config from blog posts | Every corpus has different characteristics. Validate against your data and queries. |
| Boosting a field to 10+ | Restructure the query instead. Extreme boosts mask structural issues. |

# Search Quality Reference

Deep reference for search quality metrics, judgment collection, offline evaluation, online testing, and search funnel analysis. Loaded by QUALITY mode.

---

## Metrics Reference

### Ranking Metrics

| Metric | Formula (Simplified) | Measures | Range | Good Value |
|--------|----------------------|----------|-------|------------|
| **nDCG@k** | Normalized discounted cumulative gain at rank k | Graded relevance considering position | 0–1 | > 0.6 (domain-dependent) |
| **MRR** | 1 / rank of first relevant result | How quickly user finds answer | 0–1 | > 0.5 |
| **P@k** | Relevant docs in top k / k | Precision in top results | 0–1 | > 0.4 at k=5 |
| **Recall@k** | Relevant docs in top k / total relevant | Coverage of relevant docs | 0–1 | > 0.7 for recall-critical |
| **MAP** | Mean of average precision per query | Balanced precision-recall | 0–1 | > 0.3 |
| **ERR** | Expected reciprocal rank (cascade model) | Expected user effort to find answer | 0–1 | Higher is better |

### When to Use Which

| Use Case | Primary Metric | Secondary | Why |
|----------|---------------|-----------|-----|
| Web/general search | nDCG@10 | MRR | Multiple relevant results at graded levels |
| Navigational / FAQ | MRR | P@1 | User wants the one right answer fast |
| E-commerce | nDCG@20, P@5 | Revenue per search | Multiple good products, care about top page |
| Legal / compliance | Recall@100 | P@20 | Missing a relevant document has high cost |
| Autocomplete | MRR@5 | Completion rate | User needs the suggestion quickly |
| Knowledge base | nDCG@5 | Click-through rate | First page must be good, few results shown |

### nDCG Deep Dive

Normalized Discounted Cumulative Gain accounts for both relevance grade and position. Higher-ranked positions get more weight (logarithmic discount).

**Relevance grades** (standard 4-point scale):

| Grade | Label | Definition |
|-------|-------|------------|
| 3 | Perfect | Exactly what the user wanted. Would stop searching. |
| 2 | Good | Relevant and useful. Partially addresses the need. |
| 1 | Fair | Marginally relevant. Contains some useful information. |
| 0 | Bad | Not relevant. Does not address the query. |

**Interpretation**:
- nDCG@10 = 0.8+: Strong relevance. Users consistently find good results near the top.
- nDCG@10 = 0.5–0.8: Acceptable. Room for improvement in ranking.
- nDCG@10 < 0.5: Significant relevance issues. Investigate query coverage and ranking.

### Metric Computation Gotchas

| Issue | Impact | Mitigation |
|-------|--------|------------|
| Unjudged documents | Treated as irrelevant (pessimistic) or skipped (optimistic) — changes scores | Use "judged only" nDCG or ensure judgment coverage for top results |
| Position bias in click data | Top results get more clicks regardless of relevance | Apply position debiasing (inverse propensity weighting) |
| Query set selection | Head queries dominate, tail queries under-represented | Stratify by query frequency: head (top 100), torso (100-1000), tail (1000+) |
| Small judgment sets | Noisy metric estimates | Report confidence intervals. Minimum 200 judged queries for stable nDCG. |
| Grade inflation | Annotators tend toward generous grades over time | Periodic calibration sessions. Inter-annotator agreement checks. |

---

## Judgment Collection

### Guidelines Template

Provide annotators with:

1. **Task definition**: "You will be shown a search query and a document. Rate how well the document answers the query."
2. **Grade scale**: Use the 4-point scale above. Define each grade with 2-3 examples from your domain.
3. **Edge case rules**:
   - Partially relevant: Grade 1 (Fair)
   - Relevant but outdated: Grade 1 (Fair) with note
   - Relevant to a different interpretation: Grade 1 or 0 depending on ambiguity
   - Duplicate of a higher-graded result: Grade normally (dedup is a ranking concern)
4. **Query intent**: Annotator should consider what a reasonable user meant by this query.

### Collection Methods

| Method | Volume | Quality | Cost | Latency |
|--------|--------|---------|------|---------|
| **Expert annotation** | 50-200 queries, 10-20 docs each | Highest (domain expertise) | $$$ | Days-weeks |
| **Crowdsourced** | 500-5000 queries | Good (with quality controls) | $$ | Days |
| **Click-based** | 10K+ queries (implicit) | Moderate (biased) | $ | Continuous |
| **LLM-assisted** | 1K+ queries | Good for initial pass | $ | Hours |
| **Pairwise preference** | Any scale | High agreement, lower coverage | $$ | Varies |

### Click-Based Judgments (Implicit Feedback)

| Signal | Interpretation | Reliability |
|--------|---------------|-------------|
| Click + long dwell (>30s) | Likely relevant | Moderate-High |
| Click + short dwell (<5s) | Possibly irrelevant (pogo-sticking) | Low |
| No click, high position | Possibly irrelevant (skipped) | Low (snippet may have answered) |
| Last click in session | Likely satisfying result | High |
| Reformulated query | Previous results unsatisfying | Moderate (negative signal) |

**Position debiasing**: Clicks are biased toward higher positions. Use inverse propensity scoring (IPS) to adjust: `weight = 1 / P(click | position)` estimated from randomization experiments.

### LLM-Assisted Judgment Workflow

1. Generate initial judgments with LLM (prompt includes query, document, grading scale with examples)
2. Human-review a 10-20% sample for calibration
3. Measure LLM-human agreement (target: Cohen's kappa > 0.6)
4. Flag low-confidence judgments for human review
5. Use LLM judgments for development evaluation, human judgments for final decisions

---

## Offline Evaluation

### Test Harness Design

```
Evaluation Pipeline:
1. Load judgment set (queries + graded documents)
2. For each configuration to evaluate:
   a. Run each query against the search index
   b. Collect top-k results with scores
   c. Match results to judgments
   d. Compute metrics (nDCG@k, MRR, P@k)
3. Compare configurations
4. Report with confidence intervals
```

### Statistical Significance

Use paired tests when comparing two configurations on the same query set:

| Test | When | Implementation |
|------|------|----------------|
| Paired t-test | Metric differences approximately normal | `scipy.stats.ttest_rel` |
| Wilcoxon signed-rank | Non-normal distribution, ordinal data | `scipy.stats.wilcoxon` |
| Bootstrap confidence interval | Small sample, no distributional assumption | Resample 1000x, compute metric, report 95% CI |

**Minimum detectable effect**: With 200 queries, you can reliably detect ~0.03 nDCG@10 difference at p<0.05. With 500 queries, ~0.02 difference.

### Evaluation Set Management

| Practice | Why |
|----------|-----|
| Stratify by query type | Head/torso/tail queries have different characteristics |
| Version the judgment set | Track changes over time, reproduce evaluations |
| Refresh quarterly | Corpus evolves, old judgments become stale |
| Separate dev/test sets | Prevent overfitting to evaluation queries |
| Include known failure queries | Ensure regressions are caught |

---

## Online Testing

### A/B Testing for Search

| Component | Design Decision |
|-----------|----------------|
| **Randomization unit** | User (not query) — prevents same user seeing inconsistent results |
| **Primary metric** | Engagement: click-through rate, conversion, or task completion |
| **Guardrail metrics** | Latency p50/p99, zero-result rate, error rate |
| **Sample size** | Pre-compute using minimum detectable effect and baseline variance |
| **Duration** | Minimum 1-2 weeks to capture day-of-week effects |

### Interleaving Experiments

More efficient than A/B testing for ranking changes. Show interleaved results from two ranking functions, measure which function's results get more clicks.

| Method | How | Pros |
|--------|-----|------|
| **Team Draft Interleaving** | Alternate results from each ranker (like picking teams) | Simple, well-understood |
| **Balanced Interleaving** | Ensure equal representation from each ranker | Controls for position bias better |
| **Optimized Interleaving** | Choose interleaving that maximizes statistical power | Most efficient, more complex |

**Advantage**: Interleaving detects ranking differences with 10-100x fewer queries than A/B testing because both variants are shown to the same user in the same results page.

### Guardrail Monitoring

| Metric | Alert Threshold | Why |
|--------|----------------|-----|
| Zero-result rate | > 5% increase | Users seeing empty results pages |
| Query latency p99 | > 20% increase | Performance degradation |
| Click-through rate | > 10% decrease | Results not compelling |
| Reformulation rate | > 15% increase | Users not finding what they need |
| Error rate | > 1% absolute | System stability |

---

## Search Funnel Analysis

### The Search Funnel

```
[Query Submitted]
      ↓  (abandonment: user leaves without clicking)
[Results Viewed]
      ↓  (zero-click: snippet answered or nothing relevant)
[Result Clicked]
      ↓  (pogo-stick: quick return to results)
[Content Consumed]  (dwell time > 30s)
      ↓
[Task Completed]  (conversion, resolution, etc.)
```

### Funnel Metrics

| Stage | Metric | Healthy Range | What Bad Looks Like |
|-------|--------|--------------|---------------------|
| Query -> Results | Success rate | > 95% | Errors, timeouts, zero results |
| Results -> Click | Click-through rate | 40-70% | < 30% = results not compelling |
| Click -> Dwell | Dwell rate (>30s) | > 60% | < 40% = misleading snippets or wrong results |
| Click -> Task | Conversion rate | Domain-specific | Declining = relevance or UX issue |
| Query -> Reformulation | Reformulation rate | < 25% | > 35% = users not finding what they need |

### Query Classification for Analysis

Segment funnel metrics by query class to find specific problem areas:

| Segment | Examples | Why Segment |
|---------|---------|-------------|
| Head / torso / tail | Top 100 / 100-1K / 1K+ | Tail queries often have worse relevance |
| By intent | Navigational / informational / transactional | Different intents have different success patterns |
| By result count | Zero / few (1-3) / normal / many (100+) | Too few or too many results = different problems |
| By query length | 1 word / 2-3 words / 4+ words | Short queries are more ambiguous |
| New vs returning | First query / refined query | Reformulation patterns reveal gaps |

---

## Continuous Quality Monitoring

### Dashboard Components

| Component | Shows | Refresh |
|-----------|-------|---------|
| nDCG@10 trend | Relevance over time (weekly evaluation runs) | Weekly |
| Zero-result rate by day | Queries returning no results | Daily |
| CTR by query segment | Click-through by query type/category | Daily |
| p50/p99 latency | Search performance | Real-time |
| Top zero-result queries | Specific queries that need coverage | Daily |
| Top reformulated queries | Queries users refine = relevance gap | Weekly |
| Judgment freshness | How old is the evaluation set | Monthly |

### Alerting Rules

| Condition | Alert Level | Action |
|-----------|-------------|--------|
| nDCG@10 drops > 5% week-over-week | High | Investigate recent config/data changes |
| Zero-result rate > 10% | High | Check index health, query pipeline |
| p99 latency > 2x baseline | Medium | Profile slow queries |
| CTR drops > 10% for a segment | Medium | Compare ranking for that segment |
| Judgment set > 6 months old | Low | Schedule refresh |

---

## Common Pitfalls and Positive Alternatives

| Pitfall | What to Do Instead |
|---------|-------------------|
| Using only click data for relevance | Combine click signals with expert judgments. Clicks are biased by position and snippet quality. |
| Reporting metrics without confidence intervals | Always include confidence intervals or standard deviation. A 0.02 nDCG improvement may be noise. |
| Tuning on the full evaluation set | Split into dev and test. Tune on dev, report on test. Prevents overfitting to the evaluation set. |
| Measuring only nDCG without engagement metrics | nDCG measures ranking quality. Engagement (CTR, dwell, conversion) measures user satisfaction. Track both. |
| Running A/B tests for less than a week | Day-of-week effects are real. Run for at least 1-2 full weeks. |
| Ignoring tail queries in evaluation | Tail queries are 50%+ of volume. Include them in judgment sets proportionally. |

# WordPress Live Validation — Phase Check Details

## Phase 1: NAVIGATE — Detailed Steps

**Step 1: Verify browser MCP availability**

Before any browser operation, test Playwright tools are accessible. If unavailable, exit with skip report immediately rather than failing later.

**Step 2: Navigate to the post URL**

Use `browser_navigate` with a full HTTPS URL.

**Step 3: Wait for content area**

Use `browser_wait_for` with the content selector. Try selectors in order:
1. `article` (most WordPress themes)
2. `.entry-content` (classic themes)
3. `.post-content` (premium themes)
4. `main` (fallback)

Use custom selector if provided.

**Step 4: Remove cookie/consent banners (if present)**

Use `browser_evaluate` to remove visual overlays (DOM removal only—does not interact with tracking):

```javascript
// Common cookie banner selectors
const banners = document.querySelectorAll(
  '[class*="cookie"], [class*="consent"], [id*="cookie"], [id*="consent"], .gdpr-banner'
);
banners.forEach(b => b.remove());
```

---

## Phase 2: VALIDATE — All 7 Checks

**Check 1: Title Match** (Severity: BLOCKER)

Extract the rendered title:
```javascript
const titleEl = document.querySelector('h1, .entry-title, .post-title');
titleEl ? titleEl.textContent.trim() : null;
```

If expected title provided: compare (case-insensitive, trimmed). PASS if match, BLOCKER if differ or no title found.
If no expected title: report rendered title as INFO.

**Check 2: H2 Structure** (Severity: WARNING)

Extract all H2s:
```javascript
const h2s = Array.from(document.querySelectorAll('h2')).map(h => h.textContent.trim());
JSON.stringify(h2s);
```

If expected count provided: compare. PASS if match, WARNING if differ.
Always report rendered H2 texts for inspection.

**Check 3: Image Loading** (Severity: BLOCKER)

Use `browser_network_requests`. Filter image URLs (common extensions or image MIME types). Check response status:
- 2xx: loaded successfully
- 4xx/5xx: BLOCKER (broken for readers)

Report total, loaded, and failed counts with URLs of failures.

**Check 4: JavaScript Console Errors** (Severity: WARNING)

Use `browser_console_messages`. Filter to `error` level. Exclude patterns:
- Ad networks: doubleclick, googlesyndication, adsbygoogle
- Analytics: gtag, analytics, fbevents
- Consent: cookiebot, onetrust, quantcast
- Browser extensions

Report count of genuine errors and their messages.

**Check 5: OG Tags** (Severity: WARNING)

Extract OG and social meta tags:
```javascript
const getMeta = (sel) => {
  const el = document.querySelector(sel);
  return el ? el.getAttribute('content') : null;
};
JSON.stringify({
  'og:title': getMeta('meta[property="og:title"]'),
  'og:description': getMeta('meta[property="og:description"]'),
  'og:image': getMeta('meta[property="og:image"]'),
  'og:url': getMeta('meta[property="og:url"]'),
  'twitter:card': getMeta('meta[name="twitter:card"]')
});
```

Mark WARNING for missing tags. Report each tag's value and character count.

**Check 6: Meta Description** (Severity: WARNING)

```javascript
const desc = document.querySelector('meta[name="description"]');
desc ? desc.getAttribute('content') : null;
```

PASS if present and non-empty, WARNING if missing or empty. Report value and character count.

**Check 7: Placeholder/Draft Text** (Severity: BLOCKER)

Search visible text for patterns:
```javascript
const body = document.body.innerText;
const patterns = ['[TBD]', '[TODO]', 'PLACEHOLDER', 'Lorem ipsum', '[insert', '[FIXME]'];
const found = patterns.filter(p => body.toLowerCase().includes(p.toLowerCase()));
JSON.stringify(found);
```

Mark BLOCKER if any found, PASS if none.

---

## Phase 3: RESPONSIVE CHECK — Detailed Steps

Test each viewport in sequence:

| Viewport | Width | Height | Represents |
|----------|-------|--------|------------|
| Mobile | 375 | 812 | iPhone-class |
| Tablet | 768 | 1024 | iPad-class |
| Desktop | 1440 | 900 | Standard laptop |

For each viewport:

**Step 1**: Use `browser_resize` to set dimensions.

**Step 2**: Use `browser_take_screenshot` to capture. Save to known path.

**Step 3**: Check for horizontal overflow:

```javascript
document.documentElement.scrollWidth > document.documentElement.clientWidth;
```

Mark WARNING if overflow detected—content extends beyond viewport (usually tables, images, or code blocks not responsive).

**Step 4**: Verify content container visibility:

```javascript
const content = document.querySelector('article, .entry-content, .post-content, main');
if (content) {
  const rect = content.getBoundingClientRect();
  JSON.stringify({ visible: rect.width > 0 && rect.height > 0, width: rect.width, height: rect.height });
} else {
  JSON.stringify({ visible: false });
}
```

Mark WARNING if container not visible or zero dimensions at any breakpoint.

# WordPress Live Validation -- Playwright MCP Tool Guide

Reference for the 8 Playwright MCP tools used by this skill. Covers tool signatures, usage patterns, common pitfalls, and phase mapping.

---

## Tool Overview

This skill uses 8 of the 18 available Playwright MCP tools. All interactions are **read-only** -- no clicking, typing, or form submission.

| Tool | Phases | Purpose |
|------|--------|---------|
| `browser_navigate` | 1 | Load the post URL |
| `browser_wait_for` | 1 | Wait for the content area selector |
| `browser_snapshot` | 2 | Capture DOM state as structural evidence |
| `browser_evaluate` | 1, 2, 3 | Run JavaScript to extract data from the rendered page |
| `browser_network_requests` | 2 | Inspect image loading status (4xx/5xx detection) |
| `browser_console_messages` | 2 | Detect JavaScript errors |
| `browser_resize` | 3 | Switch viewport between mobile/tablet/desktop |
| `browser_take_screenshot` | 3 | Visual evidence at each breakpoint |

---

## Tool Details

### browser_navigate

**Purpose**: Load a URL in the browser tab.

**When to use**: Phase 1 to load the post URL. Optionally in Phase 2 to verify the og:image URL resolves (if OG image fetch verification is enabled).

**Usage pattern**:
```
Navigate to: {post_url}
```

**Key behaviors**:
- Waits for the page to finish loading (network idle) before returning
- Returns the page title and URL after navigation
- Follows redirects (301/302) automatically
- If the URL returns 4xx/5xx, the navigation still "succeeds" from the tool's perspective -- the browser loaded the error page

**Pitfalls**:
- Do not assume navigation failure means the page does not exist. Check the rendered content.
- If the URL redirects to a login page, the navigation succeeds but the content is wrong. Detect this by checking for the content selector in the next step.
- HTTPS certificate errors may cause navigation to fail. Report the error clearly.

---

### browser_wait_for

**Purpose**: Wait for a CSS selector to appear in the DOM before proceeding.

**When to use**: Phase 1 after navigation, to confirm the content area loaded.

**Usage pattern**:
```
Wait for selector: article
```

Try selectors in order: `article` -> `.entry-content` -> `.post-content` -> `main`

**Key behaviors**:
- Blocks until the selector appears or timeout is reached
- Default timeout is typically 30 seconds
- Returns when the element exists in the DOM (does not guarantee visibility)

**Pitfalls**:
- If none of the default selectors match, the page may have loaded correctly but uses a non-standard theme. Capture a screenshot and DOM snapshot for manual inspection.
- The element being in the DOM does not mean it is visible. A `display: none` element passes `wait_for` but is not rendered. Phase 3 content visibility check catches this.
- Do not wait for selectors that may appear only after user interaction (e.g., modal content, tab panels).

---

### browser_snapshot

**Purpose**: Capture the current DOM state as text, showing the page structure and content.

**When to use**: Phase 2 as evidence capture after running validation checks. Provides a structural record of what the page looked like at validation time.

**Key behaviors**:
- Returns the accessibility tree / DOM structure as text
- Includes element roles, text content, and basic structure
- Does not include CSS styles or computed layout information
- Useful for verifying element existence and text content

**Pitfalls**:
- The snapshot is a text representation, not a visual one. Use `browser_take_screenshot` for visual evidence.
- Large pages produce large snapshots. The snapshot is evidence for the report, not the primary validation mechanism.
- Snapshot content may differ from `browser_evaluate` results because it represents the accessibility tree rather than raw DOM.

---

### browser_evaluate

**Purpose**: Execute JavaScript in the browser context and return the result.

**When to use**: Throughout Phases 1-3 for DOM extraction, data collection, and state checks.

**Usage patterns**:

**Extract a single value:**
```javascript
document.querySelector('h1').textContent.trim()
```

**Extract multiple values as JSON:**
```javascript
JSON.stringify({
  title: document.querySelector('h1')?.textContent?.trim(),
  h2Count: document.querySelectorAll('h2').length
})
```

**Check a boolean condition:**
```javascript
document.documentElement.scrollWidth > document.documentElement.clientWidth
```

**Remove DOM elements (cookie banners):**
```javascript
document.querySelectorAll('[class*="cookie"], [class*="consent"]').forEach(el => el.remove())
```

**Key behaviors**:
- Returns the result of the last expression evaluated
- Can access the full browser DOM API
- Runs synchronously in the page context
- Can modify the DOM (used only for cookie banner removal in this skill)

**Pitfalls**:
- Always use optional chaining (`?.`) when querying elements that may not exist. A null reference error crashes the evaluation.
- `JSON.stringify` is required for returning objects. Without it, the tool may return `[object Object]`.
- Long-running evaluations may timeout. Keep JS execution quick -- no loops over thousands of elements.
- `innerText` vs `textContent`: Use `innerText` when you want visible text only (placeholder check). Use `textContent` when you want all text including hidden elements.
- DOM modifications persist for the session. Removing cookie banners affects all subsequent operations (which is the desired behavior).

---

### browser_network_requests

**Purpose**: List network requests made by the page, including their URLs and status codes.

**When to use**: Phase 2 to check image loading status.

**Key behaviors**:
- Returns all network requests made since the page started loading
- Each entry includes the URL, status code, and resource type
- Includes requests for CSS, JS, images, fonts, and other resources

**Image filtering strategy**:
Filter requests where:
1. URL path ends with `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.svg`, `.avif`
2. OR resource type is `image`

Then classify by status code:
- 200-299: loaded
- 301-302: followed redirect (check final status)
- 4xx: broken (BLOCKER)
- 5xx: server error (BLOCKER)

**Pitfalls**:
- Lazy-loaded images may not appear in network requests if they are below the fold. Consider scrolling to the bottom of the page before checking, or accept incomplete coverage.
- Ad/tracking pixels (1x1 transparent GIFs) are technically images but not content images. Filter by size or domain if noise is excessive.
- Some CDNs return 200 with a placeholder image instead of 404. These are harder to detect -- the status code looks fine but the content is wrong. Visual inspection (screenshots) catches these.
- Cached resources may not appear as new network requests on revisit.

---

### browser_console_messages

**Purpose**: Retrieve JavaScript console messages (log, warn, error).

**When to use**: Phase 2 to detect JavaScript errors on the page.

**Key behaviors**:
- Returns all console messages since page load
- Each message has a level (log, warn, error, info) and text content
- Captures messages from all scripts (first-party and third-party)

**Error filtering strategy**:
1. Filter to `error` level only
2. Exclude messages matching benign patterns (see validation-checks.md Check 4)
3. Count remaining errors
4. Report message text for any genuine errors

**Pitfalls**:
- Third-party scripts (ads, analytics) generate the majority of console errors on most WordPress sites. Without filtering, the error count is meaningless.
- Some errors are transient (race conditions during page load). A single run captures whatever happened during this particular load.
- Console messages accumulate during the session. If multiple navigations happen (e.g., OG image verification), messages from all navigations are included. Filter by relevance if needed.
- The benign pattern filter is intentionally conservative (exclude known noise). Unknown errors are reported rather than silently filtered.

---

### browser_resize

**Purpose**: Change the browser viewport dimensions.

**When to use**: Phase 3 to test responsive layout at mobile (375x812), tablet (768x1024), and desktop (1440x900).

**Usage pattern**:
```
Resize to: 375x812
```

**Key behaviors**:
- Changes viewport width and height immediately
- Page content reflows according to CSS media queries
- Does not trigger a page reload -- the same DOM is reflowed
- Returns confirmation of the new viewport size

**Pitfalls**:
- After resizing, allow a brief moment for CSS transitions and reflow before taking screenshots or checking overflow. The browser_evaluate call acts as an implicit wait since it runs after reflow.
- Some themes use JavaScript-based responsive logic (not just CSS media queries). These may not trigger on resize without a page reload. Most WordPress themes use CSS media queries, so this is rare.
- The order of resizing matters for screenshots. Go from smallest to largest (mobile -> tablet -> desktop) to capture the most common user experience first.

---

### browser_take_screenshot

**Purpose**: Capture a visual screenshot of the current browser viewport.

**When to use**: Phase 3 at each breakpoint for visual evidence. Also in Phase 1 if the page fails to load (error evidence).

**Key behaviors**:
- Captures the visible viewport area
- Returns the screenshot as an image (viewable in Claude's response)
- Screenshots capture the current visual state including any overlays, modals, or CSS effects

**Pitfalls**:
- Screenshots capture only the above-the-fold content at the current scroll position. Long articles require scrolling for full coverage. For this skill, above-the-fold is sufficient -- the purpose is layout validation, not full-page capture.
- Cookie banners or consent overlays appear in screenshots unless removed first (Phase 1 Step 4).
- Dark mode themes may make screenshots harder to interpret visually. The DOM-level checks are not affected by visual themes.
- Screenshot file paths should be noted in the report for reference.

---

## Tools NOT Used

These Playwright MCP tools are available but intentionally excluded:

| Tool | Reason for Exclusion |
|------|---------------------|
| `browser_click` | Read-only validation -- no clicking |
| `browser_fill_form` | No form interaction needed |
| `browser_type` | No typing needed |
| `browser_select_option` | No dropdown interaction needed |
| `browser_drag` | No drag interaction needed |
| `browser_hover` | No hover interaction needed |
| `browser_press_key` | No keyboard interaction needed |
| `browser_handle_dialog` | No expected dialogs on published posts |
| `browser_navigate_back` | Single-page validation, no navigation history |
| `browser_tabs` | Single-tab workflow |
| `browser_file_upload` | No file uploads needed |
| `browser_install` | Playwright assumed pre-installed |
| `browser_close` | Handled by MCP server lifecycle |
| `browser_run_code` | `browser_evaluate` covers all JS execution needs |

If any of these tools are invoked during validation, it is a signal that the skill is doing something outside its read-only scope. Review the workflow.

---

## Availability Detection

The Playwright MCP server may not be available in all Claude Code sessions. Detection strategy:

1. Attempt `browser_navigate` in Phase 1 as the first operation
2. If the tool call fails with a "tool not found" or connection error, Playwright is not available
3. Emit a skip report and exit -- do not retry
4. The skip report should state: "Playwright MCP not available. Live validation skipped. Configure the Playwright MCP server to enable browser-based validation."

Do not attempt to install or configure Playwright from within the skill. That is the user's responsibility.

# WordPress Live Validation -- Check Specifications

Detailed specifications for each validation check, including severity rationale, edge cases, and JavaScript extraction patterns.

---

## Severity Levels

| Level | Meaning | User Impact | Examples |
|-------|---------|-------------|---------|
| BLOCKER | Readers see broken or incorrect content | Direct negative impact on reader experience | Broken images, wrong title, placeholder text visible |
| WARNING | Quality is degraded but content is readable | Indirect impact on discoverability or experience | Missing OG tags, JS errors, horizontal overflow |
| INFO | Informational, no action needed | None | Rendered title reported without comparison target |

**Severity assignment is not arbitrary.** The distinction is: does the reader see something broken (BLOCKER) or something suboptimal (WARNING)? If neither, it is INFO.

---

## Check 1: Title Match

**Severity**: BLOCKER (when comparison available), INFO (when no expected title)

**Why BLOCKER**: The title is the single most visible element on the page. If the theme renders a different title than what was uploaded, readers see the wrong content. This can happen when:
- A WordPress plugin rewrites titles (SEO plugins, auto-titling)
- The theme pulls the title from a different field than `post_title`
- HTML entities in the title are double-encoded (`&` instead of `&`)

**Extraction**:
```javascript
const titleEl = document.querySelector('h1, .entry-title, .post-title');
titleEl ? titleEl.textContent.trim() : null;
```

**Selector priority**: `h1` covers most themes. `.entry-title` is WordPress default class. `.post-title` is used by some premium themes. The first match wins.

**Comparison logic**:
- Trim whitespace from both strings
- Case-insensitive comparison
- Normalize HTML entities (`&` -> `&`, `’` -> `'`)
- If comparison fails, report both values for manual inspection

**Edge cases**:
- Post has no H1 (some themes use the title in `<header>` outside the content area): try `.entry-title` and `.post-title` selectors
- Multiple H1 elements: use the first one, report a WARNING about multiple H1s
- Title contains special characters that render differently in HTML: compare after entity normalization

---

## Check 2: H2 Structure

**Severity**: WARNING

**Why WARNING (not BLOCKER)**: A missing or reordered H2 means the article's structure changed during rendering, but the content is still readable. Common causes:
- Theme CSS hides certain headings
- A plugin modifies heading hierarchy (e.g., table-of-contents generators that restructure headings)
- WordPress block editor wraps headings differently than raw markdown

**Extraction**:
```javascript
const h2s = Array.from(document.querySelectorAll('h2')).map(h => h.textContent.trim());
JSON.stringify(h2s);
```

**Comparison logic** (when expected count is provided):
- Compare count only, not text content (themes may add prefixes, numbers, or anchors)
- If count differs, report both the rendered H2 texts and the expected count
- If rendered count is higher, the theme or a plugin may be injecting headings (e.g., "Related Posts" section)
- If rendered count is lower, a heading may be hidden by CSS or stripped by a filter

**Edge cases**:
- Table of contents plugins inject H2s at the top: the rendered count will be higher than source
- Some themes render H2s as styled `<div>` or `<span>` elements: these will not be captured by the H2 query
- WordPress "separator" blocks between sections: not H2s, should not affect count

---

## Check 3: Image Loading

**Severity**: BLOCKER

**Why BLOCKER**: A broken image (404, 403, 500) shows a broken image icon or empty space where visual content should be. This is immediately visible to readers and damages credibility.

**Method**: Use `browser_network_requests` and filter results.

**Filtering criteria**:
- URL ends with common image extensions: `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.svg`, `.avif`
- OR response Content-Type starts with `image/`
- Exclude data URIs (inline base64 images) -- these are not network requests
- Exclude known tracking pixels (1x1 images from analytics/ad platforms)

**Status classification**:
- 200-299: Loaded successfully
- 301/302 -> 200: Redirect to successful load (OK)
- 403: Access denied (BLOCKER -- CDN permission issue)
- 404: Not found (BLOCKER -- image missing)
- 500+: Server error (BLOCKER -- upstream failure)

**Edge cases**:
- Lazy-loaded images: May not appear in network requests until scrolled into view. Use `browser_evaluate` to scroll to the bottom of the page before checking network requests, or accept that lazy images may show as "not loaded" rather than "failed"
- Responsive images (`srcset`): The browser picks one source from the set. Only the selected source appears in network requests.
- External images (Unsplash, Cloudinary, etc.): These are not under WordPress control but still affect reader experience. Report failures regardless of origin.
- SVG inline: SVGs inlined as `<svg>` elements are not network requests and will not appear in the image check

---

## Check 4: JavaScript Console Errors

**Severity**: WARNING

**Why WARNING (not BLOCKER)**: Most JS errors on content sites are from third-party scripts (ads, analytics, consent managers) and do not affect content rendering. However, some JS errors do block lazy loading, break interactive elements, or prevent consent banners from functioning.

**Method**: Use `browser_console_messages` and filter to `error` level.

**Benign pattern filter** (exclude these from the error count):

```
// Ad networks
doubleclick, googlesyndication, adsbygoogle, amazon-adsystem, criteo

// Analytics
gtag, analytics, google-analytics, fbevents, facebook.net/tr, hotjar, clarity.ms

// Consent managers
cookiebot, onetrust, quantcast, cookieconsent, gdpr

// Browser/extension noise
extensions::, chrome-extension://, moz-extension://

// Common benign warnings
Failed to load resource: net::ERR_BLOCKED_BY_CLIENT  (ad blocker)
```

**Genuine error indicators** (always report these):
- Errors from the site's own domain
- `Uncaught TypeError` or `Uncaught ReferenceError` from non-filtered scripts
- Errors mentioning `wp-content`, `wp-includes`, or theme paths
- `Content Security Policy` violations from the site's own resources

**Edge cases**:
- Ad blocker in the browser blocks ad scripts, generating console errors: these are benign, filter them
- Third-party script errors can cascade into site functionality issues: report the count, let the user judge

---

## Check 5: OG Tags

**Severity**: WARNING

**Why WARNING (not BLOCKER)**: Missing OG tags do not affect on-page reader experience. They affect how the post appears when shared on social media -- broken thumbnails, generic titles, missing descriptions. Important for content distribution but not a reader-facing defect.

**Extraction**:
```javascript
const getMeta = (sel) => {
  const el = document.querySelector(sel);
  return el ? el.getAttribute('content') : null;
};
JSON.stringify({
  'og:title': getMeta('meta[property="og:title"]'),
  'og:description': getMeta('meta[property="og:description"]'),
  'og:image': getMeta('meta[property="og:image"]'),
  'og:url': getMeta('meta[property="og:url"]'),
  'twitter:card': getMeta('meta[name="twitter:card"]')
});
```

**Expected values**:

| Tag | Expected | Notes |
|-----|----------|-------|
| og:title | Non-empty, matches post title | May differ from H1 if SEO plugin overrides |
| og:description | Non-empty, 50-160 chars | Generated by SEO plugin, may differ from meta description |
| og:image | Valid URL, resolves to 200 | Should be the featured image, not the site logo |
| og:url | Matches the canonical post URL | Should not be the homepage URL |
| twitter:card | `summary` or `summary_large_image` | Missing means Twitter uses og:* fallbacks |

**OG Image verification** (optional behavior):
When enabled, navigate to the og:image URL and verify it returns 200. This catches cases where the OG image URL is set but points to a deleted or moved image.

**Edge cases**:
- SEO plugin generates different og:title than the post title: report both, this is expected behavior
- og:image points to site default image instead of featured image: report the URL for manual check
- Multiple og:image tags: report the first one (Facebook/Twitter use the first)

---

## Check 6: Meta Description

**Severity**: WARNING

**Why WARNING**: Missing meta description means search engines auto-generate one from page content, which is usually worse than a crafted description. Not visible to readers on the page itself.

**Extraction**:
```javascript
const desc = document.querySelector('meta[name="description"]');
desc ? desc.getAttribute('content') : null;
```

**Validation**:
- Present and non-empty: PASS (report value and length)
- Present but empty string: WARNING ("meta description is empty")
- Not present: WARNING ("meta description tag missing")
- Length > 160 chars: INFO ("meta description may be truncated in SERPs -- {N} chars")

**Edge cases**:
- Some themes use `og:description` but not `meta[name="description"]`: report the og:description as a note
- Description generated dynamically by JS: may not appear in the initial DOM snapshot if the SEO plugin renders client-side

---

## Check 7: Placeholder/Draft Text

**Severity**: BLOCKER

**Why BLOCKER**: Placeholder text visible to readers is an immediate credibility issue. `[TBD]`, `Lorem ipsum`, or `[TODO]` in a published post signals unfinished content.

**Extraction**:
```javascript
const body = document.body.innerText;
const patterns = ['[TBD]', '[TODO]', 'PLACEHOLDER', 'Lorem ipsum', '[insert', '[FIXME]', 'XXX'];
const found = patterns.filter(p => body.toLowerCase().includes(p.toLowerCase()));
JSON.stringify(found);
```

**Why `innerText` not `textContent`**: `innerText` returns only visible text (respects `display: none`, `visibility: hidden`). Hidden placeholder text in comments or hidden elements is not reader-facing and should not be flagged.

**Edge cases**:
- Article content legitimately discusses placeholder text (e.g., "avoid using [TBD] in production"): this is a false positive. The skill flags it; the user judges context. This is acceptable because false positives are preferable to false negatives for reader-facing text.
- WordPress admin bar or theme elements contain "placeholder" in class names: `innerText` filters these out because class names are not visible text
- Code blocks containing placeholder patterns: these are visible to readers and should be flagged (a code example with `[TODO]` may be intentional, but the user should verify)

---

## Responsive Checks

### Horizontal Overflow

**Severity**: WARNING

**Detection**:
```javascript
document.documentElement.scrollWidth > document.documentElement.clientWidth;
```

**Common causes of overflow at narrow viewports**:
- Tables without `overflow-x: auto` wrapper
- Images without `max-width: 100%`
- Code blocks with long lines and no horizontal scroll
- Fixed-width elements (iframes, embeds)
- Flexbox or grid layouts that do not wrap at small sizes

### Content Visibility

**Severity**: WARNING

**Detection**:
```javascript
const content = document.querySelector('article, .entry-content, .post-content, main');
if (content) {
  const rect = content.getBoundingClientRect();
  JSON.stringify({ visible: rect.width > 0 && rect.height > 0, width: rect.width, height: rect.height });
} else {
  JSON.stringify({ visible: false });
}
```

A content container with zero width or zero height at any breakpoint means the content is hidden -- possibly by a CSS media query that collapses the content area on mobile (a theme bug).