# Stability Anti-Patterns

Production systems do not fail because of bugs found in unit tests. They fail because of emergent behaviors that arise when systems interact under real-world conditions -- load spikes, network partitions, slow dependencies, and data growth that exceeds every assumption made during development.

Michael Nygard identifies a recurring set of anti-patterns that cause the vast majority of production outages. Recognizing these patterns is the first step toward building resilient systems.

## 1. Integration Points: The Number-One Killer

Every integration point -- every socket connection, HTTP call, database query, message queue interaction, or third-party API call -- is a potential failure point. Integration points are the number-one killer of production systems because they introduce failure modes that do not exist in isolated testing.

### How Integration Points Fail

| Failure Mode | Description | Consequence |
|-------------|-------------|-------------|
| **Connection refused** | Remote host rejects the connection | Fast failure; relatively benign |
| **Connection timeout** | Remote host does not respond to SYN | Thread blocked for 30-120 seconds (OS default) |
| **Read timeout** | Connection established but response never arrives | Thread blocked indefinitely without explicit timeout |
| **Partial response** | Connection drops mid-response | Corrupted data; parser exceptions |
| **Slow response** | Response eventually arrives but takes minutes | Threads accumulate; pool exhaustion; cascading failure |
| **Protocol violation** | Remote returns unexpected content type or format | Unhandled exceptions; crash |

### Defense Strategy

Every integration point needs a defense-in-depth strategy:

1. **Timeouts** on every connection and read operation -- never use defaults
2. **Circuit breakers** to stop calling a failing dependency
3. **Bulkheads** to isolate the failure from other parts of the system
4. **Fallbacks** to degrade gracefully when the dependency is unavailable
5. **Monitoring** to detect degradation before users notice

### Real-World Example

A retail application calls an inventory service to check stock levels. The inventory service's database enters a long garbage collection pause. The inventory service stops responding but does not close connections. The retail application's thread pool fills with threads waiting for inventory responses. The retail application can no longer serve any requests -- including requests that do not need inventory data. The entire site goes down because one dependency slowed down.

**Root cause:** No read timeout on the inventory service call. No bulkhead isolating inventory calls from other request handling.

---

## 2. Cascading Failures

A cascading failure occurs when a failure in one system causes failures in the systems that depend on it, which in turn cause failures in the systems that depend on them, and so on. The defining characteristic of a cascading failure is that the damage spreads far beyond the original failure.

### Cascade Mechanics

```
Service A (database overloaded)
    → Service B (calls A, threads block waiting)
        → Service C (calls B, threads block waiting)
            → Service D (calls C, threads block waiting)
                → User-facing application (completely unresponsive)
```

### Why Cascading Failures Are Devastating

- The original failure may be minor (one database query slow)
- The cascade amplifies the failure geometrically
- Each layer adds more blocked threads, more resource consumption
- By the time the cascade is visible, every system in the chain is degraded
- Recovery requires coordinated action across all affected systems

### Breaking the Cascade

| Pattern | How It Breaks the Cascade |
|---------|--------------------------|
| **Circuit Breaker** | Stops calling the failing service; returns error immediately |
| **Timeout** | Limits how long a caller waits; frees threads to handle other work |
| **Bulkhead** | Isolates the failing dependency's impact to a limited set of resources |
| **Fallback** | Provides degraded but functional response when dependency fails |
| **Fail Fast** | Rejects requests immediately when system knows it cannot fulfill them |

### Prevention Checklist

- [ ] Every service-to-service call has a timeout
- [ ] Circuit breakers protect against sustained failures
- [ ] Failure in one dependency does not affect unrelated features
- [ ] Fallback behavior is defined for every critical dependency
- [ ] Cascading failure scenarios are included in chaos experiments

---

## 3. Users as a Source of Load

Users are not gentle with your system. They do not arrive in an orderly queue at predictable intervals. Real user behavior generates load patterns that are fundamentally different from what synthetic tests produce.

### Unexpected User Behaviors

| Behavior | Load Impact | Example |
|----------|------------|---------|
| **Refresh storms** | Multiplies load when pages are slow | Users hit F5 repeatedly when checkout is slow |
| **Bot traffic** | Can exceed human traffic by 10-100x | Scrapers, search engines, monitoring tools |
| **Flash crowds** | Sudden, massive traffic spikes | Hacker News front page, TV mention, viral tweet |
| **Abandoned sessions** | Resource consumption without completion | Users open carts, leave; sessions consume server memory |
| **Power users** | Disproportionate resource consumption | One user with 50,000 items in a list; API consumers with no rate limit |

### Defense Strategy

- Rate limiting per user, per IP, and per API key
- Session limits and timeouts to reclaim abandoned resources
- Bot detection and separate handling (different rate limits, caching strategies)
- Graceful degradation under load (serve cached content, disable non-essential features)
- Load shedding: deliberately reject some requests to preserve service for others

---

## 4. Blocked Threads

Blocked threads are the silent killer. Unlike a crash (which is loud and obvious), blocked threads produce no errors, no exceptions, and no log entries. The system simply stops processing requests.

### Common Causes of Blocked Threads

| Cause | How It Blocks | Detection |
|-------|--------------|-----------|
| **Missing timeouts** | Thread waits indefinitely for a response | Thread dump shows threads in WAITING/TIMED_WAITING |
| **Deadlocks** | Two threads each hold a lock the other needs | Thread dump shows circular lock dependencies |
| **Synchronized access** | All threads queue for a single lock | Throughput drops to single-threaded speed |
| **DNS resolution** | DNS lookup blocks the calling thread | Threads stuck in `InetAddress.getByName()` |
| **Log file I/O** | Synchronous logging blocks application threads | Threads stuck in file write; especially during disk pressure |

### Detection and Prevention

**Detection:**
- Thread dump analysis (scheduled periodic dumps, not just during incidents)
- Thread pool utilization metrics (active/idle/max)
- Request latency distribution (sudden latency spike = possible thread starvation)
- Health checks that verify thread pool availability

**Prevention:**
- Explicit timeouts on all blocking operations
- Asynchronous I/O where possible
- Bounded queues with rejection policies (not unbounded queues that grow forever)
- Thread pool sizing based on measured workload, not defaults

---

## 5. Self-Denial Attacks

A self-denial attack is when your own system, marketing, or business operations generate load that overwhelms your infrastructure. The irony is that these are "success disasters" -- everything is working as intended, but the system cannot handle its own success.

### Common Self-Denial Scenarios

| Scenario | Mechanism | Prevention |
|----------|-----------|------------|
| **Marketing email blast** | 500,000 emails sent simultaneously, 10% click through in 5 minutes | Stagger sends over hours; pre-scale infrastructure |
| **Coupon code launch** | Viral sharing of limited coupon creates stampede | Queue-based redemption; rate limit per user |
| **Product launch countdown** | Users refresh at exactly midnight | Serve static page at launch time; queue for access |
| **Social media viral moment** | CEO's tweet goes viral, floods landing page | CDN caching; static page fallback |
| **Cron job stampede** | Every server runs cleanup job at midnight | Randomize cron schedules; use distributed job scheduling |

### Defense Strategy

- Coordinate marketing events with engineering capacity planning
- Use CDN and static page caching for high-traffic landing pages
- Implement queue-based access for limited-resource events
- Stagger scheduled jobs with random jitter
- Pre-scale infrastructure before planned events

---

## 6. Scaling Effects

Patterns that work at small scale break at large scale. A design that performs well with 10 servers may collapse at 100 servers. Scaling effects are the emergent behaviors that appear only when the system grows.

### Examples of Scaling Effects

| Pattern | Works at Small Scale | Breaks at Large Scale |
|---------|---------------------|----------------------|
| **Point-to-point connections** | 5 services = 20 connections | 50 services = 2,450 connections |
| **Broadcast messages** | 10 subscribers = manageable | 1,000 subscribers = message storm |
| **Shared database** | 5 services share one DB | 50 services = connection pool exhaustion |
| **Health check polling** | Load balancer checks 5 servers | Load balancer checks 500 servers = significant traffic |
| **Distributed locks** | Low contention with few nodes | High contention with many nodes |

### Mitigation

- Replace point-to-point with message buses or service meshes
- Use fan-out patterns with controlled concurrency
- Give each service its own data store (or at minimum, its own connection pool limits)
- Use push-based health checks or sampling strategies at scale
- Avoid distributed locks; use optimistic concurrency or partitioning instead

---

## 7. Unbalanced Capacities

When upstream and downstream systems have different capacity limits, the faster system can overwhelm the slower one. This is particularly dangerous when a frontend tier can generate more requests than a backend tier can handle.

### Common Imbalances

- Web tier: 100 servers; backend API: 10 servers
- Batch job produces 10,000 records/sec; downstream consumer processes 500/sec
- Marketing campaign drives 10x normal traffic to a service scaled for 1x

### Defense Strategy

- Back-pressure: slow producers when consumers are overwhelmed
- Queue-based buffering between tiers with different throughput
- Rate limiting at the boundary between tiers
- Capacity modeling that considers the full request chain, not individual services in isolation
- Autoscaling policies that scale the entire chain, not just the entry point

---

## 8. Dogpile / Thundering Herd

A dogpile (also called thundering herd) occurs when many threads or processes simultaneously attempt the same expensive operation, typically after a cache expires or a service recovers.

### Common Dogpile Scenarios

| Scenario | Mechanism | Impact |
|----------|-----------|--------|
| **Cache expiration** | Popular cache key expires; 1,000 threads hit the database simultaneously | Database overwhelmed; slow response; more cache misses |
| **Service recovery** | Circuit breaker half-opens; all waiting requests flood the recovering service | Service fails again immediately |
| **Cron overlap** | Slow job still running when next execution triggers | Double resource consumption; data corruption |
| **Lock release** | Mutex released; all waiting threads resume simultaneously | Resource spike; possible re-contention |

### Prevention

- **Cache stampede prevention:** Use probabilistic early expiration, lock-based recomputation (only one thread refreshes), or serve stale + refresh in background
- **Circuit breaker recovery:** Half-open state allows only a small number of test requests through
- **Cron jobs:** Use distributed locks or leader election to ensure only one instance runs
- **Gradual ramp-up:** When recovering, slowly increase traffic rather than allowing full load immediately

---

## 9. Slow Responses (Worse Than No Response)

A fast failure is annoying. A slow failure is catastrophic. When a system responds slowly instead of failing fast, it creates a chain reaction of blocked resources throughout the calling stack.

### Why Slow Is Worse Than Down

| Fast Failure | Slow Failure |
|-------------|-------------|
| Circuit breaker trips immediately | Circuit breaker does not trip (still getting responses) |
| Thread released after milliseconds | Thread blocked for seconds or minutes |
| Error is visible and actionable | Problem is invisible until pool exhaustion |
| Users see error, retry once, move on | Users see spinner, refresh, multiply load |
| Affects one request | Blocks a thread, affecting all subsequent requests |

### Detection

- Monitor latency at p99 and p999, not just p50
- Alert on latency shifts, not just absolute thresholds
- Track thread pool utilization -- rising active threads with flat throughput indicates blocking
- Trace slow requests across service boundaries to find the bottleneck

### Prevention

- Set aggressive read timeouts (seconds, not minutes)
- Implement deadline propagation: if the user's request has 5 seconds left, do not start a 10-second operation
- Use asynchronous processing for long operations
- Implement load shedding: reject requests when response time exceeds SLO

---

## 10. Unbounded Result Sets

A query that returns 10 rows in development returns 10 million rows in production. Unbounded result sets are a time bomb that detonates when data grows beyond test assumptions.

### Common Manifestations

| Query Pattern | Development | Production |
|--------------|-------------|------------|
| `SELECT * FROM orders WHERE user_id = ?` | 3 orders | 50,000 orders (power user) |
| `SELECT * FROM events WHERE date > ?` | 100 events | 2 million events (6 months of data) |
| `SELECT * FROM logs` | 500 rows | Out of memory |
| API response with nested objects | Small JSON | 200MB JSON response |

### Prevention

- **Always paginate:** Every list query must have a `LIMIT` and `OFFSET` (or cursor-based pagination)
- **Cap result sizes:** Enforce maximum result count at the API layer, even if the caller does not request it
- **Stream large results:** Use cursors or streaming for batch operations rather than loading everything into memory
- **Monitor query performance:** Track query execution time and result set sizes in production
- **Test with production-scale data:** Use anonymized production data volumes in performance testing

### Rule of Thumb

If your code ever does `results = query.getAll()` without a limit, it is a production incident waiting to happen. Every query, every API response, every list operation must have an upper bound.

# Capacity Planning

Capacity planning is the discipline of understanding how much load your system can handle, what breaks first, and how to scale before users experience degradation. It is not a one-time exercise -- it is a continuous practice that evolves as your system, traffic patterns, and infrastructure change.

## Performance Testing Taxonomy

Not all performance tests are equal. Each type answers a different question.

### Test Types

| Test Type | Question It Answers | Duration | Load Profile |
|-----------|-------------------|----------|-------------|
| **Load test** | Can the system handle expected peak traffic? | 30-60 minutes | Ramp to expected peak, hold steady |
| **Stress test** | Where does the system break? | Until failure | Ramp beyond expected peak until degradation or failure |
| **Soak test** | Does the system degrade over time? | 24-72 hours | Sustained load at 70-80% of capacity |
| **Spike test** | How does the system handle sudden bursts? | 15-30 minutes | Sudden jump from baseline to peak, then back |
| **Scalability test** | Does adding resources improve throughput linearly? | Variable | Measure throughput at different resource levels |

### Load Test Design

A good load test simulates real user behavior, not synthetic happy paths.

**Essential elements:**
- **Realistic user journeys:** Mix of browse, search, add to cart, checkout -- not just one endpoint
- **Think time:** Users do not fire requests as fast as possible; include realistic pauses between actions
- **Data variation:** Different users, different products, different search terms -- not the same request repeated
- **Ramp-up period:** Gradually increase load to avoid a cold-start stampede
- **Steady state period:** Hold at target load long enough to observe stabilization (minimum 15 minutes)
- **Ramp-down period:** Gradually decrease load to observe resource release

### Stress Test Design

The goal is to find the breaking point -- not to prove the system works under normal load.

**Key principles:**
- Increase load incrementally (e.g., 10% every 5 minutes) until you observe degradation
- Monitor all resources: CPU, memory, disk I/O, network, thread pools, connection pools, queue depths
- Record the exact load level when each metric crosses its threshold
- Document the failure mode: does the system degrade gracefully (latency increases, then errors) or fail catastrophically (crash, hang, data corruption)?
- Run the stress test multiple times to verify consistency

### Soak Test Design

Soak tests reveal problems that only manifest over time.

**What soak tests catch:**

| Problem | Mechanism | Detection |
|---------|-----------|-----------|
| **Memory leaks** | Gradual memory growth from unreleased objects | Memory usage trends upward over hours |
| **Connection leaks** | Connections borrowed from pool but never returned | Pool exhaustion after hours of operation |
| **File handle leaks** | Files opened but never closed | "Too many open files" errors after prolonged operation |
| **Log file growth** | Disk fills over extended operation | Disk utilization climbs throughout test |
| **Cache bloat** | Cache grows without eviction under sustained load | Memory or disk consumption increases monotonically |
| **Database bloat** | Temp tables, uncommitted transactions accumulate | Database performance degrades over test duration |

**Soak test requirements:**
- Run at 70-80% of measured capacity (not full stress -- you are testing endurance, not peak)
- Duration: minimum 24 hours, ideally 72 hours
- Monitor resource trends, not just snapshots -- a flat graph is healthy, a rising trend is a leak
- Compare start-of-test and end-of-test resource consumption

---

## Resource Pool Management

Resource pools -- thread pools, connection pools, object pools -- are finite and shared. Mismanaging them is one of the most common causes of production failures.

### Connection Pool Sizing

The most common question: "How many connections do I need?"

**Formula:**
```
pool_size = peak_concurrent_requests × avg_hold_time / avg_request_time
```

**But in practice:**
- Measure actual concurrent active connections under peak load
- Set pool size to measured p99 concurrency + 20-30% headroom
- Set a maximum that protects the downstream resource (databases have their own connection limits)
- Too many connections: each consumes memory on both client and server; database performance degrades with too many connections
- Too few connections: requests queue waiting for a connection; latency increases; pool exhaustion looks like a database outage

### Connection Pool Configuration

| Parameter | Purpose | Typical Value |
|-----------|---------|---------------|
| **Minimum pool size** | Connections maintained even when idle | 5-10 |
| **Maximum pool size** | Hard upper limit on connections | Based on measurement |
| **Checkout timeout** | How long to wait for a connection from the pool | 500ms - 2s |
| **Idle timeout** | How long an unused connection stays in the pool | 5-10 minutes |
| **Max lifetime** | Maximum age of a connection before forced recycling | 30-60 minutes |
| **Validation query** | Query to verify connection health before use | `SELECT 1` |
| **Validation interval** | How often idle connections are validated | 30-60 seconds |

### Connection Pool Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **No checkout timeout** | Thread waits forever for a connection | Set checkout timeout to 1-2 seconds |
| **No max lifetime** | Stale connections cause intermittent errors | Recycle connections every 30-60 minutes |
| **Pool size = DB max connections** | Leaves no connections for admin, monitoring, or other services | Pool size = (DB max - reserved) / number of application instances |
| **Ignoring connection leaks** | Pool slowly drains until exhaustion | Monitor borrowed-vs-returned; log leaked connections |
| **Default pool size** | Either wastes resources or causes starvation | Size based on measured concurrency |

---

## Thread Pool Management

Thread pools control the concurrency of your application. Getting them right is critical for both throughput and stability.

### Thread Pool Sizing

**CPU-bound workloads:**
```
threads = number_of_cores
```

**I/O-bound workloads (most web applications):**
```
threads = number_of_cores × (1 + wait_time / service_time)
```

Example: 8 cores, requests spend 80% of time waiting on I/O:
```
threads = 8 × (1 + 80/20) = 8 × 5 = 40 threads
```

### Thread Pool Configuration

| Parameter | Purpose | Consideration |
|-----------|---------|---------------|
| **Core pool size** | Threads always kept alive | Handles normal load without thread creation overhead |
| **Maximum pool size** | Hard upper limit | Handles burst load; too high causes context-switching overhead |
| **Queue capacity** | Work queue between core and max | Bounded queue with rejection policy; never unbounded |
| **Keep-alive time** | How long excess threads survive when idle | 30-60 seconds; balances responsiveness and resource usage |
| **Rejection policy** | What happens when pool and queue are both full | Reject immediately (fail fast) or caller-runs (back-pressure) |

### Thread Pool Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **Unbounded queue** | Memory grows until OOM; latency climbs invisibly | Use bounded queue; fail fast when full |
| **Single shared pool** | One slow operation starves all others | Separate pools per workload type |
| **Too many threads** | Context-switching overhead exceeds throughput gain | Measure throughput at different pool sizes; find the plateau |
| **No monitoring** | Thread starvation goes undetected until outage | Monitor active/idle/queued counts; alert on pool saturation |

---

## The Universal Scalability Law

The Universal Scalability Law (USL), developed by Neil Gunther, models how system throughput changes as you add resources (servers, threads, cores).

### The Model

```
C(N) = N / (1 + σ(N-1) + κN(N-1))
```

Where:
- **N** = number of processors/servers/threads
- **σ** (sigma) = contention parameter: fraction of work that must be serialized
- **κ** (kappa) = coherence parameter: cost of keeping shared state consistent
- **C(N)** = relative capacity at N resources

### Key Insights

| Parameter | Effect | Example |
|-----------|--------|---------|
| **σ = 0, κ = 0** | Linear scalability (ideal but impossible) | Adding 10 servers = 10x throughput |
| **σ > 0, κ = 0** | Amdahl's Law: diminishing returns | Shared lock limits parallelism |
| **σ > 0, κ > 0** | Retrograde scalability: adding resources decreases throughput | Distributed cache coherence overhead exceeds throughput gain |

### Practical Application

1. **Measure throughput** at 1, 2, 4, 8, 16 resources
2. **Fit the USL curve** to find σ and κ
3. **Predict the scalability ceiling:** the point where adding more resources stops helping (or hurts)
4. **Identify the bottleneck:** high σ means contention (locks, serialization); high κ means coherence costs (cache invalidation, distributed consensus)

---

## Capacity Modeling

A capacity model documents the relationship between load, resources, and performance for each service in your system.

### Capacity Model Template

For each service, document:

| Dimension | Current Value | Limit | Action at Limit |
|-----------|--------------|-------|-----------------|
| **Requests/sec** | 500 RPS | 2,000 RPS | Scale horizontally |
| **CPU** | 40% avg, 70% peak | 80% sustained | Add instances |
| **Memory** | 2.5 GB / 4 GB | 3.5 GB | Increase instance size or optimize |
| **DB connections** | 30 active / 50 max | 45 active | Increase pool or add read replicas |
| **Disk I/O** | 200 IOPS | 3,000 IOPS (provisioned) | Upgrade storage tier |
| **Network** | 500 Mbps | 10 Gbps | Unlikely bottleneck |

### Bottleneck Resource

Every service has a bottleneck resource -- the resource that runs out first as load increases. The capacity of the service equals the capacity of its bottleneck.

**Finding the bottleneck:**
1. Run a stress test, increasing load gradually
2. Monitor all resources simultaneously
3. The first resource to hit its limit is the bottleneck
4. All capacity planning focuses on this resource

**Common bottlenecks by service type:**

| Service Type | Typical Bottleneck |
|-------------|-------------------|
| API services | Thread pool or connection pool |
| Data-heavy services | Database connections or query throughput |
| Compute-heavy services | CPU |
| File processing | Disk I/O or memory |
| Real-time services | Network bandwidth or connection count |

---

## Capacity Myths

### Myth 1: "The Cloud Is Infinitely Scalable"

Reality:
- Auto-scaling has lag time (1-5 minutes to provision and start new instances)
- Cold starts add latency to the first requests on new instances
- Cloud providers have account-level limits (instance count, API rate limits)
- Some resources do not scale horizontally (relational databases, stateful services)
- Scaling costs money -- infinite scale means infinite cost

### Myth 2: "We'll Just Add More Servers"

Reality:
- Adding servers only helps if the bottleneck is CPU or memory on the application tier
- If the bottleneck is the database, adding application servers makes it worse (more connections, more load on the same database)
- Network hops, serialization overhead, and coordination costs increase with more servers
- Horizontal scaling requires stateless design -- session affinity, local caches, and local file storage break horizontal scaling

### Myth 3: "Our Load Tests Pass, So We're Fine"

Reality:
- Load tests with synthetic data miss hot spots in production data
- Load tests rarely simulate realistic user behavior (think times, session patterns, edge cases)
- Load test environments rarely match production topology, network latency, or data volume
- Load tests find throughput limits but not endurance problems (need soak tests)
- Passing a load test at 2x expected peak does not protect against 10x flash crowds

### Myth 4: "We Don't Need Capacity Planning -- We Have Auto-Scaling"

Reality:
- Auto-scaling reacts to load after it arrives; capacity planning anticipates load before it arrives
- Auto-scaling cannot protect against instant traffic spikes (Black Friday, viral events)
- Auto-scaling policies themselves need testing -- misconfigured policies can scale in the wrong direction or oscillate
- Cost management requires understanding baseline and peak capacity needs

---

## Performance Anti-Patterns

### Resource Contention

Multiple threads competing for the same resource (lock, connection, CPU core). Throughput plateaus or decreases as concurrency increases.

**Detection:** Throughput does not increase when adding threads/instances. CPU utilization is low despite high load. Thread dumps show threads waiting on locks.

**Fix:** Reduce lock scope. Use lock-free data structures. Partition data to reduce contention. Use read-write locks instead of exclusive locks.

### The Coordinated Omission Problem

Load testing tools that wait for a response before sending the next request undercount latency at high load. When the server slows down, the tool also slows down, making the measured throughput look stable while actually masking massive latency increases.

**Detection:** Load test shows consistent throughput even as the system degrades. Real users report much worse latency than load tests measure.

**Fix:** Use load testing tools that support coordinated omission correction (e.g., wrk2, Gatling with constant throughput mode). Measure latency independently of throughput. Use open-loop load generators that send requests at a fixed rate regardless of response time.

### N+1 Query Problem

Fetching a list of N items, then making one additional query for each item. Total queries = N + 1 instead of 1 or 2.

**Detection:** Database query count scales linearly with result set size. Response time increases linearly with page size.

**Fix:** Use eager loading / JOIN queries. Batch queries (`WHERE id IN (...)`). Implement DataLoader pattern for GraphQL.

# Chaos Engineering

Chaos engineering is the discipline of experimenting on a system in order to build confidence in its ability to withstand turbulent conditions in production. It is not about breaking things for fun -- it is a rigorous, scientific approach to discovering weaknesses before they cause outages.

> **Safety note:** This reference describes chaos engineering *concepts and planning patterns*. All failure injection experiments must be performed by authorized engineers using dedicated chaos tooling (e.g., Gremlin, Litmus, AWS Fault Injection Simulator) with proper approvals, blast radius controls, monitoring, and rollback plans. Commands shown are for reference only -- never run them without authorization and safeguards.

The fundamental insight is simple: you cannot know how your system handles failure until it actually fails. Waiting for production incidents to discover weaknesses is reactive and expensive. Chaos engineering is proactive and controlled.


## Table of Contents
1. [Principles of Chaos Engineering](#principles-of-chaos-engineering)
2. [Chaos Experiment Design](#chaos-experiment-design)
3. [Failure Injection Techniques](#failure-injection-techniques)
4. [Chaos Monkey and Netflix's Approach](#chaos-monkey-and-netflixs-approach)
5. [GameDay Exercises](#gameday-exercises)
6. [Building Confidence Through Controlled Failure](#building-confidence-through-controlled-failure)

---

## Principles of Chaos Engineering

### 1. Define Steady State

Before you can detect abnormal behavior, you must define what normal looks like. Steady state is expressed as measurable business or system metrics that indicate the system is functioning correctly.

**Good steady state definitions:**

| Metric Type | Steady State Definition | Example |
|-------------|----------------------|---------|
| **Business metric** | Orders per minute within expected range | 100-150 orders/min during business hours |
| **Error rate** | Below defined threshold | < 0.1% 5xx errors |
| **Latency** | Within SLO bounds | p99 latency < 500ms |
| **Throughput** | Within expected range | 1000-2000 RPS |
| **Availability** | All critical paths responding | Health checks green on all services |

**Bad steady state definitions:**
- "The system is working" (not measurable)
- "No alerts firing" (absence of evidence is not evidence of absence)
- "CPU below 80%" (cause-based, not symptom-based)

### 2. Formulate a Hypothesis

Every chaos experiment starts with a hypothesis: a prediction about what will happen when you inject a specific failure.

**Hypothesis format:**
```
"We believe that when [failure condition], the system will [expected behavior],
as measured by [steady state metric] remaining within [acceptable bounds]."
```

**Example hypotheses:**

| Failure | Hypothesis | Metric |
|---------|-----------|--------|
| Terminate one API instance (via chaos tooling) | System continues serving traffic with no user-visible errors | Error rate stays < 0.1% |
| Add 500ms latency to database (via chaos tooling) | Response time degrades but stays within SLO; circuit breaker does not trip | p99 < 2s; no circuit breaker events |
| Payment service returns 503 (via fault injection proxy) | Checkout shows graceful error; other features unaffected | Non-checkout error rate unchanged |
| Disk at 95% capacity (via chaos tooling) | Log rotation triggers; alerts fire; no service disruption | Disk drops below 90% within 10 minutes |

### 3. Introduce Real-World Failures

Chaos experiments should simulate failures that actually happen in production, not theoretical edge cases.

**Common failure types to simulate (via dedicated chaos tooling):**

| Category | Failures | Tooling Examples |
|----------|----------|-----------------|
| **Infrastructure** | Instance crash, disk failure, network partition | Gremlin, Litmus, AWS FIS |
| **Network** | Latency, packet loss, DNS failure | Toxiproxy, Istio fault injection, tc (traffic control) |
| **Application** | Memory pressure, CPU saturation, thread contention | stress-ng (controlled), Gremlin resource attacks |
| **Dependency** | Service unavailable, slow response, corrupt response | Toxiproxy, Envoy fault injection, mock services |
| **Cloud** | AZ failure, region degradation, API throttling | AWS FIS, GCP Fault Injection, Azure Chaos Studio |

### 4. Run in Production

Staging environments do not reproduce the complexity of production. They lack real user traffic, real data volumes, real concurrency patterns, and real interactions between services. Chaos experiments in staging build false confidence.

**But safely:**
- Start with non-production, then graduate to production
- Use the smallest blast radius possible
- Have an emergency stop mechanism to halt the experiment immediately
- Run during business hours when the team is available to respond
- Inform the on-call team before running experiments
- Never experiment during peak traffic or known risky periods

### 5. Automate and Run Continuously

A chaos experiment that runs once proves resilience at one point in time. Automated, recurring experiments prove resilience continuously.

**Automation maturity levels:**

| Level | Practice | Confidence |
|-------|----------|-----------|
| **Manual** | Engineer runs experiment by hand, observes results | Low -- depends on who runs it and when |
| **Scripted** | Experiment codified in a script, run on schedule | Medium -- repeatable but requires human analysis |
| **Automated** | Experiment runs automatically, evaluates steady state, reports results | High -- continuous verification |
| **Integrated** | Experiments run in CI/CD pipeline; failing experiment blocks deployment | Very high -- resilience is a deployment gate |

---

## Chaos Experiment Design

### Experiment Template

```
Experiment: [Name]
Date: [When]
Team: [Who is running it]
Blast Radius: [What is affected]

Hypothesis:
  When [failure condition], we expect [behavior],
  as measured by [metric] remaining within [bounds].

Steady State:
  - Metric 1: [current value, acceptable range]
  - Metric 2: [current value, acceptable range]

Method:
  1. Verify steady state
  2. Inject [specific failure]
  3. Observe for [duration]
  4. Measure [metrics]
  5. Remove failure injection
  6. Verify recovery to steady state

Abort Conditions:
  - [Metric] exceeds [threshold]
  - On-call pages for [service]
  - Customer-visible impact detected

Results:
  - Hypothesis confirmed/refuted
  - Observations: [what happened]
  - Action items: [what to fix]
```

### Blast Radius Management

Blast radius is the scope of impact if the experiment causes unexpected damage. Always minimize blast radius and expand gradually.

**Blast radius levels:**

| Level | Scope | When to Use |
|-------|-------|-------------|
| **Development** | Single developer's environment | First-time experiments, unproven hypotheses |
| **Staging** | Staging environment | Validating experiment mechanics before production |
| **Canary** | Small subset of production (1-5%) | First production experiment for a new failure type |
| **Single AZ** | One availability zone in production | Testing AZ failure resilience |
| **Full production** | All production traffic | Well-understood experiments that have been run many times |

**Blast radius controls:**
- **Targeting:** Limit experiment to specific instances, user segments, or traffic percentage
- **Duration:** Set maximum experiment duration; auto-revert after timeout
- **Emergency stop:** One-button (or automatic) experiment termination
- **Monitoring:** Real-time dashboards showing experiment impact on steady state metrics
- **Rollback:** Pre-planned steps to undo the experiment if things go wrong

---

## Failure Injection Techniques

### Process-Level Failures

| Technique | What It Simulates | Chaos Tooling |
|-----------|-------------------|--------------|
| Instance termination | Crash | Gremlin process attack, Kubernetes pod disruption budget, Litmus ChaosEngine |
| Process freeze | Hang/unresponsive | Gremlin process attack (pause), Litmus pod-cpu-hog |
| CPU saturation | Compute pressure | Gremlin CPU attack, Litmus cpu-hog, stress-ng (controlled, authorized) |
| Memory pressure | Memory exhaustion | Gremlin memory attack, Litmus pod-memory-hog |
| Process flood | Process table exhaustion | Gremlin process attack with controlled parameters |

### Network-Level Failures

| Technique | What It Simulates | Chaos Tooling |
|-----------|-------------------|--------------|
| Latency injection | Slow network | Toxiproxy, Gremlin latency attack, Istio fault injection |
| Packet loss | Unreliable network | Gremlin packet loss attack, Toxiproxy, tc netem (authorized) |
| Network partition | Network split | Gremlin blackhole attack, Litmus pod-network-partition |
| DNS failure | DNS outage | Gremlin DNS attack, Litmus pod-dns-error |
| Bandwidth limit | Constrained network | Toxiproxy bandwidth limit, Gremlin bandwidth attack |

### Dependency-Level Failures

| Technique | What It Simulates | Chaos Tooling |
|-----------|-------------------|--------------|
| Error injection proxy | Downstream errors | Toxiproxy, Envoy fault injection |
| Latency injection | Slow dependency | Toxiproxy, Istio fault injection |
| Connection limit | Pool exhaustion | Toxiproxy connection limit, Gremlin blackhole |
| Response corruption | Data integrity issues | Custom fault injection proxy |
| Certificate expiration | TLS failures | Expired test certificate in staging |

### Disk-Level Failures

| Technique | What It Simulates | Chaos Tooling |
|-----------|-------------------|--------------|
| Disk pressure | Disk full | Gremlin disk attack, Litmus disk-fill |
| Slow I/O | Storage degradation | Gremlin IO attack, dm-delay (authorized) |
| Read-only filesystem | Mount failure | Gremlin disk attack (read-only mode) |
| Data corruption | Integrity issues | Controlled corruption of test data in staging |

---

## Chaos Monkey and Netflix's Approach

Netflix pioneered chaos engineering with Chaos Monkey, which randomly terminates production instances during business hours using automated, authorized tooling with built-in safeguards.

### The Netflix Chaos Engineering Stack

| Tool | What It Does | Scope |
|------|-------------|-------|
| **Chaos Monkey** | Terminates random instances (automated, authorized) | Single instance |
| **Chaos Kong** | Simulates entire region failure | Region |
| **Latency Monkey** | Injects network latency | Network |
| **Conformity Monkey** | Finds instances not following best practices | Compliance |
| **Chaos Automation Platform (ChAP)** | Runs automated experiments with steady state comparison | Full stack |

### Key Lessons from Netflix

1. **Start small:** Chaos Monkey terminates one instance at a time. Only after years of practice did Netflix graduate to region-level chaos (Chaos Kong).
2. **Business hours only:** Run experiments when the team is available to respond. Late-night chaos is just an outage.
3. **Opt-out, not opt-in:** By default, all services are enrolled. Teams must explicitly justify opting out.
4. **No blame:** Finding a weakness is a success, not a failure. Teams that discover and fix problems through chaos engineering are celebrated.
5. **Invest in tooling:** Manual chaos is not sustainable. Invest in platforms that automate experiment execution, evaluation, and reporting.

---

## GameDay Exercises

A GameDay is a scheduled exercise where a team practices responding to a realistic failure scenario. It combines chaos engineering (inject failure) with incident response practice (detect, diagnose, mitigate, resolve).

### GameDay Structure

| Phase | Duration | Activities |
|-------|----------|-----------|
| **Preparation** | 1-2 weeks before | Define scenario, brief participants, set up monitoring |
| **Pre-game** | 30 minutes | Verify steady state, confirm all participants are ready |
| **Execution** | 1-3 hours | Inject failure, observe team response, take notes |
| **Post-game** | 1 hour | Debrief, identify what worked and what did not |
| **Follow-up** | 1-2 weeks after | File action items, track remediation, schedule next GameDay |

### GameDay Scenarios

| Scenario | Complexity | What It Tests |
|----------|-----------|---------------|
| Terminate a single instance | Low | Auto-healing, health checks, load balancing |
| Simulate database failover | Medium | Connection handling, read replica routing, data consistency |
| Full AZ failure | High | Multi-AZ architecture, DNS failover, stateful service recovery |
| Dependency outage (payment provider) | Medium | Circuit breakers, fallback behavior, user communication |
| Security incident (compromised credentials) | High | Credential rotation, access logging, incident response process |
| Data corruption | High | Backup restoration, data validation, recovery time |

### GameDay Ground Rules

1. **Safety first:** A facilitator can halt the exercise at any time if real customer impact occurs
2. **No blame:** The goal is learning, not grading individual performance
3. **Document everything:** Notes, timestamps, decisions, and outcomes
4. **Include diverse roles:** Engineers, SREs, product managers, customer support
5. **Realistic but controlled:** Use production systems but with blast radius limits
6. **Schedule during business hours:** The team should be fully staffed and alert
7. **Announce to stakeholders:** Customer support and leadership should know a GameDay is happening

### GameDay Facilitation Tips

- **The facilitator does not fix things.** Their job is to inject failures, observe the response, and take notes.
- **Start with known weaknesses.** The first GameDay should test a scenario the team suspects might fail -- the learning is in confirming the suspicion and practicing the response.
- **Increase difficulty over time.** First GameDay: terminate one instance. Tenth GameDay: simultaneous database failover + network partition + on-call engineer is unavailable.
- **Celebrate findings.** Every weakness discovered in a GameDay is a weakness that will not cause a real outage. This is a win.

---

## Building Confidence Through Controlled Failure

### The Confidence Curve

```
Confidence
    ▲
    │                              ╭───────────
    │                         ╭────╯
    │                    ╭────╯
    │               ╭────╯
    │          ╭────╯
    │     ╭────╯
    │╭────╯
    └──────────────────────────────────────────→ Experiments
      0    10    20    30    40    50    60
```

Each successful chaos experiment increases confidence that the system will survive that failure in production. Each failed experiment (where the hypothesis was disproven) reveals a weakness that, once fixed, increases actual resilience.

### Maturity Model

| Level | Practice | Organization |
|-------|----------|-------------|
| **Level 0: Reactive** | No chaos engineering; learn from production incidents | Firefighting culture |
| **Level 1: Exploratory** | Manual, ad-hoc experiments; individual teams | Curious early adopters |
| **Level 2: Systematic** | Regular GameDays; documented experiments; shared learnings | Engineering-wide practice |
| **Level 3: Automated** | Continuous automated experiments; integrated into CI/CD | Platform team provides tooling |
| **Level 4: Cultural** | Chaos engineering is expected for all services; opt-out requires justification | Resilience is a core engineering value |

### Getting Started

If you have never done chaos engineering, start here:

1. **Pick one service** -- the one you are most worried about
2. **Define its steady state** -- what metrics prove it is working?
3. **Form one hypothesis** -- "If we terminate one instance, the service will continue serving traffic"
4. **Run the experiment in staging** -- verify your tooling and monitoring work
5. **Run the experiment in production** -- with minimal blast radius and an emergency stop mechanism
6. **Document the results** -- what happened? Was the hypothesis confirmed?
7. **Fix what you found** -- if the hypothesis was disproven, fix the weakness
8. **Repeat** -- expand to more failure types, more services, more complex scenarios

### Common Objections and Responses

| Objection | Response |
|-----------|---------|
| "We can't break production on purpose!" | You are already breaking production accidentally. Chaos engineering lets you do it on your terms, when you are prepared. |
| "Our system isn't resilient enough for chaos" | That is exactly why you need chaos engineering -- to find and fix the weaknesses. Start small. |
| "We don't have time for this" | You have time for incident response, post-mortems, and customer apologies. Chaos engineering reduces all three. |
| "What if we cause an outage?" | Start with minimal blast radius in staging. Terminate one process. If that causes an outage, you have learned something invaluable. |
| "Management won't approve this" | Frame it as risk reduction, not risk creation. Show the cost of recent outages vs. the cost of preventive experiments. |

### Anti-Fragility

The ultimate goal of chaos engineering is not just resilience (surviving failure) but anti-fragility (getting stronger from failure). A system is anti-fragile when each failure makes it more resistant to future failures.

**How chaos engineering builds anti-fragility:**
- Each experiment reveals a weakness
- Each fix removes that weakness permanently
- Each automated experiment continuously verifies the fix
- Over time, the system has been tested against every common failure mode
- New failure modes are discovered faster because the team has built the muscles and tooling to find them

This is the Release It! philosophy in action: production-ready software is not software that never fails. It is software that has been designed, tested, and operated to handle failure gracefully -- because failure is not a possibility, it is a certainty.

# Deployment Strategies

The distinction between deployment (placing code on infrastructure) and release (exposing code to users) is fundamental. Conflating the two means every deployment is a high-risk event. Separating them gives you the ability to deploy with confidence, test in production, and release when ready.

Most production outages are caused by changes. Deployment strategies exist to make changes safe.


## Table of Contents
1. [Zero-Downtime Deployment](#zero-downtime-deployment)
2. [Rolling Deployment](#rolling-deployment)
3. [Blue-Green Deployment](#blue-green-deployment)
4. [Canary Releases](#canary-releases)
5. [Feature Flags](#feature-flags)
6. [Database Migrations Without Downtime](#database-migrations-without-downtime)
7. [Immutable Infrastructure](#immutable-infrastructure)
8. [Infrastructure as Code](#infrastructure-as-code)
9. [Rollback Strategy](#rollback-strategy)

---

## Zero-Downtime Deployment

Zero-downtime deployment is not optional for any system with users. Users should never see an error page because you are deploying code.

### Prerequisites for Zero-Downtime Deployment

| Requirement | Why |
|-------------|-----|
| **Backward-compatible changes** | Old code and new code run simultaneously during deployment |
| **Graceful shutdown** | In-flight requests must complete before an instance is terminated |
| **Health checks** | Load balancer must know when a new instance is ready to receive traffic |
| **Session independence** | Requests from the same user can be routed to any instance |
| **Database compatibility** | Schema changes must work with both old and new application code |

---

## Rolling Deployment

The simplest zero-downtime strategy. Replace instances one at a time (or in small batches), verifying health after each replacement.

### How It Works

```
Cluster: [v1] [v1] [v1] [v1] [v1]

Step 1:  [v2] [v1] [v1] [v1] [v1]   ← Deploy to instance 1, verify health
Step 2:  [v2] [v2] [v1] [v1] [v1]   ← Deploy to instance 2, verify health
Step 3:  [v2] [v2] [v2] [v1] [v1]   ← Deploy to instance 3, verify health
Step 4:  [v2] [v2] [v2] [v2] [v1]   ← Deploy to instance 4, verify health
Step 5:  [v2] [v2] [v2] [v2] [v2]   ← Deploy to instance 5, verify health
```

### Configuration Parameters

| Parameter | Description | Typical Value |
|-----------|-------------|---------------|
| **Max unavailable** | Maximum instances being replaced simultaneously | 1 or 25% |
| **Max surge** | Extra instances during deployment | 1 or 25% |
| **Readiness probe** | Health check before receiving traffic | HTTP 200 on `/ready` |
| **Min ready seconds** | How long instance must be healthy before proceeding | 30-60 seconds |

### Advantages and Limitations

| Advantage | Limitation |
|-----------|-----------|
| Simple to implement | Both versions run simultaneously (must be compatible) |
| Gradual rollout | Rollback requires re-deploying old version |
| No extra infrastructure | Reduced capacity during deployment |
| Built into Kubernetes | Slow for large clusters |

---

## Blue-Green Deployment

Maintain two identical production environments. One (blue) serves live traffic. Deploy to the other (green), verify, and switch the router.

### How It Works

```
                    ┌──────────────┐
Users → Router ────→│  Blue (v1)   │  ← Currently live
                    └──────────────┘
                    ┌──────────────┐
                    │  Green (v2)  │  ← Deploy here, test
                    └──────────────┘

After verification:

                    ┌──────────────┐
                    │  Blue (v1)   │  ← Standby (instant rollback)
                    └──────────────┘
                    ┌──────────────┐
Users → Router ────→│  Green (v2)  │  ← Now live
                    └──────────────┘
```

### Implementation Steps

1. Deploy new version to green environment
2. Run automated smoke tests against green
3. Optionally, route internal/test traffic to green for manual verification
4. Switch the router (load balancer, DNS, or service mesh) to green
5. Monitor for errors -- if problems arise, switch back to blue immediately
6. Keep blue running as the rollback target until the next deployment

### Advantages and Limitations

| Advantage | Limitation |
|-----------|-----------|
| Instant rollback (switch router back) | Requires double the infrastructure |
| Full testing in production-like environment | Database changes must be compatible with both versions |
| Zero capacity reduction during deployment | Stateful applications need careful session handling |
| Clear separation of deploy and release | Cost of maintaining two full environments |

---

## Canary Releases

Route a small percentage of production traffic to the new version. Monitor for errors. Gradually increase the percentage if healthy.

### How It Works

```
Phase 1: 5% → canary (v2), 95% → stable (v1)
Phase 2: 25% → canary (v2), 75% → stable (v1)
Phase 3: 50% → canary (v2), 50% → stable (v1)
Phase 4: 100% → canary (v2), 0% → stable (v1)
```

### Canary Evaluation Criteria

At each phase, evaluate before proceeding:

| Metric | Threshold | Action if Exceeded |
|--------|-----------|-------------------|
| **Error rate** | > 1% above baseline | Automatic rollback |
| **p99 latency** | > 2x baseline | Pause and investigate |
| **CPU usage** | > 80% sustained | Pause and investigate |
| **Memory usage** | Growing trend (potential leak) | Rollback |
| **Business metrics** | Conversion rate drops > 5% | Rollback |

### Automated Canary Analysis

Progressive delivery tools can automate canary evaluation:

1. Deploy canary with 5% traffic
2. Collect metrics for evaluation window (10-30 minutes)
3. Compare canary metrics against baseline (stable version) using statistical analysis
4. If pass: increase traffic percentage and repeat
5. If fail: automatic rollback and alert the team

### Canary vs. Blue-Green

| Aspect | Canary | Blue-Green |
|--------|--------|-----------|
| **Risk exposure** | Minimal (small % of traffic) | All-or-nothing switch |
| **Rollback speed** | Instant (route away from canary) | Instant (switch router) |
| **Infrastructure cost** | Minimal extra (small canary fleet) | Double infrastructure |
| **Verification depth** | Real user traffic at scale | Synthetic tests + internal traffic |
| **Complexity** | Higher (traffic splitting, metric comparison) | Lower (router switch) |

---

## Feature Flags

Feature flags (also called feature toggles) decouple deployment from release. Code is deployed but not activated until the flag is enabled.

### Types of Feature Flags

| Type | Lifetime | Purpose | Example |
|------|----------|---------|---------|
| **Release flag** | Days to weeks | Gate incomplete or untested features | `new_checkout_flow` |
| **Experiment flag** | Weeks to months | A/B testing and gradual rollout | `show_recommendations_v2` |
| **Ops flag** | Permanent | Runtime control over system behavior | `enable_expensive_query_cache` |
| **Kill switch** | Permanent | Disable features during incidents | `disable_search_suggestions` |

### Feature Flag Best Practices

**Do:**
- Use a centralized flag management system (not config files or environment variables)
- Set a default value for every flag (what happens if the flag service is down?)
- Clean up release flags after full rollout (flag debt is real technical debt)
- Log flag evaluations for debugging ("User X saw feature Y because flag Z was true")
- Test both flag states in your test suite

**Do not:**
- Use feature flags for long-lived branching (creates combinatorial testing nightmare)
- Nest feature flags (flag A enables feature which checks flag B -- unmaintainable)
- Deploy code that only works with the flag enabled (always support both states)
- Forget to remove old flags (accumulation makes the code unreadable)

### Feature Flag and Deployment Pipeline

```
1. Developer merges code with feature flag (default: off)
2. Code deploys to production (flag off -- no user impact)
3. QA enables flag for internal users and tests
4. Product enables flag for 5% of users (canary)
5. Monitor metrics for 24-48 hours
6. Ramp to 25%, 50%, 100% with monitoring at each stage
7. When 100% and stable, remove the flag and the old code path
```

---

## Database Migrations Without Downtime

Database schema changes are the most dangerous part of deployment because they are difficult to roll back and must be compatible with both old and new application code during the deployment window.

### The Expand-Contract Pattern

Never make a breaking schema change in a single step. Instead, expand (add), migrate data, then contract (remove).

### Example: Renaming a Column

**Wrong (causes downtime):**
```sql
ALTER TABLE users RENAME COLUMN name TO full_name;
-- Old code looking for "name" column fails immediately
```

**Right (zero-downtime expand-contract):**

| Step | Migration | Application Code |
|------|-----------|-----------------|
| 1. Expand | `ALTER TABLE users ADD COLUMN full_name VARCHAR(255);` | Writes to both `name` and `full_name` |
| 2. Backfill | `UPDATE users SET full_name = name WHERE full_name IS NULL;` | Reads from `full_name`, falls back to `name` |
| 3. Switch | No schema change | Reads and writes only `full_name` |
| 4. Contract | `ALTER TABLE users DROP COLUMN name;` | Only uses `full_name` |

Each step is a separate deployment. Each step is individually rollback-safe.

### Example: Adding a NOT NULL Column

**Wrong (locks table, breaks old code):**
```sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) NOT NULL DEFAULT 'pending';
-- On large tables, this locks the table for minutes/hours
```

**Right (phased approach):**

| Step | Action |
|------|--------|
| 1 | Add column as nullable: `ALTER TABLE orders ADD COLUMN status VARCHAR(20);` |
| 2 | Deploy code that writes `status` on all new rows |
| 3 | Backfill existing rows: `UPDATE orders SET status = 'pending' WHERE status IS NULL;` (in batches) |
| 4 | Add NOT NULL constraint: `ALTER TABLE orders ALTER COLUMN status SET NOT NULL;` |

### Migration Safety Checklist

- [ ] Can the old code work with the new schema?
- [ ] Can the new code work with the old schema (for rollback)?
- [ ] Are large data migrations batched (not one giant UPDATE)?
- [ ] Is the migration tested against production-volume data?
- [ ] Is the migration reversible?
- [ ] Are table locks avoided (no ALTER TABLE on large tables without online DDL)?

---

## Immutable Infrastructure

Never patch, update, or modify a running server. Instead, build a new image with the changes, deploy it, and destroy the old one.

### Why Immutable

| Mutable Infrastructure | Immutable Infrastructure |
|----------------------|-------------------------|
| SSH into servers, apply patches | Build new image with patches baked in |
| Configuration drift over time | Every instance is identical |
| "Snowflake" servers that are irreplaceable | Instances are disposable and replaceable |
| "It works on that server" debugging | Consistent behavior across all instances |
| Manual changes accumulate and are undocumented | All changes are in version control |

### Immutable Infrastructure Pipeline

```
1. Code change committed to version control
2. CI builds application artifact (binary, JAR, bundle)
3. CI builds infrastructure image (Docker image, AMI, VM image)
4. Image is tagged with version and stored in registry
5. Deployment tool replaces old instances with new image
6. Old instances are terminated (not modified, not reused)
```

### Implementation Patterns

| Pattern | Technology | Use Case |
|---------|-----------|----------|
| **Container images** | Docker, OCI | Microservices, cloud-native applications |
| **Machine images** | AMI, GCE image | VM-based workloads |
| **Serverless packages** | Lambda ZIP, Cloud Function | Event-driven workloads |
| **Helm charts** | Kubernetes + Helm | Kubernetes-native applications |

---

## Infrastructure as Code

All infrastructure -- servers, networks, databases, load balancers, DNS entries -- is defined in version-controlled code, not manually configured through web consoles or SSH sessions.

### Principles

| Principle | Practice |
|-----------|----------|
| **Everything in code** | No manual changes; all config in Terraform, CloudFormation, Pulumi, or similar |
| **Version controlled** | Infrastructure code lives in git alongside application code |
| **Reviewed** | Infrastructure changes go through code review like application changes |
| **Tested** | Infrastructure changes are validated in staging before production |
| **Reproducible** | Any environment can be recreated from code in minutes |
| **Idempotent** | Applying the same code twice produces the same result |

### Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **ClickOps** | Changes made through web console are undocumented and unreproducible | Define all infrastructure in code |
| **SSH and modify** | Manual changes create drift between instances | Use immutable infrastructure |
| **Shared credentials** | No audit trail of who changed what | Individual credentials with role-based access |
| **No staging** | Infrastructure changes tested directly in production | Maintain a staging environment that mirrors production topology |
| **Monolithic templates** | One giant infrastructure file that is impossible to review | Modularize infrastructure into composable components |

---

## Rollback Strategy

Rollback must be faster and simpler than rolling forward. If rolling back takes 30 minutes of manual steps, teams will hesitate to deploy -- and when they do deploy, they will hesitate to roll back when things go wrong.

### Rollback Approaches

| Approach | Speed | Complexity | Limitation |
|----------|-------|-----------|-----------|
| **Traffic switch** (blue-green) | Seconds | Low | Requires both versions running |
| **Revert canary** | Seconds | Low | Only for canary percentage |
| **Redeploy previous version** | Minutes | Medium | Requires previous artifact available |
| **Feature flag disable** | Seconds | Low | Only for flag-gated features |
| **Database rollback** | Minutes to hours | High | May require data migration reversal |

### Rollback Checklist

- [ ] Is the previous version's artifact still available in the registry?
- [ ] Can the previous version work with the current database schema?
- [ ] Are feature flags in a state that supports rollback?
- [ ] Has the rollback procedure been tested (not just documented)?
- [ ] Can rollback be executed by the on-call engineer without escalation?
- [ ] Is the rollback time under 5 minutes?

# Observability

You cannot operate what you cannot observe. Observability is the ability to understand the internal state of a system by examining its external outputs. It is not an afterthought or a nice-to-have -- it is a first-class design concern that must be built into every service from day one.

A well-observed system lets you answer questions you did not anticipate at design time. A poorly observed system forces you to deploy new instrumentation during an outage -- exactly when you can least afford the risk.


## Table of Contents
1. [The Three Pillars of Observability](#the-three-pillars-of-observability)
2. [Health Check Patterns](#health-check-patterns)
3. [The RED Method](#the-red-method)
4. [The USE Method](#the-use-method)
5. [SLIs, SLOs, and SLAs](#slis-slos-and-slas)
6. [Alerting Strategy](#alerting-strategy)
7. [Dashboards That Matter](#dashboards-that-matter)

---

## The Three Pillars of Observability

### 1. Structured Logs

Logs answer the question: "What happened?"

**Structured logging** means emitting logs as key-value pairs (JSON), not free-form text. Structured logs are searchable, filterable, and aggregatable. Free-form text logs require regex parsing and break when the format changes.

**Essential log fields:**

| Field | Purpose | Example |
|-------|---------|---------|
| `timestamp` | When the event occurred | `2024-01-15T14:23:45.123Z` |
| `level` | Severity (DEBUG, INFO, WARN, ERROR) | `ERROR` |
| `service` | Which service emitted the log | `payment-service` |
| `trace_id` | Correlation ID across services | `abc123def456` |
| `span_id` | Specific operation within the trace | `span789` |
| `user_id` | Which user was affected (if applicable) | `user_42` |
| `message` | Human-readable description | `Payment processing failed` |
| `error` | Error type and stack trace | `TimeoutException: read timed out` |
| `duration_ms` | How long the operation took | `5230` |
| `request_id` | Unique identifier for the request | `req_abc123` |

**Logging best practices:**
- Log at service boundaries (incoming request, outgoing call, response)
- Include enough context to understand the event without reading code
- Use consistent field names across all services
- Do not log sensitive data (passwords, tokens, PII) -- use redaction
- Log errors with full context, not just the exception message
- Use sampling for high-volume debug logs in production
- Emit logs asynchronously to avoid blocking application threads

### 2. Metrics

Metrics answer the question: "How much?"

Metrics are numeric measurements collected over time. They are cheap to store, fast to query, and essential for dashboards and alerts.

**Metric types:**

| Type | What It Measures | Example |
|------|-----------------|---------|
| **Counter** | Cumulative count of events | `http_requests_total`, `errors_total` |
| **Gauge** | Current value that can go up or down | `active_connections`, `queue_depth`, `cpu_usage` |
| **Histogram** | Distribution of values | `request_duration_seconds` (with buckets for p50, p95, p99) |
| **Summary** | Pre-calculated percentiles | `request_duration_seconds{quantile="0.99"}` |

**Metric naming conventions:**
- Use snake_case: `http_request_duration_seconds`
- Include the unit in the name: `_seconds`, `_bytes`, `_total`
- Use labels for dimensions: `http_requests_total{method="GET", status="200", endpoint="/api/users"}`
- Do not use high-cardinality labels (user IDs, request IDs) -- these explode metric storage

### 3. Distributed Traces

Traces answer the question: "Where did the time go?"

A distributed trace follows a single request across multiple services, showing the sequence of operations, their durations, and their relationships.

**Trace anatomy:**

```
Trace: user request to checkout
├── Span: API Gateway (2ms)
│   ├── Span: Auth Service - validate token (15ms)
│   ├── Span: Cart Service - get cart (45ms)
│   │   └── Span: Database - SELECT cart items (12ms)
│   ├── Span: Inventory Service - check stock (120ms)  ← Bottleneck
│   │   └── Span: Database - SELECT inventory (95ms)
│   └── Span: Payment Service - charge card (230ms)
│       └── Span: Stripe API - create charge (180ms)
Total: 412ms
```

**Trace implementation:**
- Inject trace context (trace ID, span ID) into all outgoing requests (HTTP headers, message metadata)
- Extract trace context from all incoming requests
- Create a new span for each significant operation (service call, database query, cache lookup)
- Annotate spans with relevant metadata (query, parameters, result count)
- Use sampling in production (1-10% of requests) to manage cost and volume
- Always trace errors at 100% (do not sample error traces)

---

## Health Check Patterns

Health checks tell load balancers, orchestrators, and monitoring systems whether an instance is able to serve traffic.

### Shallow Health Check

A shallow health check verifies that the process is running and can respond to HTTP requests. It does not verify dependencies.

```
GET /health
200 OK {"status": "up"}
```

**Use for:** Liveness probes (is the process alive?). If this fails, the orchestrator should restart the process.

### Deep Health Check

A deep health check verifies that the instance can actually serve requests by checking connectivity to all critical dependencies.

```
GET /health/ready
200 OK {
  "status": "ready",
  "checks": {
    "database": {"status": "up", "latency_ms": 3},
    "cache": {"status": "up", "latency_ms": 1},
    "queue": {"status": "up", "latency_ms": 5},
    "disk": {"status": "up", "free_gb": 42}
  }
}

503 Service Unavailable {
  "status": "not_ready",
  "checks": {
    "database": {"status": "down", "error": "connection refused"},
    "cache": {"status": "up", "latency_ms": 1},
    "queue": {"status": "up", "latency_ms": 5},
    "disk": {"status": "up", "free_gb": 42}
  }
}
```

**Use for:** Readiness probes (can this instance serve traffic?). If this fails, the load balancer should stop routing traffic to this instance -- but should not restart it (the problem might be a downstream dependency, not this process).

### Health Check Design Rules

| Rule | Rationale |
|------|-----------|
| **Shallow checks should be fast (<100ms)** | Frequent liveness checks should not consume significant resources |
| **Deep checks should have their own timeout** | A hanging dependency check should not make the health endpoint hang |
| **Do not cache health check results** | Health checks must reflect current state, not cached state |
| **Separate liveness from readiness** | A process can be alive but not ready (warming up, dependency down) |
| **Include version information** | Helps verify deployment status: `"version": "2.3.1", "commit": "abc123"` |
| **Rate-limit deep checks** | Running deep checks every second can stress dependencies |

---

## The RED Method

The RED method is a monitoring framework for request-driven services (APIs, web applications).

### RED Metrics

| Metric | What It Measures | Why It Matters |
|--------|-----------------|---------------|
| **Rate** | Requests per second | Is traffic normal? Dropping? Spiking? |
| **Errors** | Error rate (errors / total requests) | Are users experiencing failures? |
| **Duration** | Latency distribution (p50, p95, p99) | Are users experiencing slowness? |

### Implementation

For every service endpoint, instrument:

```
# Rate
http_requests_total{service, method, endpoint, status}

# Errors
http_errors_total{service, method, endpoint, error_type}
# Error rate = http_errors_total / http_requests_total

# Duration
http_request_duration_seconds{service, method, endpoint}
# Report as histogram with p50, p95, p99 percentiles
```

### RED Dashboard

A RED dashboard for each service should answer three questions at a glance:
1. **Is traffic arriving?** (Rate graph -- sudden drops indicate upstream problems or DNS issues)
2. **Are requests succeeding?** (Error rate graph -- spikes indicate bugs or dependency failures)
3. **Are requests fast?** (Duration graph -- p99 latency increasing indicates saturation)

---

## The USE Method

The USE method is a monitoring framework for infrastructure resources (CPU, memory, disk, network).

### USE Metrics

| Metric | What It Measures | Why It Matters |
|--------|-----------------|---------------|
| **Utilization** | Percentage of resource currently in use | High utilization means approaching capacity |
| **Saturation** | Amount of work waiting (queue depth) | Saturation means demand exceeds capacity |
| **Errors** | Count of error events for this resource | Hardware errors, packet drops, OOM kills |

### USE by Resource

| Resource | Utilization | Saturation | Errors |
|----------|------------|------------|--------|
| **CPU** | % time busy | Run queue length | Machine check exceptions |
| **Memory** | % used | Swap usage, OOM events | ECC errors, OOM kills |
| **Disk** | % capacity used, IOPS utilization | I/O queue depth | Read/write errors |
| **Network** | Bandwidth utilization | TCP retransmit queue, dropped packets | Interface errors, CRC errors |
| **Thread pool** | Active threads / max threads | Queued tasks | Rejected tasks |
| **Connection pool** | Active connections / max connections | Wait count | Timeout errors |

---

## SLIs, SLOs, and SLAs

### Definitions

| Term | Definition | Example |
|------|-----------|---------|
| **SLI** (Service Level Indicator) | A quantitative measure of a specific aspect of service quality | 99.2% of requests complete in < 200ms |
| **SLO** (Service Level Objective) | A target value for an SLI | 99.5% of requests should complete in < 200ms |
| **SLA** (Service Level Agreement) | A contractual commitment with consequences for violation | 99.9% availability; credit issued if breached |

### Choosing SLIs

Good SLIs measure what users actually experience:

| SLI Type | What It Measures | Measurement Point |
|----------|-----------------|-------------------|
| **Availability** | Proportion of successful requests | Load balancer or edge proxy |
| **Latency** | Proportion of requests faster than threshold | Application instrumentation |
| **Correctness** | Proportion of requests returning correct results | End-to-end tests or data validation |
| **Freshness** | Proportion of data updated within threshold | Data pipeline monitoring |

### Error Budget

The error budget is the allowed amount of unreliability: `error_budget = 1 - SLO`.

For a 99.9% availability SLO:
- Error budget = 0.1% = 43.8 minutes/month of downtime
- If you have consumed 30 minutes of budget, you have 13.8 minutes remaining
- If the budget is exhausted, freeze deployments and focus on reliability

**Error budget policy:**
- Budget remaining > 50%: deploy freely, run experiments
- Budget remaining 20-50%: deploy with caution, increase monitoring
- Budget remaining < 20%: freeze non-critical deploys, prioritize reliability work
- Budget exhausted: halt all feature deploys until budget replenishes

---

## Alerting Strategy

### Alert on Symptoms, Not Causes

| Cause-Based Alert (Avoid) | Symptom-Based Alert (Prefer) |
|--------------------------|------------------------------|
| CPU > 80% | p99 latency > 500ms |
| Memory > 90% | Error rate > 1% |
| Disk > 85% | Availability < 99.9% |
| Queue depth > 1000 | User-facing errors increasing |

Cause-based alerts generate noise. CPU can be at 90% and users are fine. CPU can be at 50% and users are seeing errors because of a deadlock.

### Alert Severity Levels

| Level | Criteria | Response | Example |
|-------|----------|----------|---------|
| **Critical** | Users actively impacted; error budget burning fast | Page on-call immediately | Error rate > 5% for 5 minutes |
| **Warning** | Approaching threshold; action needed soon | Notify team during business hours | Error budget burn rate 2x normal |
| **Info** | Notable but not actionable | Log and dashboard only | Deployment completed; circuit breaker tripped and recovered |

### Alerting Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **Alert on every metric** | Alert fatigue; team ignores pages | Only alert on user-facing symptoms |
| **No alert grouping** | 50 alerts fire for one incident | Group related alerts; alert on the root symptom |
| **No runbook** | On-call does not know what to do | Every alert links to a runbook with diagnostic steps |
| **Stale alerts** | Alerts for services that no longer exist | Review and prune alerts quarterly |
| **Missing alerts** | Critical failures go unnoticed | Regularly audit: "If X fails, would we know?" |

---

## Dashboards That Matter

A dashboard should answer "Is the system healthy right now?" within 5 seconds of looking at it.

### Dashboard Hierarchy

| Level | Audience | Content |
|-------|----------|---------|
| **Executive** | Leadership | SLO status (green/red), error budget remaining, incident count |
| **Service overview** | On-call engineer | RED metrics per service, dependency status |
| **Service deep-dive** | Service owner | Detailed metrics, resource utilization, deployment markers |
| **Debug** | Investigating engineer | Traces, log queries, specific metric breakdowns |

### Dashboard Design Rules

- **USE traffic lights:** Green (healthy), yellow (degraded), red (critical) at the top of every dashboard
- **Show trends, not just current values:** A metric at 70% is fine if it has been at 70% for a week; it is alarming if it was at 30% an hour ago
- **Mark deployments:** Overlay deployment events on metric graphs to correlate changes with behavior
- **Time range matters:** Default to 6 hours for operational dashboards; allow quick switching to 1h, 24h, 7d
- **Less is more:** A dashboard with 50 graphs is useless; a dashboard with 4 graphs that answer the right questions is invaluable

# Stability Patterns

Stability patterns are the countermeasures to the anti-patterns that cause production failures. Each pattern addresses specific failure modes and, when combined, creates a defense-in-depth strategy that allows systems to absorb shocks, degrade gracefully, and recover automatically.

These patterns are not theoretical -- they are battle-tested responses to the recurring failure modes described in the anti-patterns reference.


## Table of Contents
1. [1. Circuit Breaker](#1-circuit-breaker)
2. [2. Bulkheads](#2-bulkheads)
3. [3. Timeouts](#3-timeouts)
4. [4. Retry with Backoff](#4-retry-with-backoff)
5. [5. Steady State](#5-steady-state)
6. [6. Fail Fast](#6-fail-fast)
7. [7. Let It Crash](#7-let-it-crash)
8. [8. Handshaking](#8-handshaking)
9. [Pattern Combinations](#pattern-combinations)

---

## 1. Circuit Breaker

The Circuit Breaker is the single most important stability pattern. It prevents a failing downstream dependency from taking down the calling service by short-circuiting requests when failures exceed a threshold.

### State Machine

```
     success                  failure count
    ┌───────┐              exceeds threshold
    │       │                    │
    ▼       │                    ▼
 ┌────────┐ │              ┌────────┐
 │ CLOSED │─┘              │  OPEN  │
 └────────┘                └────────┘
      ▲                         │
      │    success               │  timeout expires
      │                         ▼
      │                    ┌──────────┐
      └────────────────────│HALF-OPEN │
           (trial request  └──────────┘
            succeeds)           │
                                │  trial request fails
                                └──────────────────┐
                                                   │
                                              ┌────────┐
                                              │  OPEN  │
                                              └────────┘
```

### States Explained

| State | Behavior | Transitions |
|-------|----------|-------------|
| **Closed** | Requests pass through normally; failures are counted | Transitions to Open when failure count exceeds threshold within time window |
| **Open** | All requests fail immediately without calling downstream | Transitions to Half-Open after a recovery timeout expires |
| **Half-Open** | A limited number of trial requests are allowed through | Transitions to Closed if trials succeed; back to Open if trials fail |

### Configuration Parameters

| Parameter | Description | Typical Range |
|-----------|-------------|---------------|
| **Failure threshold** | Number of failures before opening | 5-20 failures |
| **Time window** | Period over which failures are counted | 30-120 seconds |
| **Recovery timeout** | Time to wait in Open state before trying Half-Open | 15-60 seconds |
| **Trial requests** | Number of requests allowed in Half-Open | 1-5 requests |
| **Success threshold** | Consecutive successes needed to close | 3-5 successes |

### What Counts as a Failure

Not every error should trip the circuit breaker. Configure what counts:

| Should Trip | Should Not Trip |
|------------|----------------|
| Connection timeout | 400 Bad Request (client error) |
| Read timeout | 404 Not Found |
| 5xx server error | 429 Too Many Requests (handle with retry/backoff) |
| Connection refused | Business logic validation errors |
| Circuit breaker open on downstream | Request cancellation by client |

### Implementation Considerations

- **Granularity:** One circuit breaker per downstream service, or per endpoint within a service? Per-endpoint gives finer control but more complexity.
- **Fallback behavior:** When the circuit is open, what do you return? Cached data? Default value? Error response? The right answer depends on the use case.
- **Monitoring:** Every circuit breaker state change should emit a metric and a log entry. An open circuit breaker is a critical signal.
- **Coordination:** In a fleet of instances, each has its own circuit breaker state. Consider whether this is acceptable or if you need coordinated state (usually independent is fine).
- **Half-open thundering herd:** When the recovery timeout expires, only let one or two trial requests through -- not the full backlog.

### Code Pattern (Pseudocode)

```
class CircuitBreaker:
    state = CLOSED
    failure_count = 0
    last_failure_time = null

    function call(operation):
        if state == OPEN:
            if now() - last_failure_time > recovery_timeout:
                state = HALF_OPEN
            else:
                raise CircuitOpenException()

        try:
            result = operation()
            on_success()
            return result
        catch Exception:
            on_failure()
            raise

    function on_success():
        if state == HALF_OPEN:
            state = CLOSED
        failure_count = 0

    function on_failure():
        failure_count += 1
        last_failure_time = now()
        if failure_count >= threshold:
            state = OPEN
```

---

## 2. Bulkheads

Named after the watertight compartments in a ship's hull, bulkheads partition system resources so that a failure in one partition does not sink the entire system.

### Types of Bulkheads

| Type | Mechanism | Use Case |
|------|-----------|----------|
| **Thread pool isolation** | Separate thread pools per dependency | Service A gets 20 threads, Service B gets 20 threads; A's failure cannot exhaust B's pool |
| **Connection pool isolation** | Separate connection pools per downstream | Payment DB pool separate from analytics DB pool |
| **Process isolation** | Separate OS processes per workload | Background jobs in separate processes from request handling |
| **Container isolation** | Separate containers per service | Each microservice in its own container with resource limits |
| **Swim lanes** | Complete stack isolation for critical paths | Checkout flow runs on entirely separate infrastructure from browsing |

### Swim Lanes

Swim lanes are the most rigorous form of bulkheading. A swim lane is a complete, isolated stack -- from load balancer to database -- dedicated to a specific function.

**When to use swim lanes:**
- Revenue-critical paths (checkout, payment processing)
- Compliance-critical paths (authentication, audit logging)
- When a non-critical feature has historically caused outages affecting critical features

**Design rules for swim lanes:**
- No synchronous calls across swim lane boundaries
- No shared databases, caches, or message queues
- Asynchronous replication of data between lanes if needed
- Each lane has independent scaling, deployment, and monitoring

### Sizing Bulkheads

The key question: how many resources does each partition get?

- **Too generous:** Wasted resources; the partition rarely uses its full allocation
- **Too tight:** The partition cannot handle legitimate load spikes
- **Right-sized:** Based on measured throughput at p99 load, plus 20-30% headroom

**Approach:** Measure the actual concurrency for each dependency under peak load. Set the bulkhead size to p99 concurrency + 20% headroom. Set a queue/reject policy for requests beyond the limit.

---

## 3. Timeouts

Every outbound call needs a timeout. Every. Single. One. A missing timeout is a thread leak waiting to happen.

### Types of Timeouts

| Type | What It Controls | Typical Range |
|------|-----------------|---------------|
| **Connect timeout** | Time to establish a TCP connection | 500ms - 2s |
| **Read timeout** | Time to receive a response after connecting | 1s - 30s (depends on operation) |
| **Write timeout** | Time to send the request body | 1s - 10s |
| **Idle timeout** | Time a connection can sit unused in the pool | 30s - 5 min |
| **Request timeout** | Overall deadline for the entire operation | Varies by use case |

### Timeout Propagation

When Service A calls Service B, which calls Service C, timeouts must propagate down the chain. If the user's request has a 5-second deadline, Service A should not start a 10-second operation.

```
User request: 10s deadline
  → Service A: 8s deadline (2s for own processing)
    → Service B: 5s deadline (3s for A's processing)
      → Service C: 3s deadline (2s for B's processing)
```

**Implementation:** Pass the remaining deadline in a request header (e.g., `X-Request-Deadline` or gRPC deadline propagation). Each service subtracts its own processing time and passes the remainder downstream.

### Common Timeout Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Using default OS timeout (120-300s) | Threads blocked for minutes | Set explicit timeouts on every call |
| Same timeout for all operations | Fast reads wait too long; slow writes fail prematurely | Tune timeouts per operation type |
| No timeout on DNS resolution | DNS failure blocks thread for OS default (30s+) | Use async DNS or set DNS timeout |
| Not timing out connection pool acquisition | Thread waits forever for a connection from the pool | Set pool checkout timeout (500ms-2s) |
| Timeout too aggressive | Legitimate slow responses treated as failures | Set timeout based on p99 latency + margin |

### Setting Timeout Values

1. Measure the p99 latency of the operation under normal load
2. Add a margin (typically 2-3x p99)
3. Consider the user-facing deadline -- if the user will wait only 5 seconds, your timeout must be less than 5 seconds
4. Monitor timeout rates -- if timeouts fire frequently, the downstream is degrading

---

## 4. Retry with Backoff

Retries are necessary because transient failures are real -- network blips, momentary overloads, and garbage collection pauses cause temporary unavailability that resolves on its own. But naive retries (immediate, unlimited) cause more harm than good.

### Retry Strategy Components

| Component | Purpose | Implementation |
|-----------|---------|----------------|
| **Exponential backoff** | Increase delay between retries to give the downstream time to recover | Delay = base * 2^attempt (e.g., 100ms, 200ms, 400ms, 800ms) |
| **Jitter** | Randomize the delay to prevent all clients from retrying at the same instant | Delay = random(0, base * 2^attempt) |
| **Maximum retries** | Limit total attempts to prevent infinite retry loops | Typically 3-5 retries |
| **Retry budget** | Limit the percentage of requests that are retries across the entire fleet | No more than 10-20% of total requests should be retries |
| **Idempotency** | Ensure the operation is safe to retry without side effects | Use idempotency keys for mutations |

### When to Retry

| Retry | Do Not Retry |
|-------|-------------|
| 503 Service Unavailable | 400 Bad Request |
| 429 Too Many Requests (with backoff) | 401/403 Authentication/Authorization |
| Connection timeout | 404 Not Found |
| Connection reset | 422 Validation Error |
| Read timeout (for idempotent operations) | Read timeout (for non-idempotent operations without idempotency key) |

### Retry Budget

A retry budget limits the total percentage of retried requests across your entire fleet. Without a budget, a fleet of 100 instances each retrying 3 times can amplify load on a struggling downstream by 300x.

**Implementation:**
- Track the ratio of retries to original requests over a sliding window (e.g., 1 minute)
- If retry ratio exceeds the budget (e.g., 20%), stop retrying and fail fast
- This provides fleet-level protection that per-instance retry limits cannot

### The Thundering Herd Problem with Retries

When a downstream service recovers, all clients retry simultaneously, overwhelming it again. Jitter solves this:

- **No jitter:** All 1,000 clients retry at exactly t+100ms, t+200ms, t+400ms
- **Full jitter:** Each client retries at random(0, 100ms), random(0, 200ms), random(0, 400ms) -- spreading the load evenly

---

## 5. Steady State

Production systems accumulate cruft over time. Log files grow. Sessions pile up. Temporary files linger. Database tables bloat. Without active management, this accumulation eventually exhausts a resource -- disk space, memory, database capacity -- and the system fails.

### What Accumulates

| Resource | Growth Mechanism | Consequence of Neglect |
|----------|-----------------|----------------------|
| **Log files** | Application logging, access logs | Disk full; application cannot write; crash |
| **Database rows** | Event logs, audit trails, temp records | Slow queries; backup failures; storage costs |
| **Sessions** | User sessions stored server-side | Memory exhaustion; swap thrashing |
| **Temp files** | Upload processing, report generation | Disk full |
| **Cache entries** | Application cache without TTL | Memory exhaustion |
| **Message queues** | Unprocessed or dead-letter messages | Queue backpressure; memory/disk exhaustion |

### Steady State Design

Every resource that grows must have a corresponding mechanism to shrink:

- **Log rotation:** Rotate by size (500MB) and time (daily); retain N files; compress rotated files
- **Data purging:** Delete or archive records older than retention period; run during off-peak hours
- **Session cleanup:** Set maximum session duration and idle timeout; evict expired sessions automatically
- **Cache eviction:** Use TTL-based expiration; implement LRU or LFU eviction when memory limit is reached
- **Queue management:** Set maximum queue depth; dead-letter after N delivery attempts; alert on growing queue size

---

## 6. Fail Fast

If a system knows it cannot process a request successfully, it should reject the request immediately rather than consuming resources in a doomed attempt.

### When to Fail Fast

| Condition | Fail-Fast Response |
|-----------|-------------------|
| Circuit breaker is open | Return 503 immediately |
| Required parameter missing | Return 400 immediately |
| User not authenticated | Return 401 immediately |
| Resource limit exceeded (rate limit) | Return 429 immediately |
| Request deadline already expired | Return 504 immediately |
| Required downstream service unavailable | Return 503 with degraded response |

### Why Fail Fast Is Kind

Failing fast is not hostile -- it is kind. A fast rejection lets the caller know immediately that this request will not work, freeing the caller to retry elsewhere, display an error, or degrade gracefully. A slow failure wastes the caller's time, threads, and hope.

---

## 7. Let It Crash

Borrowed from Erlang's "let it crash" philosophy: when a process enters an unrecoverable or uncertain state, it is safer to terminate and restart from a known-good state than to attempt in-process recovery.

### When to Let It Crash

- Corrupted in-memory state that cannot be validated
- Resource handles (file descriptors, connections) in an unknown state
- Unhandled exception in a critical path where recovery logic is complex and error-prone
- Process consuming excessive memory or CPU with no clear cause

### Prerequisites for Let It Crash

- **Fast restart:** Process must start in seconds, not minutes
- **Supervision:** A supervisor must detect the crash and restart the process
- **State recovery:** Critical state must be externalized (database, distributed cache) -- not lost on crash
- **Health checks:** Load balancer must detect the restart and route traffic away during startup
- **Crash budget:** Monitor crash frequency; excessive crashes indicate a deeper problem, not a healthy recovery mechanism

---

## 8. Handshaking

Handshaking is a protocol-level mechanism where the server tells the client whether it is ready to accept work before the client sends a full request. This prevents the server from being overwhelmed when it is already overloaded.

### Handshaking Mechanisms

| Level | Mechanism | Example |
|-------|-----------|---------|
| **TCP** | SYN/ACK with backlog queue | OS rejects connections when listen queue is full |
| **HTTP** | 100 Continue response | Server tells client to proceed with request body, or rejects early |
| **Application** | Health check + load shedding | Load balancer checks `/ready` endpoint; server returns 503 when overloaded |
| **gRPC** | Flow control with window updates | Server signals how much data it can receive |
| **Custom** | Pre-flight capacity check | Client asks "can you handle this request?" before sending payload |

### Application-Level Handshaking

Implement a readiness check that reflects actual capacity:

```
GET /ready

200 OK          → Server can accept work
503 Unavailable → Server is overloaded; do not send requests

Readiness considers:
- Thread pool utilization < 80%
- Connection pool utilization < 80%
- Response latency < SLO threshold
- No critical dependency failures
```

This allows load balancers and service meshes to route traffic away from overloaded instances before they fail, rather than after.

## Pattern Combinations

These patterns are most effective in combination. Common pairings:

| Combination | Effect |
|-------------|--------|
| **Timeout + Circuit Breaker** | Timeout detects slow failures; circuit breaker prevents repeated attempts |
| **Circuit Breaker + Fallback** | Circuit breaker trips; fallback provides degraded response |
| **Bulkhead + Circuit Breaker** | Bulkhead limits blast radius; circuit breaker stops the bleeding |
| **Retry + Timeout + Circuit Breaker** | Retry handles transient failures; timeout limits wait; circuit breaker stops persistent failures |
| **Fail Fast + Handshaking** | Server signals overload; client fails fast without sending work |
| **Steady State + Let It Crash** | Cleanup prevents resource exhaustion; crash recovers from unknown states |

The key principle: no single pattern is sufficient. Production-ready systems layer multiple patterns to create defense in depth.