# Metrics Interpretation

After all stages complete, interpret the collected metrics as follows.

## Latency

For each endpoint, compute:

- **Latency by percentile**: p50, p75, p90, p95, p99 at each concurrency level.
- **Latency trend**: How median latency changes as concurrency increases. Compute the slope.
- **Latency stability**: Standard deviation at each stage. Flag stages where stddev > 2x the median.
- **Latency threshold violations**: At which concurrency level each percentile exceeded the target.

Classify the latency profile:

- **Flat**: Latency stays within 20% of baseline up to max concurrency. Excellent.
- **Linear degradation**: Latency increases proportionally with concurrency. Acceptable up to a point.
- **Exponential degradation**: Latency increases faster than concurrency. Bottleneck detected.
- **Cliff**: Latency suddenly spikes at a specific concurrency level. Hard limit found.

## Throughput

For each endpoint, compute:

- **Peak throughput**: Maximum requests/second achieved and at which concurrency level.
- **Throughput ceiling**: The concurrency level where adding more users no longer increases throughput (saturation point).
- **Throughput curve shape**: Linear growth, logarithmic growth, or plateau.
- **Efficiency ratio**: Throughput per concurrent user at each stage.

## Errors

For each endpoint, compute:

- **Error rate by stage**: Percentage of non-success responses at each concurrency level.
- **Error onset**: The concurrency level where errors first appear above 0.1%.
- **Error types**: Categorize into timeout, connection refused, 4xx, 5xx, and other.
- **Error rate trend**: Stable, growing linearly, or growing exponentially.

## Breaking Point

The breaking point is the concurrency level where ANY of the following first occurs. State it clearly and name which condition triggered it.

1. Error rate exceeds 1%.
2. p95 latency exceeds 5x the baseline single-user p95.
3. Throughput decreases compared to the previous stage (throughput cliff).
4. More than 5% of connections are refused or reset.

## Bottleneck Classification

Classify the likely bottleneck and provide supporting evidence from the data.

- **CPU-bound**: Latency increases linearly, throughput plateaus, no connection errors.
- **Memory-bound**: Latency is stable then suddenly spikes, often with connection resets.
- **I/O-bound (database)**: Latency variance is high, throughput has a hard ceiling, errors are timeouts.
- **I/O-bound (network)**: Connection refused errors, high timeout rate, latency spikes correlate with error spikes.
- **Connection pool exhaustion**: Sudden onset of connection errors at a specific concurrency level, latency cliff.
- **Rate limiting**: Consistent 429 status codes above a threshold, latency stable but errors spike.
- **Thread/process pool exhaustion**: Throughput plateaus, latency grows linearly, no errors until a hard cliff.

# Output Template

Create the file `api-load-report.md` in the current working directory using this exact structure.

```markdown
# API Load Test Report

**Date**: <YYYY-MM-DD HH:MM:SS timezone>
**Environment**: <prod/staging/dev or as specified>
**Tool**: <hey/wrk/ab/curl>
**Test Duration**: <total wall-clock time>

---

## Executive Summary

<2-3 sentences summarizing the overall findings. State the key throughput number, the breaking point, and the most critical recommendation.>

---

## Endpoints Tested

| # | Method | URL | Auth | Payload |
|---|--------|-----|------|---------|
| 1 | GET | https://... | Bearer | N/A |
| 2 | POST | https://... | Bearer | JSON (245 bytes) |

---

## Test Configuration

- **Concurrency stages**: <list of concurrency levels>
- **Duration per stage**: <seconds>
- **Total requests per stage**: <number>
- **Request timeout**: <seconds>
- **Success criteria**: <status codes>
- **Ramp pattern**: <step/linear/spike>

---

## Results by Endpoint

### Endpoint 1: <METHOD> <URL>

#### Latency Percentiles (ms)

| Concurrency | p50 | p75 | p90 | p95 | p99 | Max |
|-------------|-----|-----|-----|-----|-----|-----|
| 1 | ... | ... | ... | ... | ... | ... |
| 5 | ... | ... | ... | ... | ... | ... |
| ... | ... | ... | ... | ... | ... | ... |

#### Throughput

| Concurrency | Req/sec | Transfer (KB/s) | Avg Latency (ms) | Error Rate (%) |
|-------------|---------|------------------|-------------------|----------------|
| 1 | ... | ... | ... | ... |
| 5 | ... | ... | ... | ... |
| ... | ... | ... | ... | ... |

#### Error Breakdown

| Concurrency | 2xx | 4xx | 5xx | Timeout | Conn Error | Total Errors |
|-------------|-----|-----|-----|---------|------------|-------------|
| 1 | ... | ... | ... | ... | ... | ... |
| ... | ... | ... | ... | ... | ... | ... |

#### Latency Profile

<Classify as Flat / Linear / Exponential / Cliff with supporting data>

#### Breaking Point

<State the breaking point concurrency, which condition triggered it, and the specific metric values>

---

<Repeat for each endpoint>

---

## Comparative Analysis

<If multiple endpoints were tested, compare their performance profiles. Identify which endpoints are the weakest links.>

| Endpoint | Peak RPS | Breaking Point | Bottleneck Type | p95 at Peak |
|----------|----------|---------------|-----------------|-------------|
| GET /health | ... | ... | ... | ... |
| POST /search | ... | ... | ... | ... |

---

## Throughput Curves (ASCII)

<For each endpoint, render an ASCII chart showing throughput vs concurrency>

```
Throughput (req/s)
    ^
800 |          *----*----*
    |        *
600 |      *
    |    *
400 |   *
    |  *
200 | *
    |*
  0 +--+--+--+--+--+--+--> Concurrency
    1  5  10 25 50 100 200
```

---

## Latency Distribution (ASCII)

<For each endpoint, render an ASCII chart showing p50/p95/p99 vs concurrency>

```
Latency (ms)
     ^
1000 |                        * p99
     |                  *
 500 |            *           o p95
     |      o           o
 200 |o  o        .  .  .  . . p50
 100 |.  .  .
   0 +--+--+--+--+--+--+--+--> Concurrency
     1  5  10 25 50 100 200 500
```

---

## Bottleneck Analysis

### Primary Bottleneck

<Classification (CPU/Memory/IO/Connection Pool/Rate Limit/Thread Pool) with 3-5 bullet points of supporting evidence from the test data>

### Secondary Observations

<Any additional patterns observed, such as:>
- Garbage collection pauses (periodic latency spikes)
- DNS resolution overhead
- TLS handshake cost at high concurrency
- Keep-alive vs connection-per-request behavior
- Response body size variation under load

---

## Recommendations

### Critical (Address Immediately)

1. **<Recommendation title>**: <Detailed explanation with specific numbers from the test. E.g., "Add connection pooling -- connection errors begin at 50 concurrent users, suggesting the server is opening a new database connection per request. A pool of 20-30 connections should handle up to 200 concurrent users based on the observed throughput ceiling.">

2. **<Recommendation title>**: <...>

### Important (Address Before Scaling)

3. **<Recommendation title>**: <...>

4. **<Recommendation title>**: <...>

### Nice to Have (Optimization)

5. **<Recommendation title>**: <...>

6. **<Recommendation title>**: <...>

---

## Capacity Estimate

Based on the observed performance profile:

- **Current safe operating capacity**: <X concurrent users> (<Y req/sec>)
- **Maximum tested capacity**: <X concurrent users> (<Y req/sec, Z% error rate>)
- **Estimated capacity with recommended fixes**: <X concurrent users> (projected)

### Scaling Projections

| Target Users | Current Status | After Fixes | Additional Infra Needed |
|-------------|---------------|-------------|------------------------|
| 50 | OK | OK | None |
| 100 | Degraded (p95 > target) | OK (projected) | None |
| 500 | Breaking point | OK (projected) | Add replica |
| 1000 | Not viable | Marginal | Load balancer + 3 replicas |

---

## Methodology Notes

- Tool: <name and version>
- Each concurrency stage ran for <N> seconds with a <N>-second cooldown between stages
- Latency measurements include full round-trip time (DNS + connect + TLS + TTFB + transfer)
- All tests were run from <location/machine description>
- Results may vary based on network conditions, server load, and time of day
- For production capacity planning, repeat tests at different times and from multiple geographic locations

---

## Raw Data Reference

Raw output files are stored in: `<temp_directory_path>`

<List the files with brief descriptions>
```

# Rules, Error Handling, and Examples

## Safety and Conduct Rules

1. **Never test production endpoints without explicit user confirmation.** If the environment is "prod" or the URL contains "prod", "production", or appears to be a production domain, warn the user and ask for confirmation before proceeding.
2. **Respect rate limits.** If 429 responses are detected, reduce concurrency and note the rate limit. Do not continue hammering an endpoint that is returning 429s.
3. **Handle authentication carefully.** Never log or include full auth tokens. Mask them (e.g., "Bearer eyJ...****").
4. **No destructive testing by default.** Only test GET endpoints by default. For POST/PUT/DELETE, confirm the endpoint is safe to call repeatedly (idempotent, uses a test database, or has no side effects).
5. **Clean up temporary files.** Store raw results in a clearly named temp directory but do not delete them automatically; the user may want to inspect them.
6. **Report in consistent units.** Use milliseconds for latency, requests/second for throughput, and percentages for error rates. Always label units.
7. **ASCII charts are mandatory in the output.** Even though approximate, they give immediate visual understanding without external tools.
8. **Test from the same machine consistently.** Do not distribute load across machines unless the user specifically asks for distributed testing.
9. **Timeouts count as failures.** A timed-out request is a failed request, not excluded from the data.
10. **Do not extrapolate beyond tested ranges.** The scaling projections table must clearly mark projected values vs observed values.

## Error Handling

- If a tool installation fails, fall back to the next tool in the priority list. If all preferred tools fail, use the curl fallback.
- If an endpoint becomes completely unresponsive during testing (100% timeout for 30+ seconds), stop testing that endpoint at that concurrency level and move to the next stage or endpoint. Note this in the output as "endpoint became unresponsive."
- If the machine runs out of file descriptors or hits OS-level connection limits, detect the error message, surface it, and suggest increasing `ulimit -n` before retrying.
- If the run is interrupted (Ctrl+C or timeout), save whatever data has been collected so far and generate a partial output clearly marked as incomplete.

## Example Invocations

**Simple single endpoint**:
```
Load test https://api.example.com/health
Expected response time: p95 < 200ms
Concurrent users: up to 100
```

**Multiple endpoints with auth**:
```
Endpoints:
  - GET https://api.example.com/users (Bearer token: abc123)
  - POST https://api.example.com/search (Bearer token: abc123, body: {"query": "test"})
Expected: p95 < 300ms
Concurrency: 10 to 500
Environment: staging
```

**Quick smoke test**:
```
Quick load test https://api.example.com/health with 50 concurrent users
```

For quick/smoke tests, reduce to 3 stages: baseline (1), target concurrency (50), and 2x target (100). Shorten duration to 5 seconds per stage.

# Tool Commands

Run each concurrency stage with the best available tool. Wait 2 seconds between stages so the server stabilizes and carryover effects are avoided.

## hey (preferred)

```bash
hey -n <total_requests> -c <concurrency> -t <timeout> \
    -m <METHOD> \
    -H "Authorization: Bearer <token>" \
    -H "Content-Type: application/json" \
    -d '<body>' \
    <url>
```

Calculate total requests as `concurrency * (duration / estimated_response_time)`, with a minimum of `concurrency * 10` requests per stage.

## wrk

```bash
wrk -t <threads> -c <concurrency> -d <duration>s \
    -s <lua_script> \
    <url>
```

Generate a Lua script when custom methods, headers, or bodies are needed.

## ab (Apache Bench)

```bash
ab -n <total_requests> -c <concurrency> -t <timeout> \
    -H "Authorization: Bearer <token>" \
    -T "application/json" \
    -p <body_file> \
    <url>
```

## curl fallback

```bash
for i in $(seq 1 $CONCURRENCY); do
  (for j in $(seq 1 $REQUESTS_PER_USER); do
    curl -o /dev/null -s -w "%{http_code} %{time_total}\n" \
      -X <METHOD> \
      -H "Authorization: Bearer <token>" \
      -H "Content-Type: application/json" \
      -d '<body>' \
      <url>
  done) &
done
wait
```

## Default concurrency stages

Adjust to the user-specified concurrency range. If the user specifies a max of 50, stop there. If they specify a max of 1000, add stages beyond 500.

| Stage | Concurrent Users | Duration | Purpose |
|-------|-----------------|----------|---------|
| 1 | 1 | 10s | Baseline single-user latency |
| 2 | 5 | 10s | Light load behavior |
| 3 | 10 | 10s | Moderate load |
| 4 | 25 | 10s | Medium load |
| 5 | 50 | 10s | Heavy load |
| 6 | 100 | 10s | Stress test |
| 7 | 200 | 10s | Breaking point search |
| 8 | 500 | 10s | Extreme stress (optional) |

## Per-stage data to capture

- Total requests sent
- Successful responses (by status code)
- Failed responses (by status code or error type)
- Latency: min, max, mean, median (p50), p90, p95, p99
- Requests per second (throughput)
- Transfer rate (bytes/sec if available)
- Connection errors, timeouts, and resets
- Stage start and end timestamps

Store raw results in a temp directory like `/tmp/api-load-test-<timestamp>/`, one file per stage:
`raw_<endpoint_name>_c<concurrency>.txt`.