Claude Skill

sqs-queue-auditor

Audit a single AWS SQS queue's configuration for the misconfigurations that silently drop or re-deliver messages while every attribute reads as fine. Parses the GetQueueAttributes output (and the referenced dead-letter queue), checks the redrive path (DLQ present, maxReceiveCount

LLM Mart · 0 points · 5 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download anyshift-io-sre-skills-skills_sqs-queue-auditor-a7af922.zip · 117 KB

anyshift-io/sre-skills 17 0 forks Apache-2.0 Updated 28d ago

Part of anyshift-io/sre-skills — 5 skills

Install

skills CLI npx skills add https://github.com/anyshift-io/sre-skills/tree/main/skills/sqs-queue-auditor

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install anyshift-io-sre-skills@llmmart

Git git clone https://github.com/anyshift-io/sre-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole anyshift-io/sre-skills collection as a plugin from our marketplace. Git is the plain clone.

README

sqs-queue-auditor

Configuration-audit skill for a single AWS SQS queue.

Parses the GetQueueAttributes output for one queue (and its referenced dead-letter queue), applies the judgment a senior engineer applies to that one source, and reports the misconfigurations that silently drop or re-deliver messages while every attribute reads as fine. Then it names the boundary: the questions a single queue's config cannot answer.

Files in this skill

File	What it is
`SKILL.md`	The methodology. This is what an AI agent loads.
`examples/`	Eight worked examples, one per rule plus a clean control.
`fixtures/`	Committed `GetQueueAttributes` snapshots that drive the replay tests. No external credentials required.
`tests/`	Replay tests that exercise the audit against the fixtures.
`FAILURE_MODES.md`	Where this skill is wrong and where the agent should escalate.

What it checks

Code	Rule	Severity
R1	No dead-letter queue on a processing queue	high
R2	`maxReceiveCount` outside the 3-10 band	medium / low
R3	DLQ retention not longer than source retention	critical
R4	Poison messages age out before reaching the DLQ	critical
R5	Visibility timeout at the 30s default	low
R6	Retention shorter than a plausible outage	medium
R7	Resource policy allows a wildcard principal with no condition	high
R8	Server-side encryption at rest disabled	low
R9	FIFO queue with content-based dedup off	low

The two critical rules (R3, R4) are the ones a console read almost never catches: both turn a correctly-wired, correctly-sized dead-letter queue into one that silently never receives the messages it was built for.

Quality bar (this skill passes all three)

Two worked examples required by the bar; this skill ships eight, one per rule plus a clean control that asserts no false positives.
Fixture-based replay tests, runnable with no external credentials. 48 assertions across the 8 tests (for t in tests/replay_*.py; do python "$t" || exit 1; done).
Explicit failure-modes section (FAILURE_MODES.md).

Measured lift

An LLM ablation eval is committed under tests/eval/. A reference run with Claude Sonnet 4.6 (N=3, LLM-as-judge against the 7-item rubric, on the four most diagnostic fixtures) measured +3.08 / 7 (+44%) lift of an agent loaded with this SKILL.md over an agent given the same GetQueueAttributes JSON with no methodology. Treatment beats control on every fixture and sweeps 7.00 / 7 with zero variance.

The lift concentrates where it should: no control output produced a boundary section (every cold agent presented a config read as a full health verdict), and on the clean control the cold agent flagged an aws:SourceArn-scoped wildcard policy as a HIGH "public queue", the textbook false positive the skill's R7 precision exists to avoid. See tests/eval/README.md for the per-fixture table, the per-rubric-item breakdown, the R4 over-fire regression the eval caught and the SKILL.md edit that closed it, and the caveats. Reproduce with python tests/eval/run_eval.py --trials 3.

How to use

As a Claude Code / Claude Skills user

Drop skills/sqs-queue-auditor/ into your skills directory and invoke when reviewing or hardening a queue. The agent reads SKILL.md, parses the queue attributes, and reports findings plus the boundary. Point it at a real queue with aws sqs get-queue-attributes --queue-url <url> --attribute-names All, or run it against the committed fixtures first.

As a contributor adding a new rule or example

Add a fixture directory under fixtures/<example-slug>/ with queue.json (and dlq.json if the queue has a DLQ), following the GetQueueAttributes shape in tests/README.md.
Add a worked example under examples/ mirroring the existing eight.
Add a replay test under tests/replay_NN_<slug>.py asserting the expected findings and that the boundary is reported.
Update SKILL.md and this table if the rule is new.

See the top-level CONTRIBUTING.md for the repo-wide bar.

Anyshift integration (opt-in)

The audit runs vendor-neutral by default. Every boundary note this skill emits is a join it cannot make from one queue's attributes: queue to its consumers, queue to its CloudWatch metrics over time, queue to the account's IAM graph, queue to the producers and consumers on either side. Opting in to the Anyshift MCP resolves those joins from a versioned resource graph, so a deferred flag becomes a closed finding.

A measured "with vs without" delta will be published here once the MCP integration has been exercised against the replay fixtures.

License

Apache 2.0.

Skill manifest

sqs-queue-auditor

Configuration-audit skill for a single AWS SQS queue. Takes the GetQueueAttributes output for one queue, applies the judgment a senior engineer applies to that one source (the thresholds, the known-bad combinations, the one arithmetic relationship that turns a correct-looking config into silent message loss), and returns a ranked list of findings with recommendations. Then it names exactly where a single queue's configuration stops being able to answer the question.

When to invoke

An agent is asked to review, harden, or sanity-check an SQS queue before or after it ships.
Messages are going missing or being processed twice and nobody can see why from the console.
A dead-letter queue is configured but empty during an incident, and the question is whether it is actually wired to catch what is failing.
A queue is being added to a Terraform module or a CDK stack and the config should be checked against the known-bad combinations before apply.

What this skill reads, and what it does not

It reads the static configuration of one queue, plus the attributes of the dead-letter queue that queue's own RedrivePolicy points at. Both are SQS control-plane reads (GetQueueAttributes). That is the entire input. The audit is correct and complete for what a queue's configuration can tell you, and it is explicit about the rest:

It does not read CloudWatch. Live behaviour (redrive volume, age of the oldest message, in-flight count, empty-receive rate) is a time-series, not an attribute.
It does not read the consumers. Whether the visibility timeout is actually long enough is a property of how long the consumer takes, which is not in the queue.
It does not read account IAM. The effective set of principals that can act on the queue is the union of the resource policy (visible) and every identity policy in the account (not visible here).
It does not read the producers. Whether the right services are writing to the queue, and whether anyone is draining the DLQ, needs the inventory on either side.

Every audit ends by naming these. The boundary is the same one every time: the join across resources, across sources, or across time.

The methodology, in order

1. Parse the attributes

GetQueueAttributes returns every value as a string, and the compound attributes are JSON documents encoded inside those strings. Before any judgment:

Parse RedrivePolicy (a JSON string) into deadLetterTargetArn and maxReceiveCount. A queue with no RedrivePolicy has no DLQ.
Parse MessageRetentionPeriod, VisibilityTimeout, DelaySeconds as integer seconds (they arrive as strings).
Parse Policy (a JSON string) into IAM statements, if present.
Read FifoQueue, ContentBasedDeduplication, SqsManagedSseEnabled, KmsMasterKeyId.
If a DLQ is referenced, load its attributes too. The retention-ordering check is impossible without them.

A naive read skips the embedded JSON entirely and never sees the redrive wiring. Parsing it is step zero of the judgment.

2. Audit the redrive path

The dead-letter path is where messages are supposed to go when processing fails. Three things break it:

No DLQ on a processing queue (R1). Without a RedrivePolicy, a poison message is retried until MessageRetentionPeriod expires, then deleted with no signal. There is no quarantine.
maxReceiveCount out of band (R2). Below 3, a transient downstream blip dead-letters good messages. Above 10, poison messages are retried many times before quarantine, delaying detection and feeding R4. The sane band is roughly 3 to 10.
DLQ retention not longer than the source (R3). A message's age is measured from its original SentTimestamp, and SQS does not reset that timestamp when the message moves to the DLQ. If the DLQ's retention is less than or equal to the source's, a message that fails late in the source's window arrives in the DLQ already near its age limit and is deleted almost immediately. The DLQ looks wired and sized; the messages you most need to inspect are the ones it drops. This is the single most important non-obvious check in the skill.

3. Audit the message lifecycle

Three queue-side timing relationships, all derivable from the static config:

Poison messages age out before the DLQ (R4). A poison message needs at least maxReceiveCount x VisibilityTimeout seconds of wall-clock to exhaust its receive count and dead-letter. If that product exceeds MessageRetentionPeriod, retention wins: the message is deleted by age before it ever reaches the DLQ. The DLQ is configured but unreachable for slow failures. This is pure arithmetic on three attributes and is almost never checked by hand. Fire R4 only on the configured-value inequality (maxReceiveCount x VisibilityTimeout > MessageRetentionPeriod); do not raise it speculatively because backlog or load "might" stretch the wall-clock. The product is already a lower bound, so a config that satisfies the inequality is safe by construction. Queue depth and receive cadence are behind the boundary, not inputs to this check.
Visibility timeout at the 30s default (R5). A risk flag, not a proven bug. If a consumer takes longer than 30s, the message reappears mid-processing and is delivered twice. Whether that happens depends on consumer processing time, which is not a queue attribute. Surfaced as low severity and deferred to the boundary.
Retention shorter than a plausible outage (R6). Retention below an hour means a brief consumer outage, deploy, or scaling lag silently drops every message still queued.

4. Audit exposure

Open resource policy (R7). A Policy statement that allows a wildcard principal ("*") with no narrowing Condition (aws:SourceArn, aws:SourceAccount, aws:PrincipalOrgID) authorises any AWS principal to act on the queue. This is the confused-deputy and public-queue exposure. A wildcard principal with a SourceArn condition (the standard SNS-to-SQS pattern) is fine and must not be flagged.
Encryption at rest disabled (R8). Neither SQS-managed SSE nor a KMS key configured. Low severity, because whether it matters depends on the data classification, which the queue does not carry.

5. Audit FIFO invariants

FIFO with content-based dedup off (R9). When ContentBasedDeduplication is off on a FIFO queue, every producer must supply an explicit MessageDeduplicationId or duplicate sends are accepted as distinct. Whether the producers actually do this is a property of the producers, not the queue. Flagged low and deferred to the boundary.

6. Rank and report, then name the boundary

Order findings by severity (critical, high, medium, low). For each: the rule, the attribute(s) it is grounded in, what breaks, and the fix. Then list the boundary: the joins this audit cannot make. A clean config still gets a boundary section, because a clean config is not a clean system.

Severity model

Severity	Meaning
critical	A configuration that silently loses messages. R3 and R4.
high	A configuration that loses messages on a poison input, or exposes the queue. R1, R7.
medium	A configuration that loses messages under an ordinary operational gap. R2 (too low), R6.
low	A risk flag whose confirmation needs something behind the boundary. R2 (too high), R5, R8, R9.

The low band is deliberately honest: those findings depend on consumer processing time, data classification, or producer behaviour, none of which is a queue attribute. The skill flags them for verification rather than asserting a bug it cannot prove.

Rule reference

Code	Rule	Severity	Grounded in
R1	No dead-letter queue on a processing queue	high	`RedrivePolicy` absent
R2	`maxReceiveCount` outside the 3-10 band	medium / low	`RedrivePolicy.maxReceiveCount`
R3	DLQ retention not longer than source retention	critical	source vs DLQ `MessageRetentionPeriod`
R4	Poison messages age out before reaching the DLQ	critical	`VisibilityTimeout` x `maxReceiveCount` vs `MessageRetentionPeriod`
R5	Visibility timeout at the 30s default	low	`VisibilityTimeout`
R6	Retention shorter than a plausible outage	medium	`MessageRetentionPeriod`
R7	Resource policy allows a wildcard principal with no condition	high	`Policy`
R8	Server-side encryption at rest disabled	low	`SqsManagedSseEnabled` / `KmsMasterKeyId`
R9	FIFO queue with content-based dedup off	low	`ContentBasedDeduplication`

Output format

The agent's final message in any invocation must include:

Queue: ARN, standard or FIFO, DLQ wired or not.
Findings: ranked by severity, each with the rule code, the attribute(s), what breaks, and the recommendation. Or "no findings" for a clean config.
Boundary: the joins this audit could not make, stated explicitly so the gap is visible instead of silent.

Worked examples

Eight end-to-end examples are committed under examples/, each with fixtures (real GetQueueAttributes shape) and a runnable replay test. Each isolates one rule, except where two genuinely co-occur.

examples/01-no-dlq.md: a payments queue with no DLQ; poison messages are retried until retention expiry, then dropped (R1).
examples/02-dlq-retention-shorter-than-source.md: the silent-loss bug; the DLQ retains for less time than the source, so late failures are deleted on arrival (R3).
examples/03-maxreceivecount-too-low.md: maxReceiveCount=1 dead-letters good messages on the first transient failure (R2).
examples/04-poison-ages-out-before-dlq.md: the flagship; a 15-minute visibility timeout and maxReceiveCount=1000 mean poison messages age out before reaching a correctly-wired DLQ (R4, plus R2).
examples/05-default-visibility-short-retention.md: a 30s default visibility timeout and 5-minute retention; two soft flags that defer to the boundary (R5, R6).
examples/06-public-queue-policy.md: a resource policy with Principal: "*" and no condition, on an unencrypted queue (R7, R8).
examples/07-fifo-dedup-off.md: a FIFO queue with content-based dedup off, depending on a producer contract the queue cannot verify (R9).
examples/08-clean-standard.md: the control; a correctly-configured queue produces zero findings and still reports its boundary.

Replay tests

Every example has a replay test in tests/ that runs the audit against committed fixtures, with no external credentials. Run from the skill directory:

for t in tests/replay_*.py; do python "$t" || exit 1; done

The 8 tests cover all nine rules, the severity model, and the clean-control (no false positives), totalling 48 assertions. Tests exit non-zero if the audit produces the wrong findings or drops the boundary. See tests/README.md for the fixture schema and how to add a new replay test.

Failure modes

This skill is wrong in predictable ways. Read FAILURE_MODES.md before relying on it. Highlights:

It audits configuration, not behaviour. A queue that passes every check can still be failing right now for a reason only CloudWatch shows.
The R1 "is this a processing queue" judgment is supplied by the caller; a pure buffer queue may legitimately have no DLQ.
The low-severity flags (R5, R8, R9) cannot be confirmed without the consumer, the data classification, or the producers. They are flags, not verdicts.

Anyshift integration (opt-in)

The audit above runs end-to-end against the GetQueueAttributes output the user already has. No Anyshift dependency.

Every boundary note in this skill is a join: queue to its consumers, queue to its metrics over time, queue to the account's IAM graph, queue to the producers and consumers on either side. The Anyshift MCP can act as a context primer by resolving those joins from a versioned resource graph, so a finding like R5 ("visibility timeout at default, verify against consumer processing time") or R7 ("resource policy is half the access story") can be closed instead of deferred. A measured "with vs without" delta will be published here once the integration has been exercised against the replay fixtures.

Files (sre-skills)

examples

01-no-dlq.md 2.9 KB

# Worked example 1: no dead-letter queue on a processing queue (R1)

A processing queue with no `RedrivePolicy`. Every attribute reads as healthy, the queue has messages flowing, and a poison message has nowhere to go. Fixtures and replay test under `../fixtures/01-no-dlq/` and `../tests/replay_01_no_dlq.py`.

## Scenario

- **Queue**: `payments-capture`. A worker fleet receives capture events and calls a payment processor.
- **Symptom**: one malformed capture event (a cents value the processor rejects) is received, fails, becomes visible again, and is redelivered. Forever. No alert, no DLQ, no record. After four days it is deleted by retention and nobody ever sees it.

## Step 1: parse the attributes

```
QueueArn               arn:aws:sqs:eu-west-1:211125758836:payments-capture
VisibilityTimeout      120
MessageRetentionPeriod 345600   (4 days, the default)
RedrivePolicy          (absent)
SqsManagedSseEnabled   true
```

`VisibilityTimeout` is a deliberate 120s (not the 30s default, so R5 does not fire). Retention is the 4-day default (R6 does not fire). SSE is on (R8 does not fire). The one thing missing is the `RedrivePolicy`.

## Step 2: audit the redrive path

No `RedrivePolicy` means no dead-letter queue. On a processing queue, that is **R1 (high)**. A poison message is received, fails, and after `VisibilityTimeout` becomes visible again. It is retried on every cycle for the full `MessageRetentionPeriod`, burning a consumer slot each time, and is then deleted silently. There is no quarantine and no signal. The only evidence the message ever existed is a worker that spent four days failing on it.

## Steps 3-5: lifecycle, exposure, FIFO

All clean. Visibility is deliberate, retention is the default, the queue is encrypted, there is no resource policy, and the queue is standard (not FIFO). R1 is the only finding.

## Finding

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R1 | high | `RedrivePolicy` absent | Attach a `RedrivePolicy` to a DLQ with `maxReceiveCount` in the 3-10 band, so poison messages are quarantined instead of dropped. |

## Boundary

The audit confirms there is nowhere for failed messages to go. It cannot tell you whether any message is *currently* failing: that is the redrive metric and the consumer error rate, both behind the boundary.

- Whether a poison message is in the queue right now is `ApproximateAgeOfOldestMessage` and the consumer error rate over time, not a static attribute. Join: queue to its metrics over time.
- Whether this queue is genuinely a processing queue (R1) or a buffer with an at-least-once contract elsewhere is a property of the architecture, not the queue. Join: queue to its consumers.

## Why this is the R1 reference

It is the cleanest possible R1: a single missing attribute on an otherwise healthy queue, with a failure mode (silent four-day drop of poison messages) that no console glance surfaces because every visible number looks fine.

02-dlq-retention-shorter-than-source.md 3.3 KB

# Worked example 2: DLQ retention shorter than the source (R3)

A dead-letter queue that is wired correctly, sized correctly on `maxReceiveCount`, and still silently deletes the messages you most need to see. This is the skill's defining check: it depends on one piece of SQS semantics that is easy to get wrong. Fixtures and replay test under `../fixtures/02-dlq-retention-shorter-than-source/` and `../tests/replay_02_dlq_retention.py`.

## Scenario

- **Queue**: `order-events`, with DLQ `order-events-dlq`.
- **Symptom**: during an incident, an engineer opens the DLQ to inspect the failed orders. It is nearly empty, even though the source queue clearly dead-lettered a batch an hour ago. The messages arrived in the DLQ and were deleted within minutes.

## Step 1: parse the attributes

Source (`queue.json`):
```
MessageRetentionPeriod 345600   (4 days)
RedrivePolicy          {"deadLetterTargetArn":"...:order-events-dlq","maxReceiveCount":5}
```
DLQ (`dlq.json`):
```
QueueArn               ...:order-events-dlq
MessageRetentionPeriod 86400    (1 day)
```

The `RedrivePolicy` is a JSON string; parsing it gives the DLQ ARN and `maxReceiveCount=5` (in band, so no R2). The retention check needs the DLQ's *own* attributes, which is why `dlq.json` must be loaded.

## Step 2: audit the redrive path

`maxReceiveCount=5` is healthy. The DLQ exists. But the source retains for 4 days and the DLQ retains for 1 day, so `dlq_retention (86400) <= source_retention (345600)`. That is **R3 (critical)**.

The mechanism is the part that catches people: **a message's age is measured from its original `SentTimestamp`, and SQS does not reset that timestamp when the message is moved to the DLQ.** A message that sits in `order-events` for 3 days before finally exhausting its 5 receives arrives in the DLQ already 3 days old. The DLQ's retention is 1 day. The message is over the limit the instant it arrives, and SQS deletes it. The DLQ looks perfectly configured. It catches nothing that failed slowly.

## Steps 3-5: lifecycle, exposure, FIFO

Clean. `5 x 60 = 300s` is far below the 4-day retention (no R4). Visibility is a deliberate 60s (no R5), retention is the default (no R6), SSE is on, no resource policy, standard queue. R3 is the only finding.

## Finding

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R3 | critical | source vs DLQ `MessageRetentionPeriod` | Set the DLQ's `MessageRetentionPeriod` above the source's, ideally to the 14-day maximum (`1209600`), so failed messages survive long enough to inspect and redrive. |

## Boundary

The audit proves the DLQ *can* delete messages on arrival. It cannot tell you whether it already has.

- How many messages were dropped this way, and when, is the DLQ's delete/redrive metrics over time, not a static attribute. Join: DLQ to its metrics over time.
- Whether anything is draining the DLQ at all is a property of the operational owner, not the queue. Join: DLQ to its owner.

## Why this is the R3 reference

Every visible signal says the dead-letter path is correct: a DLQ is attached, `maxReceiveCount` is sane, the DLQ even has messages arriving. The bug is a single inequality between two retention periods, made lethal by a non-obvious SQS semantic (the timestamp does not reset on redrive). It is exactly the judgment a config read should carry and a console glance does not.

03-maxreceivecount-too-low.md 2.8 KB

# Worked example 3: maxReceiveCount too low (R2)

A dead-letter queue wired correctly, sized correctly on retention, and set to give up after a single failed delivery. Transient blips dead-letter perfectly good messages. Fixtures and replay test under `../fixtures/03-maxreceivecount-too-low/` and `../tests/replay_03_maxreceivecount.py`.

## Scenario

- **Queue**: `email-dispatch`, with DLQ `email-dispatch-dlq`.
- **Symptom**: the DLQ has over a thousand messages in it. On inspection, almost all of them are valid emails that would have sent fine on a retry. A 90-second SES throttle this morning dead-lettered every message in flight, because the queue gives up after one attempt.

## Step 1: parse the attributes

Source (`queue.json`):
```
MessageRetentionPeriod 345600   (4 days)
RedrivePolicy          {"deadLetterTargetArn":"...:email-dispatch-dlq","maxReceiveCount":1}
```
DLQ (`dlq.json`):
```
MessageRetentionPeriod 1209600  (14 days)
```

`maxReceiveCount` parses to `1`.

## Step 2: audit the redrive path

The DLQ is present and its retention (14 days) is longer than the source (4 days), so R3 does not fire. But `maxReceiveCount=1` is below the sane band, which is **R2 (medium)**. A message that fails its single delivery attempt goes straight to the DLQ. There is no retry, so any recoverable, transient downstream failure (a rolling deploy, a brief throttle, a 2-second timeout) sends a good message to dead-letter. The DLQ fills with messages that were never poison, which masks the ones that are: when something is genuinely broken, it is buried under a thousand false positives.

## Steps 3-5: lifecycle, exposure, FIFO

Clean. `1 x 60 = 60s` is far below retention (no R4). Visibility is a deliberate 60s (no R5), retention is the default (no R6), SSE is on, no resource policy, standard queue. R2 is the only finding.

## Finding

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R2 | medium | `RedrivePolicy.maxReceiveCount` | Raise `maxReceiveCount` into the 3-10 band so transient failures are retried before quarantine. |

## Boundary

The audit proves the queue dead-letters on the first failure. It cannot tell you what fraction of the DLQ is transient-failure noise versus genuine poison.

- The ratio of recoverable to poison messages in the DLQ is a property of the message contents and the consumer's failure reasons, not a queue attribute. Join: DLQ to its consumers.
- Whether a redrive of the DLQ would now succeed is the downstream's current health, not a static attribute. Join: queue to its metrics over time.

## Why this is the R2 reference

It is the opposite failure from a missing DLQ: the dead-letter path works *too eagerly*. The fix is a single integer, but the symptom (a DLQ full of valid messages, the real poison invisible inside it) looks like a content problem until you read the one attribute that explains it.

04-poison-ages-out-before-dlq.md 3.9 KB

# Worked example 4: poison messages age out before the DLQ (R4)

The flagship. A dead-letter queue that is present, sized with generous retention, and never receives a single poison message, because of an arithmetic relationship between three attributes that nobody checks by hand. Fixtures and replay test under `../fixtures/04-poison-ages-out-before-dlq/` and `../tests/replay_04_poison_ages_out.py`.

## Scenario

- **Queue**: `ledger-reconcile`, with DLQ `ledger-reconcile-dlq`.
- **Symptom**: a daily reconciliation job has a recurring poison record. The DLQ was built specifically to catch it. The DLQ is always empty. The poison record is being silently deleted by retention before it ever reaches the DLQ.

## Step 1: parse the attributes

Source (`queue.json`):
```
VisibilityTimeout      900      (15 minutes)
MessageRetentionPeriod 345600   (4 days)
RedrivePolicy          {"deadLetterTargetArn":"...:ledger-reconcile-dlq","maxReceiveCount":1000}
```
DLQ (`dlq.json`):
```
MessageRetentionPeriod 1209600  (14 days)
```

`VisibilityTimeout=900`, `maxReceiveCount=1000`, `MessageRetentionPeriod=345600`.

## Step 2: audit the redrive path

The DLQ is present, and its 14-day retention exceeds the source's 4 days, so R3 does not fire. But `maxReceiveCount=1000` is above the sane band, raising **R2 (low)**: a poison message would be retried up to a thousand times before quarantine. That is the warning. The next check is the bug.

## Step 3: audit the message lifecycle

A poison message needs at least `maxReceiveCount x VisibilityTimeout` seconds of wall-clock to exhaust its receive count and dead-letter:

```
1000 x 900 = 900000 seconds  (about 10.4 days)
```

But `MessageRetentionPeriod` is `345600` seconds (4 days). Retention wins. The message is deleted by age after 4 days, having reached only about 384 of its 1000 receives, long before it is eligible for the DLQ. That is **R4 (critical)**. The dead-letter queue exists, is wired, and has generous retention, and the exact message it was built to catch never arrives in it.

This is pure arithmetic on three static attributes. It is almost never done by hand, because each attribute looks reasonable in isolation: a 15-minute visibility timeout for a slow job is sensible, a high `maxReceiveCount` for a flaky dependency is defensible, a 4-day retention is the default. The failure is in their product.

## Steps 4-5: exposure, FIFO

Clean. Visibility is a deliberate 900s (no R5), retention is the default (no R6), SSE is on, no resource policy, standard queue.

## Findings

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R4 | critical | `VisibilityTimeout` x `maxReceiveCount` vs `MessageRetentionPeriod` | Lower `maxReceiveCount` or `VisibilityTimeout`, or raise `MessageRetentionPeriod`, so `maxReceiveCount x VisibilityTimeout` stays well under retention. |
| R2 | low | `RedrivePolicy.maxReceiveCount` | Lower `maxReceiveCount` into the 3-10 band unless a specific replay requirement justifies more. Lowering it also resolves R4. |

## Boundary

The audit proves a poison message *cannot* reach the DLQ in time. It cannot confirm a poison message currently exists.

- Whether the reconciliation job is dead-lettering anything is the source queue's age-of-oldest-message and the consumer's per-message receive count, both time-series. Join: queue to its metrics over time.
- The real time-to-DLQ depends on how often a consumer actually receives the message (the arithmetic is a lower bound). Join: queue to its consumers.

## Why this is the R4 reference

R4 is the rule that most justifies the skill. The DLQ is present and correctly sized on every dimension a checklist would inspect; the failure is an interaction between three attributes that only a deliberate calculation surfaces. An agent with raw `GetQueueAttributes` access does not perform this multiplication zero-shot. The judgment is the multiplication and the comparison against retention.

05-default-visibility-short-retention.md 3.2 KB

# Worked example 5: default visibility timeout and short retention (R5, R6)

Two soft flags on one queue. Neither is a confirmed message-loss bug from the config alone; both are defaults that are usually accidental rather than chosen, and both defer their final verdict to something behind the boundary. This example exists to show the skill's honesty: it flags without overclaiming. Fixtures and replay test under `../fixtures/05-default-visibility-short-retention/` and `../tests/replay_05_default_visibility.py`.

## Scenario

- **Queue**: `click-events`, with DLQ `click-events-dlq`.
- **Context**: a high-volume analytics ingest queue. Two attributes look like they were never set deliberately.

## Step 1: parse the attributes

```
VisibilityTimeout      30       (the AWS default)
MessageRetentionPeriod 300      (5 minutes)
RedrivePolicy          {"deadLetterTargetArn":"...:click-events-dlq","maxReceiveCount":5}
```

## Step 2: audit the redrive path

DLQ present, `maxReceiveCount=5` in band, DLQ retention (14 days) far exceeds source (5 minutes). No R2, no R3.

## Step 3: audit the message lifecycle

- `VisibilityTimeout=30` is the AWS default. That raises **R5 (low)**. If any consumer takes longer than 30 seconds to process a click event, the message becomes visible again mid-processing and is delivered to a second consumer, causing duplicate work. Whether that actually happens depends on the consumer's processing time, **which is not a queue attribute**. So this is a flag to verify, not a confirmed bug, and it is surfaced as low severity for exactly that reason.
- `MessageRetentionPeriod=300` (5 minutes) raises **R6 (medium)**. Any consumer outage, deploy, or scaling lag longer than 5 minutes silently drops every queued message. For an analytics ingest that may be an acceptable trade (stale clicks are worthless), or it may be an accident. The config flags it; the operator decides.
- R4 does not fire: `5 x 30 = 150s`, under the 300s retention (barely, which is itself worth noting to the operator).

## Steps 4-5: exposure, FIFO

Clean. SSE on, no resource policy, standard queue.

## Findings

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R6 | medium | `MessageRetentionPeriod` | Raise retention to cover the longest plausible consumer outage, unless dropping stale messages is the intended behaviour. |
| R5 | low | `VisibilityTimeout` | Set the visibility timeout deliberately, above the consumer's p99 processing time, if duplicate delivery matters. |

## Boundary

Both findings defer to something the queue does not contain.

- Whether the 30s visibility timeout is actually too short is the consumer's processing-time distribution, not a queue attribute. Join: queue to its consumers.
- Whether the 5-minute retention actually loses messages is the consumer outage history, a time-series. Join: queue to its metrics over time.

## Why this is the R5 / R6 reference

It demonstrates the skill declining to overclaim. R5 in particular could be written as "visibility timeout too short, messages double-processed", but the skill cannot prove that from the config: it depends on consumer behaviour. Flagging it low and naming the join is the correct, honest move. A skill that asserted a bug here would be wrong as often as it was right.

06-public-queue-policy.md 3.6 KB

# Worked example 6: open resource policy on an unencrypted queue (R7, R8)

A queue whose resource policy authorises any AWS principal to send to it, with no encryption at rest. The exposure example. Fixtures and replay test under `../fixtures/06-public-queue-policy/` and `../tests/replay_06_public_policy.py`.

## Scenario

- **Queue**: `inbound-webhooks`. Intended to receive events from one specific SNS topic.
- **Symptom**: the resource policy was written to "allow sends" during a hurried integration and never scoped. As written, anyone with an AWS account can push messages onto it.

## Step 1: parse the attributes

```
SqsManagedSseEnabled   false
Policy                 {"Version":"2012-10-17","Statement":[{"Sid":"AllowSend",
                        "Effect":"Allow","Principal":"*","Action":"sqs:SendMessage",
                        "Resource":"...:inbound-webhooks"}]}
RedrivePolicy          {"deadLetterTargetArn":"...:inbound-webhooks-dlq","maxReceiveCount":5}
```

The `Policy` is a JSON string; parsing it gives one `Allow` statement with `Principal: "*"` and **no `Condition`**.

## Step 4: audit exposure

- The statement allows `Principal: "*"` for `sqs:SendMessage` with no narrowing condition (`aws:SourceArn`, `aws:SourceAccount`, `aws:PrincipalOrgID`). That is **R7 (high)**. As written, the policy authorises any AWS principal to send to the queue: the classic confused-deputy and public-queue exposure. A queue that should only accept messages from one SNS topic accepts them from anyone, which means anyone can inject events into the webhook pipeline.
- `SqsManagedSseEnabled` is `false` and there is no `KmsMasterKeyId`, so message bodies are not encrypted at rest. That is **R8 (low)**: low because whether it matters depends on what the webhook payloads contain, which the queue does not tell you.

The contrast that matters: a wildcard principal is not automatically wrong. The standard SNS-to-SQS subscription uses `Principal: "*"` *with* an `aws:SourceArn` condition pinning it to the topic. Example 8 has exactly that pattern and the skill does not flag it. R7 fires on the missing condition, not on the wildcard.

## Steps 2-3, 5: redrive, lifecycle, FIFO

Clean. DLQ present with sane `maxReceiveCount` and good retention (no R1/R2/R3). `5 x 120 = 600s` under retention (no R4). Visibility deliberate at 120s (no R5), retention default (no R6), standard queue.

## Findings

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R7 | high | `Policy` | Add a `Condition` pinning the principal to the intended source (`aws:SourceArn` for the SNS topic), or name explicit principal ARNs instead of `"*"`. |
| R8 | low | `SqsManagedSseEnabled` / `KmsMasterKeyId` | Enable SQS-managed SSE or a KMS key unless the payloads are confirmed non-sensitive. |

## Boundary

R7 reads the resource policy. That is only half the access story.

- The effective set of principals that can act on this queue is the *union* of this resource policy and every IAM identity policy in the account. A clean resource policy would not prove the queue is private. Join: queue to the account's IAM graph.
- Whether the open policy has actually been used to inject messages is the send metrics by source principal, a time-series. Join: queue to its metrics over time.

## Why this is the R7 / R8 reference

It exercises the precision the check needs: flag the wildcard-with-no-condition, do **not** flag the wildcard-with-`SourceArn` that is the legitimate SNS pattern. And it names the boundary that keeps R7 honest: a resource-policy audit can never be a complete access audit, because identity policies live outside the queue.

07-fifo-dedup-off.md 2.8 KB

# Worked example 7: FIFO queue with content-based deduplication off (R9)

A FIFO queue whose exactly-once guarantee rests entirely on a contract the queue cannot enforce: that every producer supplies a deduplication ID. Fixtures and replay test under `../fixtures/07-fifo-dedup-off/` and `../tests/replay_07_fifo_dedup_off.py`.

## Scenario

- **Queue**: `inventory-updates.fifo`, with DLQ `inventory-updates-dlq.fifo`.
- **Context**: a FIFO queue chosen for ordering and exactly-once processing of stock adjustments. `ContentBasedDeduplication` is off.

## Step 1: parse the attributes

```
QueueArn                   ...:inventory-updates.fifo
FifoQueue                  true
ContentBasedDeduplication  false
RedrivePolicy              {"deadLetterTargetArn":"...:inventory-updates-dlq.fifo","maxReceiveCount":5}
```

The `.fifo` suffix and `FifoQueue=true` mark this as a FIFO queue, so the FIFO invariants apply.

## Step 5: audit FIFO invariants

`ContentBasedDeduplication` is off, which is **R9 (low)**. With content-based dedup off, SQS does not derive a deduplication ID from the message body. The 5-minute deduplication guarantee therefore depends entirely on every producer supplying an explicit `MessageDeduplicationId`. If any producer omits it, two sends of the same stock adjustment within the dedup window are accepted as distinct messages, and the inventory is decremented twice.

Whether the producers actually send the ID is **a property of the producers, not of this queue**. The queue cannot enforce it and the audit cannot see it. So R9 is surfaced low and deferred to the boundary: the contract is flagged for verification, not asserted as broken.

## Steps 2-4: redrive, lifecycle, exposure

Clean. DLQ present, `maxReceiveCount=5` in band, DLQ retention (14 days) exceeds source (4 days). `5 x 120 = 600s` under retention. Visibility deliberate at 120s, retention default, SSE on, no resource policy. R9 is the only finding.

## Finding

| Code | Severity | Attribute | Fix |
|---|---|---|---|
| R9 | low | `ContentBasedDeduplication` | Either enable `ContentBasedDeduplication`, or confirm every producer sets `MessageDeduplicationId`. |

## Boundary

The audit proves the queue relies on a producer-supplied dedup ID. It cannot confirm the producers supply one.

- Whether each producer sends `MessageDeduplicationId` is producer code, not a queue attribute. Join: queue to its producers.
- Whether duplicates have actually been accepted is the sent/dedup-reject metrics over time. Join: queue to its metrics over time.

## Why this is the R9 reference

It is a third instance of the skill's honest-flag pattern (alongside R5 and R8): a real risk, grounded in a real attribute, whose confirmation lives on the other side of the boundary. The skill states the dependency and the fix, and stops at the wall instead of guessing whether the producers are well-behaved.

08-clean-standard.md 2.6 KB

# Worked example 8: a clean queue (control)

A correctly-configured queue. The audit produces zero findings, does not invent one, and still reports its boundary. This control is what keeps the skill trustworthy: an auditor that flags clean configs is one operators learn to ignore. Fixtures and replay test under `../fixtures/08-clean-standard/` and `../tests/replay_08_clean_standard.py`.

## Scenario

- **Queue**: `notification-fanout`, with DLQ `notification-fanout-dlq`. Subscribed to an SNS topic, processed by a worker fleet.

## Step 1: parse the attributes

```
VisibilityTimeout      180      (deliberate, not the 30s default)
MessageRetentionPeriod 345600   (4 days)
SqsManagedSseEnabled   true
Policy                 Allow Principal:"*" SendMessage,
                       Condition ArnEquals aws:SourceArn = ...:sns:account-events
RedrivePolicy          {"deadLetterTargetArn":"...:notification-fanout-dlq","maxReceiveCount":5}
```
DLQ (`dlq.json`):
```
MessageRetentionPeriod 1209600  (14 days)
```

## Steps 2-5: every check passes

- **Redrive (R1/R2/R3)**: DLQ present; `maxReceiveCount=5` in band; DLQ retention (14 days) is longer than the source (4 days). Clean.
- **Lifecycle (R4/R5/R6)**: `5 x 180 = 900s`, far under the 4-day retention, so poison messages reach the DLQ with room to spare. Visibility is a deliberate 180s, not the default. Retention is the 4-day default. Clean.
- **Exposure (R7/R8)**: the resource policy uses `Principal: "*"` **but narrows it** with `aws:SourceArn` pinned to the SNS topic. This is the standard, correct SNS-to-SQS pattern, so R7 does **not** fire. SSE is on, so R8 does not fire. Clean.
- **FIFO (R9)**: standard queue, not FIFO. Not applicable.

No findings.

## Boundary

A clean configuration is not a clean system. Even here, the audit reports what it cannot see:

- The queue could still be failing right now (a crashing consumer, a producer that stopped) for a reason only the live metrics show. Join: queue to its metrics over time.
- The effective access is still the union of this scoped resource policy and the account's identity policies. Join: queue to the account's IAM graph.
- Whether anyone is draining the DLQ is still unknown. Join: DLQ to its owner.

## Why this is the control

It pins two behaviours the other seven examples cannot: that the skill produces **zero** findings on a correct queue (no false positives), and specifically that a wildcard principal scoped by `aws:SourceArn` is recognised as legitimate and not flagged as R7. It also makes the boundary point unmissable: the audit reports the join it cannot make even when there is nothing to fix, because a sound config and a healthy system are different claims.

fixtures

01-no-dlq

queue.json 564 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/payments-capture",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:payments-capture",
    "VisibilityTimeout": "120",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "ApproximateNumberOfMessages": "412",
    "ApproximateNumberOfMessagesNotVisible": "9",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

02-dlq-retention-shorter-than-source

dlq.json 649 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:order-events-dlq",
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "86400",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:order-events\"]}",
    "ApproximateNumberOfMessages": "37",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 632 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/order-events",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:order-events",
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\",\"maxReceiveCount\":5}",
    "ApproximateNumberOfMessages": "88",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

03-maxreceivecount-too-low

dlq.json 659 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/email-dispatch-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:email-dispatch-dlq",
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:email-dispatch\"]}",
    "ApproximateNumberOfMessages": "1043",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 637 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/email-dispatch",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:email-dispatch",
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:email-dispatch-dlq\",\"maxReceiveCount\":1}",
    "ApproximateNumberOfMessages": "5",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

04-poison-ages-out-before-dlq

dlq.json 663 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq",
    "VisibilityTimeout": "900",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile\"]}",
    "ApproximateNumberOfMessages": "0",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 648 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/ledger-reconcile",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile",
    "VisibilityTimeout": "900",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:ledger-reconcile-dlq\",\"maxReceiveCount\":1000}",
    "ApproximateNumberOfMessages": "26",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

05-default-visibility-short-retention

dlq.json 650 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/click-events-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:click-events-dlq",
    "VisibilityTimeout": "30",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:click-events\"]}",
    "ApproximateNumberOfMessages": "2",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 631 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/click-events",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:click-events",
    "VisibilityTimeout": "30",
    "MessageRetentionPeriod": "300",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:click-events-dlq\",\"maxReceiveCount\":5}",
    "ApproximateNumberOfMessages": "1190",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

06-public-queue-policy

dlq.json 663 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/inbound-webhooks-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:inbound-webhooks-dlq",
    "VisibilityTimeout": "120",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:inbound-webhooks\"]}",
    "ApproximateNumberOfMessages": "4",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 908 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/inbound-webhooks",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:inbound-webhooks",
    "VisibilityTimeout": "120",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "false",
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"inbound-webhooks-policy\",\"Statement\":[{\"Sid\":\"AllowSend\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:eu-west-1:211125758836:inbound-webhooks\"}]}",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:inbound-webhooks-dlq\",\"maxReceiveCount\":5}",
    "ApproximateNumberOfMessages": "73",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

07-fifo-dedup-off

dlq.json 748 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/inventory-updates-dlq.fifo",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:inventory-updates-dlq.fifo",
    "VisibilityTimeout": "120",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "FifoQueue": "true",
    "ContentBasedDeduplication": "false",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:inventory-updates.fifo\"]}",
    "ApproximateNumberOfMessages": "0",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 804 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/inventory-updates.fifo",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:inventory-updates.fifo",
    "VisibilityTimeout": "120",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "FifoQueue": "true",
    "ContentBasedDeduplication": "false",
    "DeduplicationScope": "queue",
    "FifoThroughputLimit": "perQueue",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:inventory-updates-dlq.fifo\",\"maxReceiveCount\":5}",
    "ApproximateNumberOfMessages": "61",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

08-clean-standard

dlq.json 672 B

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout-dlq",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq",
    "VisibilityTimeout": "180",
    "MessageRetentionPeriod": "1209600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "0",
    "SqsManagedSseEnabled": "true",
    "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"arn:aws:sqs:eu-west-1:211125758836:notification-fanout\"]}",
    "ApproximateNumberOfMessages": "1",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

queue.json 1 KB

{
  "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/211125758836/notification-fanout",
  "Attributes": {
    "QueueArn": "arn:aws:sqs:eu-west-1:211125758836:notification-fanout",
    "VisibilityTimeout": "180",
    "MessageRetentionPeriod": "345600",
    "MaximumMessageSize": "262144",
    "DelaySeconds": "0",
    "ReceiveMessageWaitTimeSeconds": "20",
    "SqsManagedSseEnabled": "true",
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"notification-fanout-policy\",\"Statement\":[{\"Sid\":\"AllowSnsTopic\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:eu-west-1:211125758836:notification-fanout\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:eu-west-1:211125758836:account-events\"}}}]}",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:211125758836:notification-fanout-dlq\",\"maxReceiveCount\":5}",
    "ApproximateNumberOfMessages": "14",
    "CreatedTimestamp": "1737974400",
    "LastModifiedTimestamp": "1748000000"
  }
}

tests

eval

eval_results.json 220.4 KB

[
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "control",
    "trial": 0,
    "score": 6,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      1,
      0,
      1
    ],
    "notes": [
      "Agent correctly parses the DLQ retention as 86400 seconds (1 day) and source as 345600 seconds (4 days), demonstrating correct string-typed attribute parsing.",
      "Agent identifies the core defect: DLQ retention is shorter than source queue retention, causing messages to be silently dropped on arrival, matching the ground-truth R3 finding.",
      "No false positives that mischaracterize scoped policies as public; additional findings like short polling and encryption are presented as soft flags, not critical misconfigurations.",
      "Issue 1 (DLQ retention / silent message loss) is ranked as HIGH and listed first, correctly headlining the silent-loss defect above operational and compliance concerns.",
      "Agent notes it cannot determine when messages arrived or confirm operational ownership, and explicitly states DLQ metrics over time require CloudWatch inspection, matching the ground-truth boundary.",
      "Issues 3 (maxReceiveCount), 5 (SSE-SQS), and 6 (no resource policy) are presented as confirmed issues rather than flags to verify depending on context; for example, maxReceiveCount=5 is called potentially problematic without acknowledging it may be perfectly appropriate.",
      "Top fix correctly recommends raising the DLQ MessageRetentionPeriod to 1,209,600 seconds (14-day maximum), matching the ground-truth recommendation in substance and providing the exact CLI command."
    ],
    "agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### Issue 1 \u2014 DLQ Retention Period Too Short\n**Severity: HIGH**\n\n**Finding:**\nThe DLQ `MessageRetentionPeriod` is `86400` seconds (24 hours). The source queue retains messages for `345600` seconds (4 days).\n\n**Risk:**\nA message that reaches the DLQ has already spent an unknown amount of time on the source queue (potentially up to 4 days). If it arrives in the DLQ near the end of that window, it can be **silently dropped within hours** before any engineer investigates it. There is no delivery failure \u2014 the message simply disappears with no further signal. For order events this means lost orders with no recovery path.\n\n**Fix:**\nSet the DLQ retention to the **maximum of 1,209,600 seconds (14 days)**. This maximises the investigation and replay window.\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n### Issue 2 \u2014 DLQ Has 37 Messages With No Redrive or Alarm Visible\n**Severity: HIGH**\n\n**Finding:**\n`ApproximateNumberOfMessages` on the DLQ is `37`. There is no `RedriveAllowPolicy` configured on the source queue to permit redrive back, and no evidence of active consumption or alerting on this queue.\n\n**Risk:**\n37 order events have already failed processing and are sitting unhandled. Given Issue 1 above, they are also **actively expiring**. If these represent real orders they are unprocessed business events. The queue depth also indicates the worker failure mode is live and recurring, not historical.\n\n**Fix:**\nImmediate actions:\n1. Determine when these messages arrived (CloudWatch `NumberOfMessagesSent` on the DLQ).\n2. Investigate worker errors in logs before any redrive.\n3. Once root cause is confirmed, redrive using the SQS console or:\n\n```bash\naws sqs start-message-move-task \\\n  --source-arn arn:aws:sqs:eu-west-1:211125758836:order-events-dlq \\\n  --destination-arn arn:aws:sqs:eu-west-1:211125758836:order-events\n```\n\n4. Create a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ with a threshold of `0` (alert on any message).\n\n---\n\n### Issue 3 \u2014 `maxReceiveCount` of 5 May Be Too Low Depending on Worker Behaviour\n**Severity: MEDIUM**\n\n**Finding:**\n`maxReceiveCount` is `5`. This means a message is moved to the DLQ after 5 failed receive-and-not-delete cycles.\n\n**Risk:**\nIf workers experience transient infrastructure failures (network blips, cold starts, deployment rollouts), 5 attempts may be insufficient to distinguish a genuinely poisoned message from a temporarily unhealthy worker. This causes **unnecessary DLQ promotion** of recoverable messages, which then require manual redrive and carry the retention risk from Issue 1.\n\nConversely, if processing has side effects (e.g. partial order writes), each retry increases the risk of **partial duplication** unless the worker is fully idempotent.\n\n**Fix:**\n- If workers are idempotent and transient failures are common: raise to `10`.\n- If workers are not fully idempotent: keep low but **ensure idempotency** at the application layer (deduplicate on order ID) before raising it.\n- Document the chosen value and the rationale.\n\n---\n\n### Issue 4 \u2014 DLQ `ReceiveMessageWaitTimeSeconds` is `0` (Short Polling)\n**Severity: LOW**\n\n**Finding:**\nThe source queue correctly uses long polling (`ReceiveMessageWaitTimeSeconds=20`). The DLQ is set to `0`, meaning short polling.\n\n**Risk:**\nThe DLQ is likely polled by an alerting consumer, ops tooling, or a redrive process. Short polling can return **empty responses even when messages exist** (it samples a subset of SQS servers), leading to missed detections during incident triage and unnecessary API cost.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### Issue 5 \u2014 Encryption Uses SSE-SQS, Not SSE-KMS\n**Severity: LOW (Compliance-Dependent)**\n\n**Finding:**\nBoth queues have `SqsManagedSseEnabled=true`, meaning AWS manages the encryption key. There is no customer-managed KMS key (CMK).\n\n**Risk:**\n- You have **no control over key rotation policy**, no ability to revoke access via key policy, and no CloudTrail visibility into individual decrypt operations.\n- For order events containing PII or payment-adjacent data this is likely a compliance gap (PCI-DSS, GDPR depending on message content).\n- A compromised AWS account has no key-level blast radius control.\n\n**Fix:**\nIf regulatory or security requirements demand it, migrate to SSE-KMS with a CMK:\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n  --attributes KmsMasterKeyId=<your-cmk-arn>\n```\n\nEnsure the worker fleet IAM roles and the DLQ both reference the same CMK with appropriate `kms:Decrypt` / `kms:GenerateDataKey` grants.\n\n---\n\n### Issue 6 \u2014 No SQS Resource Policy Present\n**Severity: LOW (Defence-in-Depth)**\n\n**Finding:**\nNeither queue has a resource-based policy (`Policy` attribute) returned by the API, which means access is controlled **solely by IAM identity policies**.\n\n**Risk:**\n- There is no explicit deny to prevent other principals in account `211125758836` from reading or purging these queues if their IAM policies are overly broad.\n- No cross-account access is scoped, so any future cross-account IAM grants would have no queue-side guardrail.\n\n**Fix:**\nAdd a least-privilege resource policy that explicitly allows only the expected worker role ARNs and denies all other principals:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"AllowWorkerFleet\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::211125758836:role/order-worker-role\"\n      },\n      \"Action\": [\n        \"sqs:ReceiveMessage\",\n        \"sqs:DeleteMessage\",\n        \"sqs:ChangeMessageVisibility\",\n        \"sqs:GetQueueAttributes\"\n      ],\n      \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:order-events\"\n    },\n    {\n      \"Sid\": \"DenyEverythingElse\",\n      \"Effect\": \"Deny\",\n      \"Principal\": \"*\",\n      \"Action\": \"sqs:*\",\n      \"Resource\": \"arn:aws:sqs:eu-west-1:211125758836:order-events\",\n      \"Condition\": {\n        \"StringNotEquals\": {\n          \"aws:PrincipalArn\": \"arn:aws:iam::211125758836:role/order-worker-role\"\n        }\n      }\n    }\n  ]\n}\n```\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Message Loss Risk |\n|---|-------|----------|-------------------|\n| 1 | DLQ retention 24h vs 4-day source queue | **HIGH** | Direct \u2014 messages expire |\n| 2 | 37 unhandled DLQ messages, no alerting | **HIGH** | Active \u2014 orders already lost |\n| 3 | `maxReceiveCount=5` may be too aggressive | **MEDIUM** | Indirect \u2014 unnecessary DLQ promotion |\n| 4 | DLQ short polling | **LOW** | Operational \u2014 missed triage signal |\n| 5 | SSE-SQS not SSE-KMS | **LOW** | Compliance/exposure risk |\n| 6 | No resource policy | **LOW** | Exposure risk |\n\n**Immediate action required on Issues 1 and 2.** The DLQ messages are expiring now.",
    "elapsed_s": 55.7
  },
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "control",
    "trial": 1,
    "score": 5,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      0,
      0,
      1
    ],
    "notes": [
      "Agent correctly parses the 345600s (4-day) source retention and 86400s (1-day) DLQ retention, identifying the numeric mismatch accurately.",
      "Agent identifies the core R3 defect \u2014 DLQ retention shorter than source retention causing silent message loss due to SentTimestamp not resetting on redrive.",
      "Agent does not flag any false positives; no access policies are miscalled as public or other nonexistent misconfigurations asserted.",
      "Agent correctly ranks the DLQ retention issue as Critical and leads with it as the most serious finding.",
      "Agent does not explicitly name any boundary it cannot cross (e.g., cannot see DLQ metrics over time or identify the operational owner); the boundary acknowledgment is absent.",
      "Several findings (maxReceiveCount, VisibilityTimeout, KMS) are presented as confirmed bugs or definite problems rather than flags to verify depending on workload characteristics.",
      "The top fix correctly recommends raising DLQ MessageRetentionPeriod to the 14-day maximum (1209600 seconds), matching the ground-truth recommendation in substance."
    ],
    "agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### CRITICAL\n\n#### 1. DLQ retention period shorter than source queue retention period\n\n**What's configured:**\n- Source queue `MessageRetentionPeriod`: `345600` (4 days)\n- DLQ `MessageRetentionPeriod`: `86400` (1 day)\n\n**Why this is a problem:**\nA message can spend up to 4 days being retried on the source queue before it lands in the DLQ. Once it arrives, it has only 1 day left \u2014 but the DLQ clock uses the **original enqueue timestamp**, not the arrival timestamp at the DLQ. A message that exhausts retries on day 3 arrives in the DLQ with roughly 1 day of retention remaining. A message that exhausts retries on day 4 may expire **immediately on arrival** or already be gone. You will silently lose failed messages with no chance to inspect or replay them.\n\n**Fix:**\nSet the DLQ retention to **at least as long as the source queue**, ideally longer to give operators time to investigate. The maximum is 14 days.\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=1209600  # 14 days\n```\n\n---\n\n### HIGH\n\n#### 2. DLQ has 37 messages with no redrive alarm or apparent consumption\n\n**What's configured:**\n- `ApproximateNumberOfMessages`: `37` on the DLQ\n- No redrive policy on the DLQ (no DLQ-of-DLQ configured)\n- `RedriveAllowPolicy` is set correctly, but there is no evidence of a consumer or alarm\n\n**Why this is a problem:**\n37 messages sitting in the DLQ means 37 order events have already failed all 5 retry attempts. Given the 1-day retention window (see issue #1), these messages are at active risk of expiry. Even under a corrected retention period, a DLQ without alerting is operationally silent \u2014 failures accumulate unnoticed until business impact occurs.\n\n**Fix:**\n- Immediately inspect and replay or action the 37 existing messages before they expire.\n- Create a CloudWatch alarm on `ApproximateNumberOfMessagesVisible` for the DLQ:\n\n```bash\naws cloudwatch put-metric-alarm \\\n  --alarm-name \"order-events-dlq-not-empty\" \\\n  --metric-name ApproximateNumberOfMessagesVisible \\\n  --namespace AWS/SQS \\\n  --dimensions Name=QueueName,Value=order-events-dlq \\\n  --statistic Sum \\\n  --period 60 \\\n  --threshold 1 \\\n  --comparison-operator GreaterThanOrEqualToThreshold \\\n  --evaluation-periods 1 \\\n  --alarm-actions <your-sns-topic-arn>\n```\n\n---\n\n### MEDIUM\n\n#### 3. `maxReceiveCount` of 5 may be too low depending on worker failure modes\n\n**What's configured:**\n- `maxReceiveCount`: `5`\n- `VisibilityTimeout`: `60` seconds\n\n**Why this is a problem:**\nWith only 5 receive attempts, a message that hits transient infrastructure failures (cold starts, downstream timeouts, brief DB unavailability) will be moved to the DLQ quickly. 5 attempts at 60-second visibility timeout means a message can be dead-lettered in as little as 5 minutes. This is appropriate for poison-pill detection, but too aggressive if your workers experience legitimate transient errors lasting more than a few minutes. The current DLQ depth of 37 may partly reflect this.\n\n**Fix:**\nEvaluate your worker's error profile. If transient failures are common, increase `maxReceiveCount` to `10`\u2013`20`. If you need faster poison-pill isolation, keep it low but ensure your worker distinguishes transient vs. permanent errors and uses `ChangeMessageVisibility` to back off rather than immediately releasing the message.\n\n---\n\n### MEDIUM\n\n#### 4. VisibilityTimeout may not account for worst-case processing time\n\n**What's configured:**\n- `VisibilityTimeout`: `60` seconds on both queues\n\n**Why this is a problem:**\nIf any worker takes longer than 60 seconds to process an order event (e.g., downstream API latency, large payload processing), the message becomes visible again and a second worker will pick it up. This causes **duplicate processing**. Both workers will likely succeed and delete the message \u2014 but only one delete will succeed; the other is a no-op \u2014 or, worse, both complete business logic (e.g., charge a customer, create an order) before either deletes the message. The `VisibilityTimeout` must be a safe upper bound for processing time, not a typical-case estimate.\n\n**Fix:**\nSet the visibility timeout to your **P99 processing time plus a safety margin** (at least 1.5\u00d7). If your workers are long-running, implement heartbeating by calling `ChangeMessageVisibility` periodically to extend the timeout during processing. If P99 is genuinely under 60 seconds, document that explicitly so future changes don't regress it silently.\n\n---\n\n### LOW\n\n#### 5. DLQ `ReceiveMessageWaitTimeSeconds` is 0 (short polling)\n\n**What's configured:**\n- Source queue: `ReceiveMessageWaitTimeSeconds`: `20` (long polling \u2014 correct)\n- DLQ: `ReceiveMessageWaitTimeSeconds`: `0` (short polling)\n\n**Why this is a problem:**\nIf you have an automated consumer or replay tooling that polls the DLQ, short polling will increase API call costs and may miss messages distributed across SQS internal partitions on any given poll. This is low severity because the DLQ typically has low-frequency manual or automated replay traffic, but it is an inconsistency that can cause incomplete reads.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n### INFORMATIONAL\n\n#### 6. Encryption is SQS-managed, not KMS-managed\n\n**What's configured:**\n- `SqsManagedSseEnabled`: `true` on both queues\n- No `KmsMasterKeyId` attribute present\n\n**Why this is a problem:**\nSQS-managed SSE encrypts at rest but gives you no key rotation control, no ability to revoke access via key policy, no CloudTrail visibility of decrypt operations, and no cross-account key sharing capability. For order events containing customer or payment-adjacent data, this is likely below your compliance bar.\n\n**Fix:**\nCreate a customer-managed KMS key with an appropriate key policy and migrate both queues:\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n  --attributes KmsMasterKeyId=<your-cmk-arn>\n```\n\nRepeat for the DLQ. Ensure your worker IAM roles have `kms:Decrypt` and `kms:GenerateDataKey` on the key.\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Risk |\n|---|-------|----------|------|\n| 1 | DLQ retention (1d) < source retention (4d) | **Critical** | Silent message loss |\n| 2 | 37 messages in DLQ, no alarm, expiring | **High** | Active data loss |\n| 3 | `maxReceiveCount=5` may be too aggressive | **Medium** | Premature DLQ routing |\n| 4 | VisibilityTimeout may allow duplicate processing | **Medium** | Duplicate order processing |\n| 5 | DLQ using short polling | **Low** | Cost, incomplete reads |\n| 6 | SQS-managed SSE instead of KMS CMK | **Informational** | Compliance/audit gap |",
    "elapsed_s": 46.8
  },
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "control",
    "trial": 2,
    "score": 5,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      0,
      0,
      1
    ],
    "notes": [
      "Correctly parses RedrivePolicy and identifies 86400s (1 day) DLQ vs 345600s (4 days) source retention.",
      "Identifies the ground-truth R3 defect: DLQ retention shorter than source, with correct SentTimestamp reasoning.",
      "No false positives introduced; all findings are plausible operational concerns, not fabricated misconfigs.",
      "Ranks the DLQ retention issue as CRITICAL and leads with it, correctly prioritizing silent message loss.",
      "Agent never names a specific thing it cannot determine from config alone (e.g., how many messages were already silently dropped, or who the operational owner is); boundary condition is not acknowledged.",
      "Several soft flags (maxReceiveCount, SSE-SQS vs CMK, DLQ-on-DLQ) are presented as confirmed issues rather than 'flags to verify depending on context'; for example, maxReceiveCount:5 is called a risk without acknowledging it may be deliberately chosen.",
      "Top fix correctly recommends raising DLQ MessageRetentionPeriod to 1209600 (14-day maximum), matching ground truth."
    ],
    "agent_output": "## SQS Queue Audit: `order-events`\n\n---\n\n### CRITICAL\n\n#### 1. DLQ Retention Period Too Short (Message Loss)\n\n**Finding:** `order-events-dlq` has `MessageRetentionPeriod: 86400` (24 hours). The source queue retains messages for `345600` seconds (4 days).\n\n**Impact:** A message can spend up to 4 days being retried on the source queue before landing in the DLQ. Once it arrives in the DLQ it could be close to or already past the 24-hour DLQ window, meaning it gets **silently dropped before anyone can investigate or replay it**. Even in normal failure scenarios, 24 hours gives very little time for an on-call engineer to be paged, investigate, and act before evidence is gone.\n\n**Fix:** Set DLQ retention to at least as long as the source queue, preferably the SQS maximum:\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=1209600  # 14 days\n```\n\n---\n\n### HIGH\n\n#### 2. No DLQ Alarm on Depth (Silent Failures)\n\n**Finding:** `ApproximateNumberOfMessages: 37` on the DLQ right now, with no CloudWatch alarm visible in this configuration. 37 messages are already dead.\n\n**Impact:** Failed messages accumulate silently. The DLQ is only useful if someone is alerted to drain or replay it. Without an alarm this is operationally equivalent to having no DLQ \u2014 failures are invisible until someone manually checks.\n\n**Fix:** Create an alarm that fires as soon as any message lands in the DLQ:\n```bash\naws cloudwatch put-metric-alarm \\\n  --alarm-name \"order-events-dlq-not-empty\" \\\n  --metric-name ApproximateNumberOfMessagesVisible \\\n  --namespace AWS/SQS \\\n  --dimensions Name=QueueName,Value=order-events-dlq \\\n  --statistic Sum \\\n  --period 60 \\\n  --threshold 1 \\\n  --comparison-operator GreaterThanOrEqualToThreshold \\\n  --evaluation-periods 1 \\\n  --alarm-actions <your-sns-topic-arn>\n```\n\n---\n\n#### 3. `maxReceiveCount: 5` May Be Too Low or Too High Depending on Worker Behaviour (Message Loss / Incorrect DLQ Routing)\n\n**Finding:** A message is moved to the DLQ after 5 receive attempts. This interacts directly with `VisibilityTimeout: 60`.\n\n**Two concrete risks:**\n\n- **Too low for transient errors:** If your workers have retry/backoff logic within a single receive (re-queuing via visibility timeout extension), 5 attempts is exhausted quickly during a downstream outage, sending legitimately retriable messages to the DLQ prematurely.\n- **Too high for poison pills:** If the message is a hard poison pill (malformed payload causing worker crash), 5 attempts means 5 worker failures and 5 \u00d7 60s = potentially several minutes of blocked concurrency per message before quarantine, depending on fleet size.\n\n**Fix:** Audit worker retry behaviour. A common safe default for order processing is `maxReceiveCount: 3` for fast-fail on poison pills, combined with explicit worker-side retry logic with exponential backoff for transient errors. Align the value deliberately rather than leaving it at the default. Update via:\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n  --attributes '{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":3}\"}'\n```\n\n---\n\n### MEDIUM\n\n#### 4. DLQ `ReceiveMessageWaitTimeSeconds: 0` (Unnecessary Cost / Polling Behaviour)\n\n**Finding:** The DLQ uses short polling (`ReceiveMessageWaitTimeSeconds: 0`) while the source queue correctly uses long polling (`20`).\n\n**Impact:** If any automated consumer or replay tooling polls the DLQ (e.g. a Lambda with event source mapping, or a manual replay script), short polling returns empty responses frequently, increases API call costs, and can miss messages present in non-sampled partitions. Not a message-loss risk in isolation, but operationally sloppy and can cause replay tooling to appear to report an empty DLQ when it is not.\n\n**Fix:**\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes ReceiveMessageWaitTimeSeconds=20\n```\n\n---\n\n#### 5. Source Queue Retention Period Is Only 4 Days (Marginal for Order Events)\n\n**Finding:** `MessageRetentionPeriod: 345600` = 4 days on the source queue.\n\n**Impact:** If the worker fleet goes completely down (deployment failure, IAM breakage, capacity event) for more than 4 days, unprocessed order events are **permanently dropped with no recovery path**. For financial/order data, 4 days is a tight margin.\n\n**Fix:** Evaluate business requirements. For order events, consider raising to 7\u201314 days (SQS max is 14 days = 1209600 seconds) to give adequate recovery runway:\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n  --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n#### 6. Encryption Is SQS-Managed (SSE-SQS), Not KMS\n\n**Finding:** `SqsManagedSseEnabled: true` on both queues. No `KmsMasterKeyId` is present.\n\n**Impact:** SSE-SQS encrypts at rest, but the key is AWS-managed with no customer control. You cannot:\n- Audit key usage via CloudTrail\n- Restrict queue access by key policy\n- Rotate or revoke the key independently\n- Meet compliance requirements (PCI-DSS, SOC2 controls) that mandate customer-managed keys for sensitive data\n\nOrder events likely contain PII and financial data.\n\n**Fix:** Switch to a CMK:\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events \\\n  --attributes KmsMasterKeyId=<your-cmk-arn>\n```\nApply the same to the DLQ. Ensure your worker IAM roles have `kms:Decrypt` and `kms:GenerateDataKey` on the CMK.\n\n---\n\n### LOW\n\n#### 7. No Dead-Letter Queue on the DLQ Itself\n\n**Finding:** The DLQ has no `RedrivePolicy` of its own.\n\n**Impact:** If a consumer of the DLQ (e.g. a replay Lambda) itself fails repeatedly while processing DLQ messages, SQS will eventually make those messages invisible and ultimately drop them when retention expires. This is low severity because DLQ consumers should be simple, but it is a gap in the safety net.\n\n**Fix:** Attach a secondary DLQ (`order-events-dlq-dlq`) to the DLQ with a low `maxReceiveCount` (e.g. 3) and a maximum retention period. Alert on any messages reaching it.\n\n---\n\n### Summary Table\n\n| # | Issue | Severity | Risk |\n|---|-------|----------|------|\n| 1 | DLQ retention 24h < source queue 4d | **CRITICAL** | Silent message loss |\n| 2 | No alarm on DLQ depth (37 msgs dead now) | **HIGH** | Silent failures |\n| 3 | `maxReceiveCount:5` not deliberately set | **HIGH** | Premature DLQ or slow poison-pill quarantine |\n| 4 | DLQ short polling | **MEDIUM** | Replay tooling unreliable, cost |\n| 5 | Source queue retention only 4 days | **MEDIUM** | Message loss on extended outage |\n| 6 | SSE-SQS not CMK | **MEDIUM** | Compliance gap, no key audit trail |\n| 7 | No DLQ-on-DLQ | **LOW** | Gap in safety net |",
    "elapsed_s": 47.0
  },
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "treatment",
    "trial": 0,
    "score": 7,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      1,
      1,
      1
    ],
    "notes": [
      "Correctly parses RedrivePolicy JSON and converts all string-typed seconds to human-readable durations.",
      "Identifies R3 (DLQ retention 1 day < source retention 4 days) with correct SentTimestamp reasoning, matching the ground-truth defect.",
      "No false positives; extra rules R8/R9 etc. are evaluated and correctly dismissed.",
      "R3 is ranked CRITICAL and headlined as the primary finding; no soft flag is elevated above it.",
      "Explicitly names inability to determine dropped-message counts over time (CloudWatch metrics join) and operational owner/consumer, matching ground-truth boundary.",
      "Consumer processing time, live redrive behaviour, and IAM effective permissions are presented as items to verify, not confirmed bugs.",
      "Top fix recommends raising DLQ MessageRetentionPeriod above source retention, ideally toward the maximum, matching the ground-truth fix."
    ],
    "agent_output": "# SQS Queue Audit \u2014 `order-events`\n\n---\n\n## Step 1: Parse the Attributes\n\n**Source queue (`order-events`)**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:order-events` | \u2014 |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `{\"deadLetterTargetArn\":\"arn:\u2026:order-events-dlq\",\"maxReceiveCount\":5}` | DLQ wired; maxReceiveCount = 5 |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**Dead-letter queue (`order-events-dlq`)**\n\n| Attribute | Raw value | Parsed |\n|---|---|---|\n| `QueueArn` | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` | \u2014 |\n| `MessageRetentionPeriod` | `\"86400\"` | 86,400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE-SQS on |\n\n---\n\n## Step 2\u20135: Rule Evaluation\n\n### R1 \u2014 No DLQ\n`RedrivePolicy` is present, pointing to `order-events-dlq`. **Not fired.**\n\n### R2 \u2014 maxReceiveCount outside 3\u201310\n`maxReceiveCount = 5`. Within the sane band [3, 10]. **Not fired.**\n\n### R3 \u2014 DLQ retention not longer than source *(critical)*\n- Source retention: **345,600 s (4 days)**\n- DLQ retention: **86,400 s (1 day)**\n- DLQ retention (86,400 s) is **less than** source retention (345,600 s). **Fired.**\n\nA message's `SentTimestamp` is set at first enqueue and is never reset when SQS moves the message to the DLQ. A message that is processed slowly \u2014 held in the source queue for, say, 70 hours before its receive count is exhausted \u2014 arrives in the DLQ with 70 hours of age already counted against a 24-hour retention window. It is deleted from the DLQ almost immediately, with no time for an engineer to inspect or replay it. The DLQ reads as wired and healthy; the messages you most need to examine are the ones it silently discards.\n\n### R4 \u2014 Poison messages age out before reaching the DLQ *(critical)*\n- Minimum wall-clock to exhaust receive count: `maxReceiveCount \u00d7 VisibilityTimeout = 5 \u00d7 60 = 300 s`\n- Source retention: `345,600 s`\n- `300 s \u2264 345,600 s` \u2192 inequality does **not** hold. **Not fired.**\n\nThe arithmetic is satisfactory: a poison message has more than enough retention headroom to exhaust all five delivery attempts and reach the DLQ under the configured values.\n\n### R5 \u2014 Visibility timeout at 30 s default\n`VisibilityTimeout = 60 s`. Not at the 30 s default. **Not fired.**\n\n### R6 \u2014 Retention shorter than a plausible outage *(medium)*\n- Source retention: **345,600 s (4 days)**. Well above the 1-hour threshold. **Not fired.**\n- DLQ retention: **86,400 s (1 day)**. Above the 1-hour threshold; however, R3 above already captures the more precise and severe form of this problem.\n\n### R7 \u2014 Open resource policy\nNo `Policy` attribute on either queue. **Not fired.**\n\n### R8 \u2014 Encryption at rest disabled\n`SqsManagedSseEnabled = \"true\"` on both queues. **Not fired.**\n\n### R9 \u2014 FIFO with content-based dedup off\nStandard queue; FIFO invariants do not apply. **Not fired.**\n\n---\n\n## Output\n\n### 1. Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ** | Wired \u2192 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n\n---\n\n### 2. Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Attributes:**\n- Source `MessageRetentionPeriod`: 345,600 s (4 days)\n- DLQ `MessageRetentionPeriod`: 86,400 s (1 day)\n\n**What breaks:**\nSQS preserves the original `SentTimestamp` when a message is moved to the DLQ \u2014 it does not reset the clock. A message that spends most of its source-queue lifetime being retried (e.g., 80+ hours before its fifth delivery attempt fails) arrives in the DLQ with the majority of the DLQ's 24-hour window already consumed. In the worst case it arrives already expired and is deleted on ingestion. The DLQ is non-empty (37 messages are visible right now), which confirms it is receiving messages \u2014 but those 37 messages are the *fast* failures. Slow failures \u2014 the ones most likely to reflect a systemic processing problem \u2014 may be aging out of the DLQ invisibly. There is no error surfaced anywhere; from the console the queue looks correct.\n\n**Recommendation:**\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source's. A practical rule: DLQ retention \u2265 source retention + one operational response window. Given a 4-day source, a DLQ retention of **7 days** (604,800 s) is a reasonable baseline and SQS's own recommendation for DLQ configurations. Update via:\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=604800\n```\n\nOr in Terraform/CDK, set `message_retention_seconds = 604800` on the DLQ resource and ensure the value is parameterised relative to the source so the relationship is maintained if the source retention is ever changed.\n\n---\n\n*No further findings. All other rules evaluated clean against the configured values.*\n\n---\n\n### 3. Boundary\n\nThe following questions this audit cannot answer from queue configuration alone. They are not omissions; they are the joins across resources, time, and account that a single `GetQueueAttributes` read cannot provide. A clean answer to R3 does not close them.\n\n**Consumer processing time vs. visibility timeout (R5 territory)**\nThe visibility timeout is 60 s. Whether that is sufficient depends entirely on how long the worker fleet takes to process an order event. If any processing path \u2014 database lookups, downstream HTTP calls, enrichment logic \u2014 routinely exceeds 60 s, the message reappears mid-processing and is delivered to a second worker concurrently. This audit cannot see consumer processing time. Check your CloudWatch `ApproximateAgeOfOldestMessage` and worker execution duration metrics, and extend `VisibilityTimeout` or use `ChangeMessageVisibility` heartbeating if there is any margin.\n\n**Live redrive behaviour**\nThere are currently 37 messages in the DLQ and 88 in the source queue. The audit cannot determine whether the DLQ messages are accumulating (a processing regression), are a stable residue from a past incident, or are being drained. The R3 finding means some messages from the source queue may already have been silently deleted by the DLQ before they could be counted. CloudWatch `NumberOfMessagesSent` and `NumberOfMessagesDeleted` on the DLQ over time is the only way to quantify the actual loss.\n\n**IAM effective permissions**\nThere is no resource policy on `order-events`. The effective set of principals that can send to, receive from, purge, or delete the queue is determined entirely by identity-based IAM policies in account `211125758836`. This audit cannot read those. The access story is half-told: no resource policy does not mean restricted access, it means all access control is in identity policies that are not visible here.\n\n**DLQ consumer and replay path**\nThe DLQ has a `RedriveAllowPolicy` scoped to the source queue ARN, which is correct. But whether anyone is actually consuming the DLQ \u2014 alerting on it, triaging messages, and replaying valid ones \u2014 is a property of the operational setup, not the configuration. A DLQ that fills silently and is never drained provides no recovery guarantee. Confirm there is an alarm on `ApproximateNumberOfMessagesVisible > 0` for `order-events-dlq` and a documented runbook for replay.\n\n**Producer contract**\nWhether the right services are writing to `order-events`, at the expected rate, with the expected message shape, is not visible in queue attributes. A misconfigured producer that stops sending is indistinguishable from a quiet queue at the configuration layer.",
    "elapsed_s": 67.7
  },
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "treatment",
    "trial": 1,
    "score": 7,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      1,
      1,
      1
    ],
    "notes": [
      "Correctly parses the JSON-encoded RedrivePolicy (maxReceiveCount=5, DLQ ARN) and converts all string-typed seconds to human-readable durations.",
      "Correctly identifies R3 as the sole finding: DLQ retention (1 day) shorter than source retention (4 days), causing silent message loss on arrival.",
      "No false positives; all other rules are correctly evaluated as passing, including the absent resource policy and SSE state.",
      "R3 is ranked CRITICAL and headlined first, correctly prioritizing the silent message-loss defect above all other checks.",
      "Explicitly names boundaries the audit cannot cross, including DLQ operational behavior over time and IAM-based access control, matching the ground-truth join.",
      "Consumer processing time and producer behavior are presented as items to verify rather than confirmed bugs.",
      "Top fix is to raise the DLQ MessageRetentionPeriod above the source's (recommends 7 days vs 4-day source), matching the ground-truth recommendation in substance."
    ],
    "agent_output": "# SQS Queue Audit: order-events\n\n---\n\n## Step 1 \u2014 Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345 600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE on |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":5}\"` | DLQ wired; `maxReceiveCount` = 5 |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**Dead-letter queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| `MessageRetentionPeriod` | `\"86400\"` | 86 400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE on |\n\n---\n\n## Step 2 \u2014 Work the arithmetic before judgment\n\n**R3 check \u2014 DLQ retention vs source retention**\n\n```\nSource retention : 345 600 s (4 days)\nDLQ retention   :  86 400 s (1 day)\n86 400 < 345 600  \u2192  DLQ retention is SHORTER than source retention  \u2192  R3 FIRES\n```\n\n**R4 check \u2014 poison messages aging out before reaching the DLQ**\n\n```\nmaxReceiveCount \u00d7 VisibilityTimeout  =  5 \u00d7 60  =  300 s\nMessageRetentionPeriod               =  345 600 s\n300 < 345 600  \u2192  inequality NOT satisfied  \u2192  R4 does NOT fire\n```\n\n**R2 check \u2014 maxReceiveCount band (3\u201310)**\n\n```\nmaxReceiveCount = 5  \u2192  within [3, 10]  \u2192  R2 does NOT fire\n```\n\n**R5 check \u2014 visibility timeout at the 30 s default**\n\n```\nVisibilityTimeout = 60 s  \u2260  30 s  \u2192  R5 does NOT fire\n```\n\n**R6 check \u2014 retention shorter than a plausible outage**\n\n```\nSource retention = 345 600 s (4 days) > 3 600 s (1 hour)  \u2192  R6 does NOT fire\n```\n\n**R7 check \u2014 open resource policy**\n\n```\nPolicy = absent  \u2192  R7 does NOT fire\n```\n\n**R8 check \u2014 encryption at rest**\n\n```\nSqsManagedSseEnabled = true (source)\nSqsManagedSseEnabled = true (DLQ)\n\u2192  R8 does NOT fire\n```\n\n**R9 check \u2014 FIFO dedup**\n\n```\nFifoQueue = absent (standard queue)  \u2192  R9 does NOT fire\n```\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2192 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| **maxReceiveCount** | 5 |\n| **Source retention** | 345 600 s (4 days) |\n| **DLQ retention** | 86 400 s (1 day) |\n| **Visibility timeout** | 60 s |\n| **Encryption** | SQS-managed SSE, both queues |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Rule:** R3  \n**Attributes:**\n- Source `MessageRetentionPeriod`: `345600` (4 days)\n- DLQ `MessageRetentionPeriod`: `86400` (1 day)\n\n**What breaks:**\n\nSQS measures a message's age from its original `SentTimestamp` and does **not** reset that timestamp when the message is moved to the DLQ. A message that is processed and fails repeatedly across several days can arrive in the DLQ already carrying most or all of its 4-day age. With the DLQ set to only 1 day of retention, any message that took longer than 1 day to exhaust its receive count arrives in the DLQ and is deleted almost immediately \u2014 often within seconds. The DLQ will appear correctly wired, non-empty at a glance, and operationally silent about the loss.\n\nThis is the worst-case scenario for this class of bug: it is invisible in the console, it affects exactly the messages that were hardest to process (the ones that failed repeatedly over an extended period), and those are precisely the messages an on-call engineer most needs to inspect during an incident.\n\nWith 37 messages currently sitting in the DLQ (`ApproximateNumberOfMessages: 37`), it is worth checking immediately whether any of those messages are close to or already past their age limit.\n\n**Recommendation:**\n\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source's retention. The standard pattern is source retention plus a meaningful investigation buffer. With a 4-day source:\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=604800   # 7 days\n```\n\nA 7-day DLQ retention against a 4-day source gives a 3-day investigation window even for a message that failed on its last possible attempt at the very end of the source window. In Terraform:\n\n```hcl\nresource \"aws_sqs_queue\" \"order_events_dlq\" {\n  name                      = \"order-events-dlq\"\n  message_retention_seconds = 604800  # 7 days; must exceed source (345600 / 4 days)\n}\n```\n\nTreat the rule as a **hard invariant**: DLQ retention must always be set to a value strictly greater than source retention, and the two values must be reviewed together whenever either is changed.\n\n---\n\n### No further findings\n\nAll other rules pass cleanly against the static configuration:\n\n| Rule | Result | Notes |\n|---|---|---|\n| R1 \u2014 No DLQ | \u2705 Pass | DLQ is wired |\n| R2 \u2014 maxReceiveCount band | \u2705 Pass | 5 is within [3, 10] |\n| R4 \u2014 Poison ages out before DLQ | \u2705 Pass | 5 \u00d7 60 s = 300 s \u226a 345 600 s |\n| R5 \u2014 Default visibility timeout | \u2705 Pass | 60 s, not the 30 s default |\n| R6 \u2014 Short retention | \u2705 Pass | 4-day source retention |\n| R7 \u2014 Open resource policy | \u2705 Pass | No resource policy present |\n| R8 \u2014 Encryption disabled | \u2705 Pass | SSE enabled on both queues |\n| R9 \u2014 FIFO dedup off | \u2705 N/A | Standard queue |\n\n---\n\n### Boundary\n\nThis audit is complete and correct for what the static configuration of one queue can tell you. The following questions cannot be answered from this configuration alone and remain open.\n\n**1. Consumer processing time vs visibility timeout**\n\nThe 60-second visibility timeout is not at the 30-second default, so R5 did not fire. Whether 60 seconds is actually sufficient depends entirely on how long the worker fleet takes to process a single order event. If any worker takes longer than 60 seconds \u2014 including under load, during cold starts, or when a downstream dependency is slow \u2014 the message reappears in the queue and is delivered to a second worker while the first is still processing it. That produces duplicate processing, not message loss, but it is the same class of silent operational problem. This cannot be determined from queue attributes; it requires measurement of consumer processing time or inspection of consumer code.\n\n**2. Live queue behaviour**\n\nThe audit is against static configuration, not behaviour. The following CloudWatch metrics are not visible here and may indicate problems a clean config cannot reveal: `ApproximateAgeOfOldestMessage` (if rising, messages are not being drained fast enough), `NumberOfMessagesMoved` (actual DLQ redrive volume), `NumberOfEmptyReceives` (whether consumers are polling an idle queue at cost). The 88 messages currently in the source queue and 37 in the DLQ are point-in-time snapshots, not trends.\n\n**3. The effective access-control boundary**\n\nNo resource policy is attached to this queue, which means access is governed entirely by IAM identity policies in the account. This audit cannot read those. Any IAM role or user in account `211125758836` with `sqs:SendMessage`, `sqs:ReceiveMessage`, or `sqs:DeleteMessage` on this queue's ARN (or on `*`) has access, and none of that is visible here. The effective principal set is the union of the resource policy (empty) and every identity policy in the account \u2014 only half of which is present in this audit.\n\n**4. Producer behaviour and message contracts**\n\nWhether the right services are writing to `order-events`, whether message bodies conform to the schema the worker fleet expects, and whether any producer is sending poison messages (malformed payloads that will always fail deserialization) is not a queue attribute. A clean redrive config and a correctly-sized DLQ do not prevent a producer sending messages that will always dead-letter.\n\n**5. DLQ drainage and alerting**\n\nThe DLQ currently holds 37 messages. Whether anyone is actively monitoring it, alerting on a non-zero `ApproximateNumberOfMessages`, or draining it through a redrive-to-source process is not captured in queue configuration. A DLQ that fills silently and is never inspected provides no operational value. Confirm that a CloudWatch alarm on `ApproximateNumberOfMessages > 0` is attached to this DLQ and routed to an actionable alert channel.",
    "elapsed_s": 56.9
  },
  {
    "queue": "02-dlq-retention-shorter-than-source",
    "condition": "treatment",
    "trial": 2,
    "score": 7,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      1,
      1,
      1
    ],
    "notes": [
      "Correctly parses the embedded RedrivePolicy JSON, maxReceiveCount=5, and all string-typed seconds into human-readable durations.",
      "Identifies R3 as the sole critical finding: DLQ retention (1 day) is shorter than source retention (4 days), with the correct SentTimestamp-not-reset reasoning.",
      "No false positives; all other rules are correctly marked as passing with accurate reasoning.",
      "R3 is headlined as critical and framed as silent message loss, matching the ground-truth severity ranking.",
      "Explicitly states it cannot determine metrics over time (dropped messages, backlog trends) and cannot see the DLQ's operational owner/consumer, matching the ground-truth boundary.",
      "Consumer processing time, DLQ monitoring, and producer behaviour are all presented as risks to verify rather than confirmed bugs.",
      "Top fix is to raise DLQ MessageRetentionPeriod above the source's, recommending the 14-day maximum (1209600 s), matching the ground-truth recommendation exactly."
    ],
    "agent_output": "# SQS Queue Audit \u2014 `order-events`\n\n---\n\n## Step 0: Parse the attributes\n\n**Source queue**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| `VisibilityTimeout` | `\"60\"` | 60 s |\n| `MessageRetentionPeriod` | `\"345600\"` | 345,600 s (4 days) |\n| `DelaySeconds` | `\"0\"` | 0 s |\n| `RedrivePolicy` | `\"{\\\"deadLetterTargetArn\\\":\\\"arn:aws:sqs:eu-west-1:211125758836:order-events-dlq\\\",\\\"maxReceiveCount\\\":5}\"` | DLQ ARN: `order-events-dlq`; `maxReceiveCount`: 5 |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n| `Policy` | absent | No resource policy |\n| `FifoQueue` | absent | Standard queue |\n\n**DLQ**\n\n| Attribute | Raw | Parsed |\n|---|---|---|\n| `QueueArn` | \u2014 | `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n| `MessageRetentionPeriod` | `\"86400\"` | 86,400 s (1 day) |\n| `SqsManagedSseEnabled` | `\"true\"` | SSE enabled |\n\n---\n\n## Step 1: Apply each rule\n\n### R1 \u2014 No DLQ on a processing queue\n`RedrivePolicy` is present. DLQ is wired. **No finding.**\n\n### R2 \u2014 `maxReceiveCount` outside 3\u201310\n`maxReceiveCount` = 5. Within band. **No finding.**\n\n### R3 \u2014 DLQ retention not longer than source retention\n- Source retention: **345,600 s (4 days)**\n- DLQ retention: **86,400 s (1 day)**\n- Condition to fire: DLQ retention \u2264 source retention \u2192 86,400 \u2264 345,600 \u2192 **true**\n\n**Finding: R3 (critical)**\n\n### R4 \u2014 Poison messages age out before reaching the DLQ\nArithmetic: `maxReceiveCount \u00d7 VisibilityTimeout` vs `MessageRetentionPeriod`\n\n```\n5 \u00d7 60 = 300 s\n300 s vs 345,600 s\n300 < 345,600 \u2192 inequality is NOT satisfied\n```\n\nThe product (300 s) is well within the source retention window (4 days). A poison message can exhaust all five receive attempts and still reach the DLQ with 4 days minus 5 minutes of runway remaining. **No finding.**\n\n### R5 \u2014 Visibility timeout at the 30 s default\n`VisibilityTimeout` = 60 s (not the 30 s default). **No finding.**\n\n### R6 \u2014 Retention shorter than a plausible outage\nSource `MessageRetentionPeriod` = 345,600 s (4 days). Well above the 1-hour threshold. **No finding.**\n\n### R7 \u2014 Resource policy allows a wildcard principal with no condition\n`Policy` is absent. **No finding.**\n\n### R8 \u2014 Server-side encryption at rest disabled\n`SqsManagedSseEnabled` = `\"true\"` on both queues. **No finding.**\n\n### R9 \u2014 FIFO with content-based dedup off\n`FifoQueue` attribute absent; this is a standard queue. **No finding.**\n\n---\n\n## Structured Output\n\n---\n\n### Queue\n\n| Field | Value |\n|---|---|\n| **ARN** | `arn:aws:sqs:eu-west-1:211125758836:order-events` |\n| **Type** | Standard |\n| **DLQ wired** | Yes \u2014 `arn:aws:sqs:eu-west-1:211125758836:order-events-dlq` |\n\n---\n\n### Findings\n\n#### \ud83d\udd34 CRITICAL \u2014 R3: DLQ retention is shorter than source retention\n\n**Attributes**\n- Source `MessageRetentionPeriod`: `345600` s **(4 days)**\n- DLQ `MessageRetentionPeriod`: `86400` s **(1 day)**\n\n**What breaks**\n\nSQS does not reset a message's `SentTimestamp` when it is moved to the DLQ. A message's age is measured from when it was first sent to the source queue, and its remaining lifetime in the DLQ is `DLQ_retention \u2212 message_age_on_arrival`.\n\nConsider a message that fails its fifth (final) receive attempt near the end of its window in the source queue \u2014 for example, after 3 days. When it lands in the DLQ it is already 3 days old. With a DLQ retention of only 1 day, that message is deleted within hours of arrival. The DLQ appears correctly wired; the ApproximateNumberOfMessages counter may even show messages present. The ones you most need to inspect \u2014 late, repeated failures \u2014 are the ones that expire silently on arrival.\n\nIn the current configuration the gap is severe: a message could arrive in the DLQ with up to **3 days of age already accumulated** (source retention 4 days, minus the minimum time needed to exhaust 5 receive attempts at 60 s each, which is negligible), and be deleted the same day.\n\n**Recommendation**\n\nSet the DLQ's `MessageRetentionPeriod` to **strictly greater than** the source queue's retention. The safe rule of thumb is DLQ retention = source retention + a comfortable investigation buffer. A minimum safe value here is anything above 345,600 s; a practical choice is **1,209,600 s (14 days)**, which is the SQS maximum and is the standard recommendation for a DLQ that must reliably hold post-mortem evidence.\n\n```bash\naws sqs set-queue-attributes \\\n  --queue-url https://sqs.eu-west-1.amazonaws.com/211125758836/order-events-dlq \\\n  --attributes MessageRetentionPeriod=1209600\n```\n\n---\n\n### No further findings\n\nAll other rules pass:\n\n| Rule | Result | Notes |\n|---|---|---|\n| R1 | \u2705 Pass | DLQ is wired |\n| R2 | \u2705 Pass | `maxReceiveCount` = 5, within 3\u201310 band |\n| R4 | \u2705 Pass | 5 \u00d7 60 s = 300 s \u226a 345,600 s retention; poison messages reach the DLQ with ample time remaining |\n| R5 | \u2705 Pass | Visibility timeout is 60 s, not the 30 s default |\n| R6 | \u2705 Pass | Source retention = 4 days, above the 1-hour minimum |\n| R7 | \u2705 Pass | No resource policy present |\n| R8 | \u2705 Pass | SQS-managed SSE enabled on both queues |\n| R9 | \u2705 Pass | Standard queue, FIFO dedup contract is not applicable |\n\n---\n\n### Boundary\n\nThe following questions are live risks for this queue that **this audit cannot answer**, because the answers require joins across resources, time-series data, or the systems on either side of the queue.\n\n1. **Consumer processing time vs visibility timeout.** The 60 s visibility timeout passes the \"not at default\" check, but whether it is long enough depends entirely on how long the order-event worker takes. If processing takes longer than 60 s under load, the message reappears and is delivered twice \u2014 a duplicate order event. This requires a CloudWatch `ApproximateAgeOfOldestMessage` trace or a timed sample of consumer execution, neither of which is a queue attribute.\n\n2. **Whether anyone is monitoring or draining the DLQ.** There are currently **37 messages** in `order-events-dlq`. The queue has no consumer configured in this attribute set, and there is no CloudWatch alarm configuration visible here. R3 above makes this urgent: messages in the DLQ right now are accumulating age and will be silently deleted within 1 day of their original send time. Whether an alert fires, and whether a human or automated process is draining and inspecting the DLQ, is outside this audit.\n\n3. **Effective access control (the IAM union).** There is no resource policy on the source queue, so access is governed entirely by identity-based IAM policies on the principals that produce and consume from it. This audit cannot read those policies. The effective set of principals that can send to or receive from `order-events` is invisible here. A resource policy is not mandatory, but the absence of one means there is no queue-side guard; access depends entirely on the IAM posture of every role in the account.\n\n4. **Producer behaviour \u2014 are the right services writing, with the right attributes?** Whether order events are being published by the expected services, at the expected volume, and with valid message bodies and attributes (e.g., correct `MessageGroupId` patterns if this queue is ever converted to FIFO) is a property of the producers, not the queue.\n\n5. **Live behaviour \u2014 is processing keeping up?** There are **88 messages** in flight on the source queue. Whether this represents normal depth or a growing backlog requires a time-series from CloudWatch (`ApproximateNumberOfMessages`, `NumberOfMessagesSent`, `NumberOfMessagesDeleted`). A growing backlog with a 4-day retention window is survivable; the same backlog with the current 1-day DLQ retention (R3) means the oldest failing messages are already at risk.",
    "elapsed_s": 55.2
  },
  {
    "queue": "04-poison-ages-out-before-dlq",
    "condition": "control",
    "trial": 0,
    "score": 5,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      0,
      0,
      1
    ],
    "notes": [
      "Agent correctly parses maxReceiveCount (1000), VisibilityTimeout (900s), and MessageRetentionPeriod (345600s) from the config.",
      "Agent identifies the core defect: 1000 \u00d7 900s = 900000s far exceeds 345600s retention, so poison messages are deleted before reaching the DLQ \u2014 matching R2 and R4 in substance.",
      "No false positives introduced; the additional findings (DLQ visibility timeout, short polling, KMS, FIFO) are soft flags, not assertions of misconfiguration absent from ground truth.",
      "Agent headlines the silent message-loss defect as CRITICAL and leads with it, correctly prioritizing it above all other findings.",
      "Agent does not name any specific thing it cannot determine from config alone (e.g., actual receive cadence or consumer behavior) that matches the ground-truth boundary statement.",
      "Several flags (DLQ visibility timeout, FIFO/idempotency need) are presented as confirmed issues rather than 'verify with consumer/operational context' flags, violating honesty on soft flags.",
      "Top fix correctly recommends lowering maxReceiveCount to 3\u20135 so messages reach the DLQ well within the retention window, matching the ground-truth recommendation in substance."
    ],
    "agent_output": "## SQS Audit: `ledger-reconcile`\n\n---\n\n### CRITICAL \u2014 `maxReceiveCount: 1000`\n\n**What it is:** The redrive policy only moves a message to the DLQ after it has been received and failed **1,000 times**.\n\n**Why it matters:** For a daily reconciliation worker, a poison-pill message (malformed payload, unresolvable reference, etc.) will be retried up to 1,000 times before being quarantined. With a `VisibilityTimeout` of 900 seconds, a single stuck message can consume worker cycles for up to **10,000,000 seconds (~115 days)** before reaching the DLQ \u2014 far longer than the 4-day `MessageRetentionPeriod`, meaning **it will be silently dropped from the source queue before it ever reaches the DLQ**. This is a message loss guarantee, not a safety net.\n\n**Fix:**\n```\nmaxReceiveCount: 3 to 5\n```\nFor a once-daily job, even `3` is generous. Set it low enough that a bad message reaches the DLQ well within the source queue's retention window.\n\n---\n\n### HIGH \u2014 Source queue `MessageRetentionPeriod: 345600` (4 days) is shorter than the DLQ retention (14 days) but interacts lethally with `maxReceiveCount: 1000`\n\n**What it is:** This is a consequence of the above, but worth calling out independently. The source queue retains messages for only **4 days**. With `maxReceiveCount: 1000` and a 900-second visibility timeout, any message that fails consistently will expire in the source queue before exhausting its receive count. SQS then **deletes it without moving it to the DLQ**. The DLQ's 14-day retention is irrelevant if messages never arrive there.\n\n**Fix:** Fix `maxReceiveCount` first (see above). Independently, consider whether 4-day retention on the source is appropriate for a daily job \u2014 if the worker is down for a long weekend, you have margin, but it is tight. 7 days (`604800`) is a safer baseline for a financial reconciliation workload.\n\n---\n\n### HIGH \u2014 `VisibilityTimeout: 900` on the DLQ is operationally misleading\n\n**What it is:** The DLQ has a 900-second (15-minute) visibility timeout. The DLQ for a reconciliation queue should typically be **human-inspected or processed by a separate alerting/replay consumer**, not a fast-polling worker. A 900-second timeout means that if an operator or automated replay consumer reads a message and crashes, it is invisible for 15 minutes before reappearing.\n\nMore importantly, if a replay consumer re-drives messages back to the source queue, the 900-second timeout on the DLQ could cause **duplicate re-drives** if the visibility timeout expires before the consumer confirms deletion.\n\n**Fix:** Align the DLQ visibility timeout with the actual consumption pattern. If it is human/tooling-driven replay, a shorter timeout (e.g., `30`\u2013`60` seconds) is more appropriate so messages surface quickly after a failed inspection attempt. If it is automated, match it to the replay consumer's processing SLA.\n\n---\n\n### MEDIUM \u2014 `ReceiveMessageWaitTimeSeconds: 0` on the DLQ (short polling)\n\n**What it is:** The DLQ uses short polling. This means consumers checking the DLQ will frequently get empty responses even when messages exist (SQS short polling samples only a subset of servers), leading to **false negatives, missed alerts, and wasted API calls**.\n\n**Why it matters for a DLQ specifically:** DLQ monitoring is often the only signal that reconciliation is silently failing. A short-polling consumer or CloudWatch alarm driven by `ApproximateNumberOfMessages` may lag or miss spikes.\n\n**Fix:**\n```\nReceiveMessageWaitTimeSeconds: 20\n```\nEnable long polling on the DLQ, consistent with the source queue. Als

judge_prompt.md 1.4 KB

# Judge prompt template

The runner fills this template per graded output. Reproduced here so a different judge implementation can be swapped in.

```
QUEUE: {id}
NAME: {queue}

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): {expected_codes}  (severity: {top_severity}; clean: {clean})
- What the defect is: {expected_headline}
- Correct top fix: {expected_top_fix}
- Boundary the audit cannot cross: {expected_boundary_join}

AGENT AUDIT OUTPUT:
{agent_output}

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: parses the embedded RedrivePolicy and string-typed seconds correctly.
2. Findings: identifies the ground-truth defect(s) by substance (or correctly reports none for the clean control).
3. No false positives: does not assert a material misconfiguration the ground truth lacks (esp. not calling an aws:SourceArn-scoped wildcard "public").
4. Criticality: ranks silent-message-loss defects as most serious; does not headline a soft flag.
5. Boundary: names at least one thing it cannot determine from config alone, matching the ground-truth join.
6. Honesty on soft flags: presents flags that depend on consumers/data/producers as flags to verify, not confirmed bugs.
7. Recommendation: top fix matches the ground-truth fix in substance.

Return JSON only, no prose:
{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence per item>", ...]
}
```

queues.py 7.6 KB

"""
Per-fixture queue contexts and expected answers, used by run_eval.py.

The "expected_*" fields are the deterministic answers from _audit.py run against
each fixture (see tests/replay_*.py). They are the source of truth the judge model
compares the agent's output against, so they are computed here by importing the
reference implementation rather than hand-copied (which would drift).

Stdlib only. No external dependencies.
"""

from __future__ import annotations

import json
import sys
from pathlib import Path

TESTS_DIR = Path(__file__).resolve().parent.parent
FIXTURES_DIR = TESTS_DIR.parent / "fixtures"

sys.path.insert(0, str(TESTS_DIR))
from _audit import run_audit  # noqa: E402

# Each entry pairs a fixture with the human-readable context the eval feeds the agent,
# plus the rule(s) the deterministic audit produces (the judge's ground truth). Keep this
# list aligned with the replay_*.py files under tests/.
QUEUES = [
    {
        "id": "01-no-dlq",
        "queue": "payments-capture",
        "role": "A worker fleet receives capture events and calls a payment processor.",
        "is_processing_queue": True,
        "expected_headline": "No dead-letter queue on a processing queue; poison messages are retried until retention expiry, then silently deleted.",
        "expected_top_fix": "Attach a RedrivePolicy to a DLQ with maxReceiveCount in the 3-10 band.",
        "expected_boundary_join": "queue to its metrics over time (is a poison message in the queue right now) and queue to its consumers (is this really a processing queue).",
    },
    {
        "id": "02-dlq-retention-shorter-than-source",
        "queue": "order-events",
        "role": "Order events processed by a worker fleet, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "DLQ retention (1 day) is not longer than the source (4 days); because SentTimestamp does not reset on redrive, late failures arrive in the DLQ already past its limit and are deleted on arrival.",
        "expected_top_fix": "Raise the DLQ MessageRetentionPeriod above the source's, ideally to the 14-day maximum.",
        "expected_boundary_join": "DLQ to its metrics over time (how many were dropped) and DLQ to its operational owner.",
    },
    {
        "id": "03-maxreceivecount-too-low",
        "queue": "email-dispatch",
        "role": "Sends transactional email via an upstream provider, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "maxReceiveCount=1 dead-letters good messages on the first transient failure; the DLQ fills with recoverable messages that mask genuine poison.",
        "expected_top_fix": "Raise maxReceiveCount into the 3-10 band so transient failures are retried before quarantine.",
        "expected_boundary_join": "DLQ to its consumers (ratio of transient to poison) and queue to its metrics over time.",
    },
    {
        "id": "04-poison-ages-out-before-dlq",
        "queue": "ledger-reconcile",
        "role": "A daily reconciliation worker, with a dead-letter queue attached and generous retention.",
        "is_processing_queue": True,
        "expected_headline": "maxReceiveCount (1000) x VisibilityTimeout (900s) = 900000s exceeds MessageRetentionPeriod (345600s), so poison messages age out and are deleted before they ever reach the correctly-wired DLQ.",
        "expected_top_fix": "Lower maxReceiveCount or VisibilityTimeout (or raise retention) so maxReceiveCount x VisibilityTimeout stays well under retention.",
        "expected_boundary_join": "queue to its metrics over time and queue to its consumers (real receive cadence).",
    },
    {
        "id": "05-default-visibility-short-retention",
        "queue": "click-events",
        "role": "A high-volume analytics ingest queue, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "Two soft flags: VisibilityTimeout at the 30s default (possible double-delivery) and MessageRetentionPeriod at 300s (a short outage loses messages). Neither is a confirmed bug from config alone.",
        "expected_top_fix": "Raise retention to cover a plausible outage; set VisibilityTimeout deliberately above consumer p99 processing time.",
        "expected_boundary_join": "queue to its consumers (processing-time distribution) and queue to its metrics over time.",
    },
    {
        "id": "06-public-queue-policy",
        "queue": "inbound-webhooks",
        "role": "Intended to receive events from one specific SNS topic, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "Resource policy allows Principal:* for SendMessage with no narrowing Condition (a public queue), and server-side encryption at rest is off.",
        "expected_top_fix": "Add a Condition pinning the principal to the intended aws:SourceArn (or name explicit principal ARNs); enable SSE.",
        "expected_boundary_join": "queue to the account's IAM graph (resource policy is only half the effective access).",
    },
    {
        "id": "07-fifo-dedup-off",
        "queue": "inventory-updates.fifo",
        "role": "A FIFO queue chosen for ordering and exactly-once processing of stock adjustments, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "FIFO queue with ContentBasedDeduplication off: exactly-once now depends entirely on every producer supplying a MessageDeduplicationId, which the queue cannot enforce.",
        "expected_top_fix": "Enable ContentBasedDeduplication, or confirm every producer sets MessageDeduplicationId.",
        "expected_boundary_join": "queue to its producers (do they send the dedup ID).",
    },
    {
        "id": "08-clean-standard",
        "queue": "notification-fanout",
        "role": "Subscribed to an SNS topic, processed by a worker fleet, with a dead-letter queue attached.",
        "is_processing_queue": True,
        "expected_headline": "No findings. The queue is correctly configured, including a wildcard-principal policy that is correctly narrowed by aws:SourceArn (the legitimate SNS-to-SQS pattern).",
        "expected_top_fix": "None. Do not invent a finding. Still report the boundary.",
        "expected_boundary_join": "queue to its metrics over time, queue to the account's IAM graph, DLQ to its owner (a clean config is not a clean system).",
    },
]


def fixture_dir(queue: dict) -> Path:
    return FIXTURES_DIR / queue["id"]


def load_fixture_text(queue: dict) -> str:
    """The raw GetQueueAttributes JSON the agent is given: the source queue and its DLQ."""
    d = fixture_dir(queue)
    parts = []
    q = json.loads((d / "queue.json").read_text())
    parts.append("SOURCE QUEUE (aws sqs get-queue-attributes --attribute-names All):\n" + json.dumps(q, indent=2))
    dlq_path = d / "dlq.json"
    if dlq_path.exists():
        dlq = json.loads(dlq_path.read_text())
        parts.append("DEAD-LETTER QUEUE (aws sqs get-queue-attributes --attribute-names All):\n" + json.dumps(dlq, indent=2))
    return "\n\n".join(parts)


def expected_audit(queue: dict) -> dict:
    """Run the deterministic reference audit to get the ground-truth findings for the judge."""
    audit = run_audit(fixture_dir(queue), is_processing_queue=queue["is_processing_queue"])
    return {
        "codes": sorted(audit.codes()),
        "top_severity": audit.top_severity,
        "clean": audit.clean,
        "boundary_count": len(audit.boundary),
    }


if __name__ == "__main__":
    # `python tests/eval/queues.py` prints the ground-truth answers, no API needed.
    for q in QUEUES:
        exp = expected_audit(q)
        print(f"{q['id']:<40} codes={exp['codes']!s:<18} top={exp['top_severity']} clean={exp['clean']}")

README.md 8.4 KB

# Ablation eval

Does loading `SKILL.md` measurably improve an LLM agent's SQS audit? The replay tests under `tests/replay_*.py` prove the methodology *logic* produces correct findings against fixtures. They do not prove that an agent following the SKILL.md prose *does better* than an agent without it.

This eval answers that question. It runs the same agent in two conditions (control = no skill, treatment = SKILL.md loaded) against the same `GetQueueAttributes` fixtures, scores both against a 7-item rubric ([`rubric.md`](./rubric.md)), and reports the **lift** (= treatment score - control score). A skill is "valuable" when the lift is consistently positive across fixtures.

## Quickstart

```bash
# Install the only non-stdlib dependency
pip install anthropic

# Set your API key
export ANTHROPIC_API_KEY=sk-ant-...

# Smoke test: 1 trial per cell, 3 fixtures
python tests/eval/run_eval.py --trials 1 --fixtures 02,04,08

# Full run: 5 trials per cell, all 8 fixtures (~160 LLM calls, expect 10-30 USD)
python tests/eval/run_eval.py --trials 5
```

The script writes raw per-trial results to `eval_results.json` and prints a per-fixture summary table with aggregate lift and a verdict. `python tests/eval/queues.py` prints the deterministic ground-truth findings for every fixture with no API key required.

### Resumable runs

Every `(fixture, condition, trial)` cell is written to `--output` the moment it completes (atomic temp + rename), and on start the runner loads whatever is already there and skips the cells it finds. A crash, a rate-limit, or a Ctrl-C loses at most the one in-flight cell. To finish an interrupted run, **re-run the exact same command**: it fills only the gaps and reprints the full summary.

```bash
python tests/eval/run_eval.py --trials 5            # crashes after 60 of 80 cells
python tests/eval/run_eval.py --trials 5            # runs only the missing 20
python tests/eval/run_eval.py --trials 5 --force    # ignore prior results, re-run all 80
```

A cell whose agent or judge call raises is not recorded, so it is retried on the next run rather than poisoning the file. Keep `--trials` and `--fixtures` identical across resumes, since the cell identity is `(fixture, condition, trial)`.

## Reference run results

The committed reference run (`eval_results.json`). The setup:

- Model: Claude Sonnet 4.6 (`sonnet`)
- Trials per cell: **N=3**
- Fixtures: the four most diagnostic (02 R3, 04 R4, 05 R5/R6, 08 clean control), each run in both conditions
- Scoring: LLM-as-judge against the 7-item rubric, anchored to the deterministic reference audit (`_audit.py`) as ground truth

Each cell is the mean of three 7-item scores (0-7).

| Fixture | Control (N=3) | Treatment (N=3) | Lift |
|---|---:|---:|---:|
| 02 DLQ retention < source (R3) | 5.33 | 7.00 | +1.67 |
| 04 Poison ages out (R4) | 5.33 | 7.00 | +1.67 |
| 05 Default visibility + short retention (R5/R6) | 3.67 | 7.00 | +3.33 |
| 08 Clean control | 1.33 | 7.00 | +5.67 |
| **Aggregate mean** | **3.92** | **7.00** | **+3.08** |

**Lift: +3.08 / 7 (+44%).** Treatment beats control on all four fixtures (4 positive, 0 zero, 0 negative) and sweeps **7.00 / 7 on every fixture with zero variance** (treatment stdev 0.00 across all cells). Verdict: skill is clearly valuable.

### Where the lift comes from

The per-item breakdown is more informative than the totals:

| Rubric item | What control did | What the skill fixed |
|---|---|---|
| 5. Boundary | **0 of 4** control outputs produced a boundary section. Every cold agent presented a config read as a complete health verdict. | Every treatment output named the consumer / IAM-union / metrics-over-time joins it could not cross. |
| 3. No false positives | On the clean control (08), the cold agent flagged the `aws:SourceArn`-scoped wildcard policy as a HIGH "public queue" (the textbook false positive) and invented a "maxReceiveCount=5 too low" finding. | Treatment returned zero findings and explicitly recognised the narrowed wildcard as the legitimate SNS-to-SQS pattern. |
| 4. Criticality | On 05, control rated 5-minute retention CRITICAL and the 30s default visibility HIGH. | Treatment graded them medium / low. |
| 6. Honesty on soft flags | Control asserted the 30s visibility timeout as an "almost always wrong" HIGH defect. | Treatment presented it as a flag deferred to consumer processing time. |

The clean control (08) shows the largest single lift (+6). That is expected: a cold agent's instinct is to find something, and an unconditioned wildcard principal is the most tempting false positive in the corpus. The skill's value there is teaching the agent when *not* to fire.

### A regression the eval caught, then closed

An earlier scoring pass (N=1) did **not** sweep 7/7: on fixture 05, treatment scored 5/7 because it **over-fired R4 as critical**, reasoning that "under load, wall-clock could push a message past the 300s retention" even though the configured-value arithmetic (`5 x 30 = 150s < 300s`) says R4 does not fire. That cost the no-false-positives and criticality items.

This is exactly the kind of failure pattern the eval exists to surface. It drove one targeted edit to `SKILL.md`:

| Failure pattern | Affected fixture | Edit |
|---|---|---|
| R4 raised speculatively on "under load" reasoning rather than the configured-value inequality | 05 | R4 step now states: fire **only** when `maxReceiveCount x VisibilityTimeout > MessageRetentionPeriod` on the configured values; the product is already a lower bound, and queue depth / receive cadence are behind the boundary, not inputs to the check. |

The N=3 reference run above is **after** this edit, and the regression is closed: fixture 05 treatment now scores 7.00/7 across all three trials. The R4 over-fire does not recur.

### Caveats on these specific numbers

- **N=3 per cell** surfaces variance but is still a small sample. Treatment stdev is 0.00 on every cell (stable); control varies by up to ~1.15, so individual control cells may shift +/-1 on a re-run.
- **Four fixtures, not eight.** The reference run used the four most diagnostic fixtures to keep cost down. The committed harness runs all eight; the omitted four (01 R1, 03 R2, 06 R7/R8, 07 R9) are single-finding cases where the cold agent does comparatively well, so including them would *raise* the control mean and *shrink* the headline lift. The four-fixture number is the harder comparison, not the flattering one.
- **LLM-as-judge.** The judge is itself a model and can be wrong. Spot-check graded outputs to calibrate trust.
- **Sonnet only.** Opus may push control scores higher (more careful reasoning without guidance), reducing the absolute lift. Re-run with the production model for the honest comparison.

## What the eval does

For each fixture, the script runs N trials in two conditions:

- **Control**: the agent is given the raw `GetQueueAttributes` JSON (source queue + DLQ) and a generic "audit this SQS queue for misconfigurations" prompt. It uses whatever it brings from training.
- **Treatment**: the agent is given the same JSON plus the full `SKILL.md` as the methodology to follow.

Each output is graded by an LLM judge against the 7-item rubric. The judge is given the deterministic reference audit (`_audit.py`, via `queues.py`) as ground truth, so grading is anchored to a known-good answer rather than the judge's own opinion.

## Files

| File | Purpose |
|---|---|
| `run_eval.py` | The runner. Calls the API in both conditions, calls the judge, aggregates. Needs `ANTHROPIC_API_KEY`. |
| `queues.py` | Per-fixture contexts and ground-truth findings (computed by importing `_audit.py`, so they never drift). Runs offline. |
| `rubric.md` | The 7-item rubric, one sentence per item. |
| `judge_prompt.md` | The judge prompt template, for swapping in a different judge. |

## When to re-run this

- After any non-trivial edit to `SKILL.md`. The prose is load-bearing; re-run with at least N=3 to catch regressions.
- After adding a worked example. If the lift on the new fixture is near zero, `SKILL.md` probably needs to cover that case.
- Before submitting the skill to a marketplace. A documented lift is a contributor-trust signal.

## What the eval does NOT measure

- **Narrative quality.** A correct finding stated tersely scores the same as one dressed up.
- **Speed / cost.** No wall-clock or token accounting (both conditions get the same budget).
- **Real-world generalization.** The eight fixtures are constructed, not pulled from production. High lift here is a regression guard plus a credibility signal, not proof of operational value.

rubric.md 2.6 KB

# Eval rubric: `sqs-queue-auditor`

Seven binary items (1 = pass, 0 = fail). No partial credit. The judge is given the deterministic reference audit (from `_audit.py`) as ground truth.

1. **Parse.** The agent correctly reads the queue configuration: it parses the embedded `RedrivePolicy` JSON string (identifying whether a DLQ exists and the `maxReceiveCount`), and treats the string-typed second values (`VisibilityTimeout`, `MessageRetentionPeriod`) as numbers. An agent that never opens the embedded JSON, or that misreads "no RedrivePolicy" vs "DLQ present", fails.

2. **Findings.** The agent identifies the misconfiguration(s) the ground truth lists for this fixture (by substance, not by rule code: it need not say "R4", but it must describe the same defect). For the clean control, the agent reports no defect.

3. **No false positives.** The agent does not assert a material misconfiguration that the ground truth does not contain. On the clean control this is the whole game (a wildcard principal narrowed by `aws:SourceArn` must NOT be called public). On other fixtures, inventing extra critical/high defects fails this item.

4. **Criticality.** The agent ranks severity correctly: it identifies the silent-message-loss defects (DLQ-retention-ordering, poison-ages-out) as the most serious, and does not present a low-severity flag as if it were the headline. For fixtures whose only findings are soft flags, the agent correctly treats them as lower-severity.

5. **Boundary.** The agent names at least one thing it cannot determine from the queue configuration alone, matching the ground-truth join (consumer processing time, live metrics over time, the IAM identity-policy union, or producer behaviour). An agent that presents its config audit as a complete health verdict fails.

6. **Honesty on soft flags.** Where the fixture involves a flag that depends on something outside the queue (default visibility timeout, encryption-vs-data-classification, FIFO dedup-vs-producers), the agent presents it as a flag to verify rather than asserting a confirmed bug. For fixtures with no soft flag, this item passes as long as the agent does not overclaim certainty elsewhere.

7. **Recommendation.** The agent's top recommended fix matches the ground-truth fix in substance (e.g. "raise DLQ retention to the maximum", "add an aws:SourceArn condition", "raise maxReceiveCount into a sane band").

A perfect audit scores 7. The control condition (no skill) typically loses points on items 4, 5, and 6: a cold agent finds the obvious defects but tends to present a configuration read as a full health check, misses the silent-loss arithmetic, and states soft flags with unearned certainty.

run_eval.py 11.5 KB

"""
Ablation eval for the sqs-queue-auditor skill.

For each fixture, runs N trials in two conditions:
- Control: the agent is given the raw GetQueueAttributes JSON and a generic "audit this
  SQS queue" prompt. It uses whatever it brings from training.
- Treatment: the agent is given the same JSON plus SKILL.md as the methodology to follow.

Each agent output is graded against the 7-item rubric (rubric.md) by an LLM judge, anchored
to the deterministic reference audit (_audit.py) as ground truth.

Final report: per-fixture mean score (control vs treatment), lift (= treatment - control),
stdev across trials, and a verdict.

Requirements:
- ANTHROPIC_API_KEY environment variable.
- `pip install anthropic` (the only non-stdlib dependency in the repo; isolated to tests/eval/).

Usage:
    python tests/eval/run_eval.py --trials 5
    python tests/eval/run_eval.py --trials 1 --fixtures 02,04,08   # smoke test

Resumable: every (fixture, condition, trial) cell is written to --output the moment it
completes, and on start the runner loads whatever is already in --output and skips the
cells it finds. A crash, a rate-limit, or a Ctrl-C loses at most the one in-flight cell;
re-run the exact same command and it fills only the gaps. Use --force to ignore existing
results and re-run every cell. A cell whose agent or judge call raises is simply not
recorded, so it is retried on the next run.

Cost note: 8 fixtures x 2 conditions x 5 trials, plus a judge call per output, is
~80 agent calls + ~80 judge calls = ~160 LLM calls. Expect $10-30 depending on model
(Sonnet recommended for cost; Opus for highest agent quality).
"""

from __future__ import annotations

import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path

try:
    from anthropic import Anthropic
except ImportError:
    print("ERROR: anthropic SDK not installed. Run: pip install anthropic", file=sys.stderr)
    sys.exit(1)

sys.path.insert(0, str(Path(__file__).parent))
from queues import QUEUES, load_fixture_text, expected_audit  # noqa: E402

REPO_SKILL_MD = Path(__file__).resolve().parent.parent.parent / "SKILL.md"

DEFAULT_AGENT_MODEL = os.environ.get("EVAL_AGENT_MODEL", "claude-sonnet-4-6")
DEFAULT_JUDGE_MODEL = os.environ.get("EVAL_JUDGE_MODEL", "claude-sonnet-4-6")
MAX_TOKENS = 4096


def build_control_prompt(queue: dict) -> str:
    return f"""You are an SRE reviewing an AWS SQS queue configuration for misconfigurations.

Queue: {queue['queue']}
Role: {queue['role']}

Here is the full configuration, exactly as returned by the SQS API:

{load_fixture_text(queue)}

Audit this queue. Identify any misconfiguration that could drop, duplicate, or lose messages, or expose the queue. For each, give the severity and the fix. Be specific."""


def build_treatment_prompt(queue: dict, skill_md_text: str) -> str:
    return f"""You are an SRE auditing an AWS SQS queue, following the methodology below exactly.

METHODOLOGY (SKILL.md):

{skill_md_text}

QUEUE CONTEXT:

Queue: {queue['queue']}
Role: {queue['role']}

CONFIGURATION (GetQueueAttributes output):

{load_fixture_text(queue)}

Apply the methodology end-to-end. Produce the structured output the methodology's "Output format" section prescribes (queue, findings ranked by severity, boundary)."""


JUDGE_SYSTEM = """You are an expert AWS / SRE evaluator grading an SQS queue audit against a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial credit.

You will be given a known-good answer from a deterministic reference audit, the agent's audit output, and the 7 rubric items.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}"""


def build_judge_prompt(queue: dict, agent_output: str) -> str:
    exp = expected_audit(queue)
    return f"""QUEUE: {queue['id']}
NAME: {queue['queue']}

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): {exp['codes']}  (severity: {exp['top_severity']}; clean: {exp['clean']})
- What the defect is: {queue['expected_headline']}
- Correct top fix: {queue['expected_top_fix']}
- Boundary the audit cannot cross: {queue['expected_boundary_join']}

AGENT AUDIT OUTPUT:
{agent_output}

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: parses the embedded RedrivePolicy and string-typed seconds correctly.
2. Findings: identifies the ground-truth defect(s) by substance (or correctly reports none for the clean control).
3. No false positives: does not assert a material misconfiguration the ground truth lacks (esp. not calling an aws:SourceArn-scoped wildcard "public").
4. Criticality: ranks silent-message-loss defects as most serious; does not headline a soft flag.
5. Boundary: names at least one thing it cannot determine from config alone, matching the ground-truth join.
6. Honesty on soft flags: presents flags that depend on consumers/data/producers as flags to verify, not confirmed bugs.
7. Recommendation: top fix matches the ground-truth fix in substance.

Return JSON only."""


def cell_key(result: dict) -> tuple[str, str, int]:
    """Identity of one (fixture, condition, trial) cell, used to skip completed work."""
    return (result["queue"], result["condition"], result["trial"])


def load_existing(path: Path) -> list[dict]:
    """Load prior results from a previous (possibly crashed) run; empty list if none/corrupt."""
    if not path.exists():
        return []
    try:
        data = json.loads(path.read_text())
        return data if isinstance(data, list) else []
    except (json.JSONDecodeError, OSError):
        print(f"  WARN: could not read existing {path}, starting fresh", file=sys.stderr)
        return []


def write_results(path: Path, results: list[dict]) -> None:
    """Atomically rewrite the results file (temp + rename) so a crash never truncates it."""
    tmp = path.with_name(path.name + ".tmp")
    tmp.write_text(json.dumps(results, indent=2))
    tmp.replace(path)


def run_agent(client: Anthropic, model: str, prompt: str) -> str:
    resp = client.messages.create(
        model=model,
        max_tokens=MAX_TOKENS,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(block.text for block in resp.content if block.type == "text")


def run_judge(client: Anthropic, model: str, queue: dict, agent_output: str) -> dict:
    resp = client.messages.create(
        model=model,
        max_tokens=1024,
        system=JUDGE_SYSTEM,
        messages=[{"role": "user", "content": build_judge_prompt(queue, agent_output)}],
    )
    raw = "".join(block.text for block in resp.content if block.type == "text").strip()
    if raw.startswith("```"):
        raw = raw.split("```", 2)[1]
        if raw.startswith("json"):
            raw = raw[4:]
        raw = raw.rsplit("```", 1)[0]
    return json.loads(raw.strip())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--trials", type=int, default=5, help="Trials per (fixture, condition) cell")
    parser.add_argument("--fixtures", default="", help="Comma-separated fixture IDs (prefix match); empty = all")
    parser.add_argument("--agent-model", default=DEFAULT_AGENT_MODEL)
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--output", default="eval_results.json", help="Where to write the raw results")
    parser.add_argument("--force", action="store_true", help="Ignore existing results in --output and re-run every cell")
    args = parser.parse_args()

    if "ANTHROPIC_API_KEY" not in os.environ:
        print("ERROR: ANTHROPIC_API_KEY not set", file=sys.stderr)
        return 1

    client = Anthropic()
    skill_md_text = REPO_SKILL_MD.read_text()

    to_run = QUEUES
    if args.fixtures:
        filters = [f.strip() for f in args.fixtures.split(",")]
        to_run = [q for q in QUEUES if any(q["id"].startswith(f) for f in filters)]

    output_path = Path(args.output)
    results = [] if args.force else load_existing(output_path)
    done = {cell_key(r) for r in results}

    # The full grid of cells this invocation is responsible for.
    planned = [
        (queue, condition, trial)
        for queue in to_run
        for condition in ("control", "treatment")
        for trial in range(args.trials)
    ]
    remaining = [cell for cell in planned if (cell[0]["id"], cell[1], cell[2]) not in done]

    print(f"Grid: {len(to_run)} fixtures x 2 conditions x {args.trials} trials = {len(planned)} cells")
    if done and not args.force:
        print(f"Resuming: {len(planned) - len(remaining)} cell(s) already in {output_path.name}, {len(remaining)} to run")
    print(f"Agent model: {args.agent_model}, Judge model: {args.judge_model}\n")

    for queue, condition, trial in remaining:
        t_start = time.time()
        prompt = build_control_prompt(queue) if condition == "control" else build_treatment_prompt(queue, skill_md_text)
        try:
            agent_output = run_agent(client, args.agent_model, prompt)
            judge_result = run_judge(client, args.judge_model, queue, agent_output)
            score = sum(judge_result["scores"])
        except Exception as e:
            print(f"  ERROR on {queue['id']} {condition} trial {trial}: {e} (will retry on next run)", file=sys.stderr)
            continue
        elapsed = time.time() - t_start
        results.append({
            "queue": queue["id"],
            "condition": condition,
            "trial": trial,
            "score": score,
            "scores_by_item": judge_result["scores"],
            "notes": judge_result.get("notes", []),
            "agent_output": agent_output,
            "elapsed_s": round(elapsed, 1),
        })
        # Persist after every cell so a crash loses at most this one.
        write_results(output_path, results)
        print(f"  {queue['id']:<40} | {condition:9s} | trial {trial} | score {score}/7 | {elapsed:.0f}s")

    print(f"\nRaw results: {output_path}\n")
    print_summary(results, to_run)
    return 0


def print_summary(results: list[dict], to_run: list[dict]) -> None:
    by_cell: dict[tuple[str, str], list[int]] = {}
    for r in results:
        by_cell.setdefault((r["queue"], r["condition"]), []).append(r["score"])

    print(f"{'Fixture':<40} {'Control mean':>14} {'Treatment mean':>16} {'Lift':>8} {'C-std':>7} {'T-std':>7}")
    print("-" * 94)
    lifts = []
    for queue in to_run:
        control = by_cell.get((queue["id"], "control"), [])
        treatment = by_cell.get((queue["id"], "treatment"), [])
        if not control or not treatment:
            continue
        c_mean, t_mean = statistics.mean(control), statistics.mean(treatment)
        lift = t_mean - c_mean
        lifts.append(lift)
        c_std = statistics.stdev(control) if len(control) > 1 else 0.0
        t_std = statistics.stdev(treatment) if len(treatment) > 1 else 0.0
        print(f"{queue['id']:<40} {c_mean:>14.2f} {t_mean:>16.2f} {lift:>+8.2f} {c_std:>7.2f} {t_std:>7.2f}")
    print("-" * 94)
    if lifts:
        aggregate = statistics.mean(lifts)
        positive = sum(1 for l in lifts if l > 0)
        zero = sum(1 for l in lifts if l == 0)
        negative = sum(1 for l in lifts if l < 0)
        print(f"\nAggregate lift: {aggregate:+.2f}/7 across {len(lifts)} fixtures")
        print(f"  Positive lift: {positive}, Zero: {zero}, Negative: {negative}")
        verdict = (
            "Skill is clearly valuable" if aggregate >= 1.0 and positive >= 2 * negative
            else "Skill provides marginal lift" if aggregate >= 0.3
            else "Skill is not clearly adding value; investigate why before shipping"
        )
        print(f"  Verdict: {verdict}")


if __name__ == "__main__":
    sys.exit(main())

README.md 4.2 KB

# Replay tests for `sqs-queue-auditor`

Stdlib-only Python tests that exercise the audit in [`../SKILL.md`](../SKILL.md) against committed fixtures. No external credentials required.

## Running the tests

From the skill directory (`skills/sqs-queue-auditor/`):

```bash
for t in tests/replay_*.py; do python "$t" || exit 1; done
```

Each test prints `PASS` or `FAIL` and exits with the appropriate code. The current suite has 8 tests covering all nine rules (R1-R9), the severity model, and a clean control that asserts no false positives, totalling 48 assertions. Wire them into CI as plain `python` invocations.

## What the tests assert

Each replay test loads the fixtures for one worked example, runs the reference audit (`_audit.py`) against them, and asserts:

- The queue is parsed correctly (ARN, FIFO flag, DLQ presence).
- The expected rule(s) fire, and only those (each example isolates a rule, except where two genuinely co-occur).
- The severity is correct (the critical rules R3 / R4 are asserted critical; the soft flags R5 / R8 / R9 are asserted low).
- The boundary is reported, and names the specific join the example depends on (consumer time, IAM union, producer contract).

A test fails when the audit regresses on any of these. Treat a failed replay test as a regression in `SKILL.md` or in the reference implementation, not a test bug.

## Fixture schema

Each example has its own fixture directory under `../fixtures/<example-slug>/`. Files mirror the real AWS `GetQueueAttributes` response: a single JSON object with an `Attributes` map whose values are **all strings**, and whose compound attributes (`RedrivePolicy`, `Policy`, `RedriveAllowPolicy`) are JSON documents **encoded as strings**.

| File | Required | Purpose |
|---|---|---|
| `queue.json` | yes | The source queue's `GetQueueAttributes` output. |
| `dlq.json` | when the queue has a DLQ | The dead-letter queue's `GetQueueAttributes` output. Needed for the R3 retention-ordering check; its ARN is the source queue's `RedrivePolicy.deadLetterTargetArn`. |

Key attributes the audit reads:

| Attribute | Type (as stored) | Used by |
|---|---|---|
| `QueueArn` | string | identification |
| `VisibilityTimeout` | string seconds | R4, R5 |
| `MessageRetentionPeriod` | string seconds | R3, R4, R6 |
| `RedrivePolicy` | JSON string (`deadLetterTargetArn`, `maxReceiveCount`) | R1, R2, R3, R4 |
| `Policy` | JSON string (IAM policy document) | R7 |
| `SqsManagedSseEnabled` / `KmsMasterKeyId` | string / string | R8 |
| `FifoQueue` / `ContentBasedDeduplication` | string booleans | R9 |

The reference implementation (`_audit.py`) accepts either the full `{"Attributes": {...}}` envelope or a bare attribute map, and treats a missing `dlq.json` as "DLQ attributes not provided" (R3 skipped).

## Adding a new replay test

When you contribute a new worked example to the skill:

1. Drop fixtures under `../fixtures/<example-slug>/` following the schema above.
2. Add `replay_NN_<slug>.py` in this directory, modeled on the existing eight. Use the shared `report` helper in `_replay.py`.
3. Assert the expected findings, the severity, and that the boundary names the relevant join.
4. Run locally, commit, and reference the test in the example's markdown narrative.

A new test that does not exercise a rule, severity, or boundary join the existing tests do not exercise will fail review. The point of the replay corpus is breadth.

## Why stdlib only

Skills get adopted when they run anywhere with zero setup. A `pip install` is an adoption tax. The reference implementation uses only `json`, `pathlib`, `dataclasses`, and `typing`. If a future test requires a third-party dependency (e.g. `boto3` or `pytest`), that's a signal the skill is leaking implementation detail: the audit operates on the `GetQueueAttributes` output, not on a live AWS connection.

## Why the reference implementation is deterministic

`_audit.py` is a deterministic stand-in for what an AI agent does when it follows `SKILL.md`. It exists so the replay tests can assert that the methodology, applied to known fixtures, produces the expected findings. A natural follow-up is to run the same fixtures through an actual LLM agent loaded with `SKILL.md` and assert it produces the same findings and names the same boundary; that work is out of scope for the first reference skill.

replay_01_no_dlq.py 1.5 KB

"""
Replay test for examples/01-no-dlq.md.

Stdlib only. Run with: `python tests/replay_01_no_dlq.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "01-no-dlq"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)

    assertions = [
        (audit.queue_arn.endswith(":payments-capture"), f"unexpected queue arn {audit.queue_arn}"),
        (audit.has_dlq is False, "queue has no RedrivePolicy, has_dlq should be False"),

        # R1 is the finding: no DLQ on a processing queue.
        ("R1" in audit.codes(), f"expected R1 (no DLQ), got {sorted(audit.codes())}"),
        (next(f.severity for f in audit.findings if f.code == "R1") == "high", "R1 should be high severity"),

        # The queue sets a deliberate visibility timeout (120) and the default retention,
        # so R5 / R6 must NOT fire: R1 is the only finding.
        (audit.codes() == {"R1"}, f"expected exactly {{R1}}, got {sorted(audit.codes())}"),

        # The wall is always named.
        (any("consumer" in b.lower() for b in audit.boundary), "boundary must name the consumer join"),
        (len(audit.boundary) >= 4, f"expected >=4 boundary notes, got {len(audit.boundary)}"),
    ]

    return report("replay_01_no_dlq", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_02_dlq_retention.py 1.5 KB

"""
Replay test for examples/02-dlq-retention-shorter-than-source.md.

Stdlib only. Run with: `python tests/replay_02_dlq_retention.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "02-dlq-retention-shorter-than-source"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)
    r3 = next((f for f in audit.findings if f.code == "R3"), None)

    assertions = [
        (audit.has_dlq is True, "queue has a RedrivePolicy, has_dlq should be True"),

        # R3 is the critical finding: DLQ retention (1d) <= source retention (4d).
        (r3 is not None, f"expected R3 (DLQ retention ordering), got {sorted(audit.codes())}"),
        (r3 is not None and r3.severity == "critical", "R3 must be critical"),
        (audit.top_severity == "critical", f"top severity should be critical, got {audit.top_severity}"),

        # maxReceiveCount (5) is in band and 5*60 << retention, so R2 / R4 must NOT fire.
        (audit.codes() == {"R3"}, f"expected exactly {{R3}}, got {sorted(audit.codes())}"),

        # The detail must call out the non-resetting SentTimestamp, the crux of the bug.
        (r3 is not None and "senttimestamp" in r3.detail.lower(), "R3 detail must explain the non-resetting SentTimestamp"),
    ]

    return report("replay_02_dlq_retention", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_03_maxreceivecount.py 1.3 KB

"""
Replay test for examples/03-maxreceivecount-too-low.md.

Stdlib only. Run with: `python tests/replay_03_maxreceivecount.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "03-maxreceivecount-too-low"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)
    r2 = next((f for f in audit.findings if f.code == "R2"), None)

    assertions = [
        (audit.has_dlq is True, "queue has a DLQ, has_dlq should be True"),

        # R2 fires for maxReceiveCount=1: transient failures dead-letter good messages.
        (r2 is not None, f"expected R2 (maxReceiveCount band), got {sorted(audit.codes())}"),
        (r2 is not None and r2.severity == "medium", "R2-low should be medium severity"),
        (r2 is not None and "1" in r2.title, "R2 title should name the offending maxReceiveCount"),

        # DLQ retention (14d) > source (4d) so no R3; 1*60 << retention so no R4.
        (audit.codes() == {"R2"}, f"expected exactly {{R2}}, got {sorted(audit.codes())}"),
    ]

    return report("replay_03_maxreceivecount", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_04_poison_ages_out.py 1.6 KB

"""
Replay test for examples/04-poison-ages-out-before-dlq.md.

Stdlib only. Run with: `python tests/replay_04_poison_ages_out.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "04-poison-ages-out-before-dlq"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)
    r4 = next((f for f in audit.findings if f.code == "R4"), None)

    assertions = [
        (audit.has_dlq is True, "queue has a DLQ wired, has_dlq should be True"),

        # R4 is the flagship critical finding: 1000 * 900s = 900000s > 345600s retention,
        # so poison messages age out before they ever reach the (correctly-wired) DLQ.
        (r4 is not None, f"expected R4 (poison ages out), got {sorted(audit.codes())}"),
        (r4 is not None and r4.severity == "critical", "R4 must be critical"),
        (r4 is not None and "900000" in r4.detail, "R4 detail must show the worst-case 900000s figure"),
        (audit.top_severity == "critical", f"top severity should be critical, got {audit.top_severity}"),

        # maxReceiveCount=1000 also trips R2-high (low). DLQ retention (14d) > source so no R3.
        ("R2" in audit.codes(), "maxReceiveCount=1000 should also raise R2-high"),
        (audit.codes() == {"R4", "R2"}, f"expected exactly {{R4, R2}}, got {sorted(audit.codes())}"),
    ]

    return report("replay_04_poison_ages_out", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_05_default_visibility.py 1.8 KB

"""
Replay test for examples/05-default-visibility-short-retention.md.

Stdlib only. Run with: `python tests/replay_05_default_visibility.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "05-default-visibility-short-retention"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)

    assertions = [
        # Two soft flags fire together: R5 (default 30s visibility) and R6 (300s retention).
        ("R5" in audit.codes(), f"expected R5 (default visibility), got {sorted(audit.codes())}"),
        ("R6" in audit.codes(), f"expected R6 (short retention), got {sorted(audit.codes())}"),
        (audit.codes() == {"R5", "R6"}, f"expected exactly {{R5, R6}}, got {sorted(audit.codes())}"),

        # Neither is critical: these are flags to verify, not confirmed message-loss bugs.
        # 30s * 5 = 150s < 300s retention so R4 must NOT fire.
        (audit.top_severity == "medium", f"top severity should be medium (R6), got {audit.top_severity}"),
        (all(f.severity != "critical" for f in audit.findings), "no critical findings expected here"),

        # R5 must honestly defer to the consumer-processing-time boundary, not assert a bug.
        (next(f.severity for f in audit.findings if f.code == "R5") == "low", "R5 is a low-severity risk flag"),
        (any("processing time" in b.lower() or "consumer" in b.lower() for b in audit.boundary),
         "boundary must name the consumer-processing-time join R5 depends on"),
    ]

    return report("replay_05_default_visibility", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_06_public_policy.py 1.4 KB

"""
Replay test for examples/06-public-queue-policy.md.

Stdlib only. Run with: `python tests/replay_06_public_policy.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "06-public-queue-policy"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)
    r7 = next((f for f in audit.findings if f.code == "R7"), None)

    assertions = [
        # R7: wildcard principal with no narrowing condition (a public queue).
        (r7 is not None, f"expected R7 (open resource policy), got {sorted(audit.codes())}"),
        (r7 is not None and r7.severity == "high", "R7 should be high severity"),

        # R8 also fires: SqsManagedSseEnabled is false and no KMS key.
        ("R8" in audit.codes(), f"expected R8 (encryption off), got {sorted(audit.codes())}"),
        (audit.codes() == {"R7", "R8"}, f"expected exactly {{R7, R8}}, got {sorted(audit.codes())}"),

        # The audit must name the IAM-union boundary: the resource policy is only half the story.
        (any("iam" in b.lower() for b in audit.boundary), "boundary must name the IAM identity-policy union"),
    ]

    return report("replay_06_public_policy", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_07_fifo_dedup_off.py 1.4 KB

"""
Replay test for examples/07-fifo-dedup-off.md.

Stdlib only. Run with: `python tests/replay_07_fifo_dedup_off.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "07-fifo-dedup-off"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)
    r9 = next((f for f in audit.findings if f.code == "R9"), None)

    assertions = [
        (audit.is_fifo is True, "queue ends in .fifo, is_fifo should be True"),

        # R9: FIFO with content-based dedup off depends on producers sending dedup IDs.
        (r9 is not None, f"expected R9 (FIFO dedup contract), got {sorted(audit.codes())}"),
        (r9 is not None and r9.severity == "low", "R9 is a low-severity contract flag"),
        (audit.codes() == {"R9"}, f"expected exactly {{R9}}, got {sorted(audit.codes())}"),

        # The finding must defer the actual verification to the producer side (the wall).
        (r9 is not None and "producer" in r9.detail.lower(), "R9 detail must name the producer dependency"),
        (any("producer" in b.lower() for b in audit.boundary), "boundary must name the producer join"),
    ]

    return report("replay_07_fifo_dedup_off", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

replay_08_clean_standard.py 1.6 KB

"""
Replay test for examples/08-clean-standard.md.

The control case: a correctly-configured queue must produce zero findings. This
guards against false positives, which are how an auditor loses an operator's trust.

Stdlib only. Run with: `python tests/replay_08_clean_standard.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _audit import run_audit  # noqa: E402
from _replay import report  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "08-clean-standard"


def main() -> int:
    audit = run_audit(FIXTURE_DIR, is_processing_queue=True)

    assertions = [
        (audit.has_dlq is True, "queue has a well-sized DLQ"),

        # Zero findings: deliberate visibility (180s), default retention, DLQ at 14d,
        # maxReceiveCount 5, SSE on, and a resource policy scoped by aws:SourceArn.
        (audit.clean is True, f"clean queue should produce no findings, got {sorted(audit.codes())}"),
        (audit.top_severity is None, f"clean queue should have no top severity, got {audit.top_severity}"),

        # The wildcard-principal policy here is narrowed by aws:SourceArn, so R7 must NOT fire.
        ("R7" not in audit.codes(), "an aws:SourceArn-conditioned policy must not trip R7"),

        # Even a clean queue still reports the boundary: a clean config is not a clean system.
        (len(audit.boundary) >= 4, f"expected >=4 boundary notes even when clean, got {len(audit.boundary)}"),
    ]

    return report("replay_08_clean_standard", audit, assertions)


if __name__ == "__main__":
    sys.exit(main())

_audit.py 21.8 KB

"""
Reference implementation of the sqs-queue-auditor methodology.

This module is a deterministic stand-in for what an AI agent does when it
follows SKILL.md. It exists so replay tests can assert that the methodology,
applied to known `GetQueueAttributes` fixtures, produces the expected findings
and the expected boundary (the questions the queue config alone cannot answer).

Input shape mirrors the real AWS SQS API: `GetQueueAttributes` returns every
attribute as a string, and the compound attributes (`RedrivePolicy`, `Policy`,
`RedriveAllowPolicy`) are JSON documents encoded *inside* those strings. Parsing
that correctly is part of the judgment this skill encodes: a naive read treats
`MessageRetentionPeriod` as already-numeric and never opens the embedded
RedrivePolicy at all.

Stdlib only. No external dependencies. No external credentials. Runs anywhere
Python 3.10+ runs.
"""

from __future__ import annotations

import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any

# AWS defaults and limits (seconds).
DEFAULT_VISIBILITY_TIMEOUT = 30
DEFAULT_RETENTION = 345_600          # 4 days, the SQS default
MAX_RETENTION = 1_209_600            # 14 days, the SQS maximum
SHORT_RETENTION_THRESHOLD = 3_600    # below 1h, a brief consumer outage loses data

# maxReceiveCount sane band. Below: a single transient failure dead-letters good
# messages. Above: poison messages are retried many times before quarantine, which
# delays detection and (with a large visibility timeout) feeds the R4 age-out bug.
MAXRECEIVE_MIN_SANE = 3
MAXRECEIVE_MAX_SANE = 10

_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3}

# Condition keys that narrow an otherwise-open principal to something legitimate
# (a specific source service, account, or org). Their presence turns a `Principal:*`
# statement from "the public internet" into "this SNS topic / this account".
_NARROWING_CONDITION_KEYS = (
    "aws:SourceArn",
    "aws:SourceAccount",
    "aws:PrincipalOrgID",
    "aws:PrincipalAccount",
    "aws:SourceOwner",
)


@dataclass
class Finding:
    """One queue-side misconfiguration, derived from the static attributes alone."""

    code: str          # R1..R9
    severity: str      # critical | high | medium | low
    attribute: str     # the attribute(s) the finding is grounded in
    title: str
    detail: str
    recommendation: str


@dataclass
class Audit:
    """Structured output of the methodology, one per queue."""

    queue_arn: str
    is_fifo: bool = False
    has_dlq: bool = False
    findings: list[Finding] = field(default_factory=list)
    # The wall: questions a single queue's config cannot answer. Each item names a
    # join (across resources, across sources, or across time) the audit cannot make.
    boundary: list[str] = field(default_factory=list)

    @property
    def clean(self) -> bool:
        return len(self.findings) == 0

    @property
    def top_severity(self) -> str | None:
        if not self.findings:
            return None
        return min(self.findings, key=lambda f: _SEVERITY_RANK[f.severity]).severity

    def codes(self) -> set[str]:
        return {f.code for f in self.findings}


def load_attributes(path: Path) -> dict[str, str]:
    """Load a GetQueueAttributes fixture. Returns the `Attributes` string map."""
    with path.open() as f:
        doc = json.load(f)
    # Accept either the raw API envelope ({"Attributes": {...}}) or a bare map.
    return doc.get("Attributes", doc)


def _int(attrs: dict[str, str], key: str, default: int | None = None) -> int | None:
    """SQS returns every attribute as a string. Parse one as an int."""
    raw = attrs.get(key)
    if raw is None:
        return default
    try:
        return int(raw)
    except (TypeError, ValueError):
        return default


def _json_attr(attrs: dict[str, str], key: str) -> Any:
    """Parse a compound attribute whose value is a JSON document encoded as a string."""
    raw = attrs.get(key)
    if not raw:
        return None
    if isinstance(raw, (dict, list)):
        return raw
    try:
        return json.loads(raw)
    except (TypeError, ValueError):
        return None


def parse_redrive_policy(attrs: dict[str, str]) -> tuple[str | None, int | None]:
    """Return (deadLetterTargetArn, maxReceiveCount) from the RedrivePolicy, or (None, None)."""
    policy = _json_attr(attrs, "RedrivePolicy")
    if not isinstance(policy, dict):
        return (None, None)
    arn = policy.get("deadLetterTargetArn")
    raw_count = policy.get("maxReceiveCount")
    try:
        count = int(raw_count) if raw_count is not None else None
    except (TypeError, ValueError):
        count = None
    return (arn, count)


def _principal_is_wildcard(principal: Any) -> bool:
    """True if the statement principal includes the `*` wildcard (anyone)."""
    if principal == "*":
        return True
    if isinstance(principal, dict):
        for value in principal.values():
            if value == "*":
                return True
            if isinstance(value, list) and "*" in value:
                return True
    return False


def _statement_is_narrowed(statement: dict) -> bool:
    """True if a Condition block scopes the statement to a specific source/account/org."""
    condition = statement.get("Condition")
    if not isinstance(condition, dict):
        return False
    for operator_block in condition.values():
        if not isinstance(operator_block, dict):
            continue
        for key in operator_block:
            if key in _NARROWING_CONDITION_KEYS:
                return True
    return False


# --- The checks. Each maps to one rule code documented in SKILL.md / FAILURE_MODES.md. ---


def check_redrive_wiring(
    attrs: dict[str, str],
    dlq_attrs: dict[str, str] | None,
    is_processing_queue: bool,
) -> tuple[list[Finding], bool]:
    """R1 (no DLQ), R2 (maxReceiveCount band), R3 (DLQ retention ordering)."""
    findings: list[Finding] = []
    dlq_arn, max_receive = parse_redrive_policy(attrs)
    has_dlq = dlq_arn is not None

    if not has_dlq:
        if is_processing_queue:
            findings.append(Finding(
                code="R1",
                severity="high",
                attribute="RedrivePolicy",
                title="No dead-letter queue on a processing queue",
                detail=(
                    "The queue has no RedrivePolicy. A message the consumer can never "
                    "process (a poison message) is received, fails, becomes visible "
                    "again, and is retried until MessageRetentionPeriod expires, at which "
                    "point SQS deletes it silently. There is no quarantine and no signal: "
                    "the message is simply gone, and the only evidence is a consumer that "
                    "burned cycles on it for the whole retention window."
                ),
                recommendation=(
                    "Attach a RedrivePolicy pointing at a dead-letter queue with a "
                    "maxReceiveCount in the 3-10 band, so poison messages are quarantined "
                    "for inspection instead of dropped."
                ),
            ))
        return (findings, has_dlq)

    # DLQ is present: validate maxReceiveCount and retention ordering.
    if max_receive is not None:
        if max_receive < MAXRECEIVE_MIN_SANE:
            findings.append(Finding(
                code="R2",
                severity="medium",
                attribute="RedrivePolicy.maxReceiveCount",
                title=f"maxReceiveCount is {max_receive}: transient failures dead-letter good messages",
                detail=(
                    f"With maxReceiveCount={max_receive}, a message that fails "
                    f"{max_receive} delivery attempt(s) goes to the DLQ. A brief, "
                    "recoverable downstream blip (a rolling deploy, a 2-second timeout) "
                    "is enough to send a perfectly good message to the dead-letter queue, "
                    "where it sits unprocessed. The DLQ fills with messages that were never "
                    "poison, masking the ones that are."
                ),
                recommendation="Raise maxReceiveCount into the 3-10 band so transient failures are retried before quarantine.",
            ))
        elif max_receive > MAXRECEIVE_MAX_SANE:
            findings.append(Finding(
                code="R2",
                severity="low",
                attribute="RedrivePolicy.maxReceiveCount",
                title=f"maxReceiveCount is {max_receive}: poison messages are retried before quarantine",
                detail=(
                    f"maxReceiveCount={max_receive} means a poison message is delivered "
                    f"up to {max_receive} times before reaching the DLQ. Detection of a "
                    "genuinely broken message is delayed by that many cycles, and consumer "
                    "capacity is spent reprocessing it each time. Combined with a long "
                    "visibility timeout this also feeds the age-out failure (R4)."
                ),
                recommendation="Lower maxReceiveCount into the 3-10 band unless a specific replay requirement justifies more.",
            ))

    if dlq_attrs is not None:
        source_retention = _int(attrs, "MessageRetentionPeriod", DEFAULT_RETENTION) or DEFAULT_RETENTION
        dlq_retention = _int(dlq_attrs, "MessageRetentionPeriod", DEFAULT_RETENTION) or DEFAULT_RETENTION
        if dlq_retention <= source_retention:
            findings.append(Finding(
                code="R3",
                severity="critical",
                attribute="MessageRetentionPeriod (source vs DLQ)",
                title="DLQ retention is not longer than the source: redriven messages can be deleted on arrival",
                detail=(
                    f"Source MessageRetentionPeriod is {source_retention}s; the DLQ's is "
                    f"{dlq_retention}s. A message's age is measured from its original "
                    "SentTimestamp, and SQS does not reset that timestamp when the message "
                    "is moved to the DLQ. A message that fails late in the source queue's "
                    "retention window therefore arrives in the DLQ already near (or past) "
                    f"the DLQ's {dlq_retention}s limit, and is deleted almost immediately. "
                    "The dead-letter queue looks correctly wired, yet the messages you most "
                    "need to inspect are the ones it silently drops."
                ),
                recommendation=(
                    f"Set the DLQ's MessageRetentionPeriod above the source's, ideally to "
                    f"the maximum ({MAX_RETENTION}s / 14 days), so failed messages survive "
                    "long enough to investigate and redrive."
                ),
            ))

    return (findings, has_dlq)


def check_lifecycle_timing(attrs: dict[str, str]) -> list[Finding]:
    """R4 (poison ages out before DLQ), R5 (default visibility timeout), R6 (short retention)."""
    findings: list[Finding] = []
    visibility = _int(attrs, "VisibilityTimeout", DEFAULT_VISIBILITY_TIMEOUT) or DEFAULT_VISIBILITY_TIMEOUT
    retention = _int(attrs, "MessageRetentionPeriod", DEFAULT_RETENTION) or DEFAULT_RETENTION
    _, max_receive = parse_redrive_policy(attrs)

    # R4: worst-case time for a poison message to exhaust its receive count.
    # Each failed delivery holds the message invisible for `visibility` seconds, so a
    # message needs at least maxReceiveCount * visibility seconds of wall-clock to reach
    # the DLQ. If that exceeds retention, the message ages out and is deleted *before*
    # it ever dead-letters: the DLQ is configured but unreachable for slow failures.
    if max_receive is not None and max_receive > 0:
        worst_case = max_receive * visibility
        if worst_case > retention:
            findings.append(Finding(
                code="R4",
                severity="critical",
                attribute="VisibilityTimeout x maxReceiveCount vs MessageRetentionPeriod",
                title="Poison messages age out of the source queue before reaching the DLQ",
                detail=(
                    f"maxReceiveCount={max_receive} and VisibilityTimeout={visibility}s "
                    f"means a poison message needs at least {worst_case}s to exhaust its "
                    f"receive count, but MessageRetentionPeriod is only {retention}s. "
                    "Retention wins: the message is deleted by age before it is ever moved "
                    "to the dead-letter queue. The DLQ exists and looks correct, yet the "
                    "exact messages it was built to catch never arrive in it."
                ),
                recommendation=(
                    "Lower maxReceiveCount or VisibilityTimeout (or raise "
                    "MessageRetentionPeriod) so that maxReceiveCount x VisibilityTimeout "
                    "stays well under the retention window."
                ),
            ))

    # R5: visibility timeout left at the 30s default. This is a risk flag, not a proven
    # bug: whether 30s is too short depends on consumer processing time, which is not a
    # queue attribute (see boundary). Surfaced as low severity for that reason.
    if visibility == DEFAULT_VISIBILITY_TIMEOUT:
        findings.append(Finding(
            code="R5",
            severity="low",
            attribute="VisibilityTimeout",
            title="VisibilityTimeout is at the 30s default",
            detail=(
                "VisibilityTimeout is 30s, the AWS default, which is frequently left "
                "unchanged rather than chosen. If any consumer takes longer than 30s to "
                "process a message, the message becomes visible again mid-processing and "
                "is delivered to a second consumer, causing duplicate work. Whether that "
                "actually happens depends on consumer processing time, which this audit "
                "cannot see (see boundary): this is a flag to verify, not a confirmed bug."
            ),
            recommendation="Set VisibilityTimeout deliberately, sized above the consumer's p99 processing time (commonly 6x a Lambda timeout).",
        ))

    # R6: retention so short a brief outage loses data.
    if retention < SHORT_RETENTION_THRESHOLD:
        findings.append(Finding(
            code="R6",
            severity="medium",
            attribute="MessageRetentionPeriod",
            title=f"MessageRetentionPeriod is {retention}s: a short consumer outage loses messages",
            detail=(
                f"Messages are deleted after {retention}s whether or not they were "
                "processed. A consumer outage, a deploy, or a scaling lag longer than "
                f"{retention}s silently drops every message still in the queue. The "
                "default is 4 days for a reason: it absorbs ordinary operational gaps."
            ),
            recommendation="Raise MessageRetentionPeriod to cover the longest plausible consumer outage, typically at least the 4-day default.",
        ))

    return findings


def check_exposure(attrs: dict[str, str]) -> list[Finding]:
    """R7 (open resource policy), R8 (encryption at rest disabled)."""
    findings: list[Finding] = []

    policy = _json_attr(attrs, "Policy")
    if isinstance(policy, dict):
        statements = policy.get("Statement", [])
        if isinstance(statements, dict):
            statements = [statements]
        for stmt in statements:
            if not isinstance(stmt, dict):
                continue
            if stmt.get("Effect") != "Allow":
                continue
            if _principal_is_wildcard(stmt.get("Principal")) and not _statement_is_narrowed(stmt):
                findings.append(Finding(
                    code="R7",
                    severity="high",
                    attribute="Policy",
                    title="Queue resource policy allows a wildcard principal with no narrowing condition",
                    detail=(
                        "A statement grants access to Principal \"*\" with no "
                        "aws:SourceArn / aws:SourceAccount / aws:PrincipalOrgID condition. "
                        "As written, the resource policy authorises any AWS principal to act "
                        "on this queue. This is the classic confused-deputy and public-queue "
                        "exposure: a service that should only accept messages from one SNS "
                        "topic accepts them from anyone."
                    ),
                    recommendation=(
                        "Add a Condition that pins the principal to the intended source "
                        "(aws:SourceArn for an SNS topic / S3 bucket, or aws:SourceAccount "
                        "/ aws:PrincipalOrgID for an account or org), or name explicit "
                        "principal ARNs instead of \"*\"."
                    ),
                ))
                break  # one R7 per queue is enough

    sse_managed = (attrs.get("SqsManagedSseEnabled", "false") or "false").lower() == "true"
    has_kms = bool(attrs.get("KmsMasterKeyId"))
    if not sse_managed and not has_kms:
        findings.append(Finding(
            code="R8",
            severity="low",
            attribute="SqsManagedSseEnabled / KmsMasterKeyId",
            title="Server-side encryption at rest is disabled",
            detail=(
                "Neither SQS-managed SSE nor a KMS key is configured, so message bodies "
                "are not encrypted at rest. Whether that matters depends on what the queue "
                "carries, which this audit cannot determine (see boundary): flagged as a "
                "low-severity default worth confirming against the data classification."
            ),
            recommendation="Enable SQS-managed SSE (SqsManagedSseEnabled) or a KMS key unless the data is confirmed non-sensitive.",
        ))

    return findings


def check_fifo_invariants(attrs: dict[str, str]) -> list[Finding]:
    """R9 (FIFO deduplication requires producer cooperation when content-based dedup is off)."""
    findings: list[Finding] = []
    is_fifo = (attrs.get("FifoQueue", "false") or "false").lower() == "true"
    if not is_fifo:
        return findings
    content_dedup = (attrs.get("ContentBasedDeduplication", "false") or "false").lower() == "true"
    if not content_dedup:
        findings.append(Finding(
            code="R9",
            severity="low",
            attribute="ContentBasedDeduplication",
            title="FIFO queue with content-based deduplication off requires producer-supplied dedup IDs",
            detail=(
                "ContentBasedDeduplication is off on a FIFO queue, so SQS will not derive "
                "a deduplication ID from the message body. Every producer must supply an "
                "explicit MessageDeduplicationId, or duplicate sends within the 5-minute "
                "dedup window are accepted as distinct messages. Whether the producers "
                "actually send that ID is a property of the producers, not of this queue "
                "(see boundary): flagged so the contract is verified rather than assumed."
            ),
            recommendation="Either enable ContentBasedDeduplication, or confirm every producer sets MessageDeduplicationId.",
        ))
    return findings


def _boundary_notes(attrs: dict[str, str], has_dlq: bool) -> list[str]:
    """The wall. Every audit names what the static config cannot answer."""
    notes = [
        "Consumer processing time is not a queue attribute. Whether VisibilityTimeout is "
        "actually long enough to avoid double-delivery (R5) needs the consumer's runtime "
        "behaviour, which lives outside SQS. Join: queue to its consumers.",
        "Live behaviour (redrive volume, ApproximateAgeOfOldestMessage, in-flight count, "
        "empty-receive rate) is CloudWatch time-series, not static attributes. This audit "
        "reads the configuration, not what the queue is doing right now. Join: queue to its "
        "metrics over time.",
        "The effective set of principals that can SendMessage / ReceiveMessage is the union "
        "of this resource Policy and every IAM identity policy in the account. Only the "
        "resource policy is visible here (R7). Join: queue to the account's IAM graph.",
        "Whether the producers writing to this queue are the intended ones, and whether "
        "anything is draining the DLQ at all, needs the producer and consumer inventory. "
        "Join: queue to the services on either side of it.",
    ]
    if has_dlq:
        notes.append(
            "A DLQ with messages in it is only useful if something inspects and redrives "
            "them. This audit confirms the DLQ is wired and sized correctly; it cannot "
            "confirm anyone is watching it. Join: DLQ to its operational owner."
        )
    return notes


def run_audit(
    fixture_dir: Path,
    is_processing_queue: bool = True,
) -> Audit:
    """End-to-end: load the queue (and its DLQ if present), run all checks, return the Audit.

    `is_processing_queue` tells the audit whether a missing DLQ is a finding (R1). A pure
    buffer/fan-out queue with an at-least-once contract elsewhere may legitimately have no
    DLQ; the caller asserts the queue's role rather than the audit guessing it.
    """
    attrs = load_attributes(fixture_dir / "queue.json")
    dlq_path = fixture_dir / "dlq.json"
    dlq_attrs = load_attributes(dlq_path) if dlq_path.exists() else None

    findings: list[Finding] = []
    redrive_findings, has_dlq = check_redrive_wiring(attrs, dlq_attrs, is_processing_queue)
    findings += redrive_findings
    findings += check_lifecycle_timing(attrs)
    findings += check_exposure(attrs)
    findings += check_fifo_invariants(attrs)

    findings.sort(key=lambda f: (_SEVERITY_RANK[f.severity], f.code))

    return Audit(
        queue_arn=attrs.get("QueueArn", ""),
        is_fifo=(attrs.get("FifoQueue", "false") or "false").lower() == "true",
        has_dlq=has_dlq,
        findings=findings,
        boundary=_boundary_notes(attrs, has_dlq),
    )

_replay.py 720 B

"""
Shared reporting helper for the replay tests. Stdlib only.

Each replay_NN_*.py loads one fixture, runs the audit, and hands a list of
(ok, message) assertion tuples to `report`. Keeps the per-test files focused on
the assertions that matter for that fixture.
"""

from __future__ import annotations


def report(name: str, audit, assertions) -> int:
    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print(f"FAIL: {name}")
        for msg in failed:
            print(f"  - {msg}")
        return 1
    print(f"PASS: {name} ({len(assertions)} assertions)")
    codes = sorted(audit.codes()) or ["none"]
    print(f"  findings: {codes} (top severity: {audit.top_severity})")
    return 0

FAILURE_MODES.md 6.6 KB

# Failure modes: `sqs-queue-auditor`

This skill is wrong in predictable ways. The list below is the reason it ships with a quality bar that mandates fixture-based replay tests: every failure mode here is a regression vector and gets a test once it shows up in the wild.

## The defining limit: configuration, not behaviour

This skill reads one queue's static attributes. It does not read what the queue is doing. A queue can pass every check in `SKILL.md` and still be dropping messages right now for a reason only the live telemetry shows: a consumer that crashes on a specific payload, a producer that stopped sending, a redrive that is firing constantly. **A clean audit means the configuration is sound, not that the system is healthy.** Every audit says this in its boundary section. Read it as load-bearing, not boilerplate.

## Methodology-level failure modes

### F1. The "is this a processing queue" judgment is supplied, not derived

R1 (no DLQ) only fires when the caller asserts the queue is a processing queue. A pure buffer or fan-out queue with an at-least-once contract enforced elsewhere may legitimately have no dead-letter queue. The audit cannot tell a processing queue from a buffer from the attributes alone; the queue's role is a property of the architecture around it.

**Mitigation in the methodology**: the caller passes `is_processing_queue`. When the role is unknown, the audit should be run as `is_processing_queue=True` (the safe default that surfaces the missing DLQ) and the finding read as "confirm this queue's role" rather than "this is definitely wrong".

**Escalation rule**: if the queue's role is genuinely unknown, surface R1 as a question, not a verdict.

### F2. The R4 age-out estimate is a lower bound, not a guarantee

R4 computes `maxReceiveCount x VisibilityTimeout` as the wall-clock a poison message needs to reach the DLQ, and flags when that exceeds retention. The product is a *lower* bound: it assumes a consumer receives the message roughly once per visibility window. If consumers poll less often, the real time-to-DLQ is longer, which makes the age-out worse, not better. The direction is safe (the flag never under-warns), but the exact margin depends on receive cadence, which is behind the boundary.

**Mitigation in the methodology**: R4 is reported as "poison messages can age out", with the arithmetic shown, so the operator can judge the margin against their actual consumer cadence.

### F3. R3 assumes the DLQ retention is the binding constraint

R3 flags `dlq_retention <= source_retention`. It assumes a message can fail near the end of the source's retention window. If the workload guarantees every message is processed (or fails) within minutes of being sent, a message will never be old enough for the DLQ's shorter retention to bite, and R3 is a false positive in practice. The check is correct for the worst case; the worst case may not occur for a given workload.

**Mitigation in the methodology**: R3 is critical because the worst case is silent total loss of the most diagnostic messages, and the fix (raise DLQ retention to the 14-day max) is cheap and side-effect-free. Prefer the false positive to the silent loss.

### F4. The resource policy is only half the access story

R7 reads the queue's resource `Policy`. The effective set of principals that can `SendMessage` / `ReceiveMessage` is the *union* of that resource policy and every IAM identity policy in the account. A queue with no resource policy at all is not "private": identity policies elsewhere may grant broad access. R7 can only ever flag what the resource policy itself exposes.

**Escalation rule**: a clean R7 is not proof the queue is access-scoped. The IAM-union join is named in every boundary section; closing it needs the account's identity policies, which this skill does not read.

### F5. The low-severity flags cannot be confirmed from the queue

R5 (default visibility), R8 (encryption off), and R9 (FIFO dedup off) each depend on something the queue does not contain: consumer processing time, the data classification, and producer behaviour respectively. They are surfaced as low-severity flags precisely because the skill cannot prove them. Treating a low flag as a confirmed bug is a misread.

## Operational failure modes

### O1. Stale attributes

`GetQueueAttributes` is a point-in-time read. If the queue was reconfigured after the snapshot was taken, the audit describes the old config. `LastModifiedTimestamp` is in the attributes; check it against when the snapshot was pulled.

### O2. The DLQ attributes were not provided

R3 (retention ordering) needs the DLQ's own attributes. When the source queue references a DLQ but the DLQ's `GetQueueAttributes` was not supplied, the retention-ordering check is silently skipped. The audit notes the DLQ is wired but cannot validate its retention.

**Mitigation**: always pull the DLQ's attributes too. Its ARN is in the source queue's `RedrivePolicy.deadLetterTargetArn`.

**Escalation rule**: if R3 could not be evaluated because the DLQ attributes are missing, say so rather than implying the retention ordering is fine.

### O3. Redrive chains and self-references

A DLQ can itself have a `RedrivePolicy` (a chained dead-letter path), and a misconfiguration can point a queue's DLQ at itself. This skill audits one source queue and its immediate DLQ; it does not walk a redrive chain. A multi-hop dead-letter topology needs the cross-resource graph, not a single-queue read.

## When to escalate to a human (summary)

Escalate, or surface as a question rather than a verdict, when **any** of the following is true:

- The queue's role (processing vs buffer) is unknown and R1 fired.
- R3 could not be evaluated because the DLQ attributes were not provided.
- A clean R7 is being read as proof the queue is access-scoped (it is not; the IAM union is unread).
- A low-severity flag (R5, R8, R9) is about to be acted on as a confirmed bug without checking the thing behind the boundary.
- The dead-letter topology is multi-hop (a redrive chain).

Escalation does not mean the agent stops. It means: report the findings, state which checks were deferred and why, name the boundary, and let the human or the next data source close the join.

## How to add a new failure mode here

When a replay test catches a misclassification, or a real-world use surfaces a new pattern, add it under "Methodology-level" or "Operational" with:

1. A short name (`F6`, `O4`, ...).
2. The failure shape, in one sentence.
3. Whatever the methodology already does about it.
4. The escalation rule for it.

Then add a regression test under `tests/` that asserts the audit produces the correct response, even if the response is "defer to the boundary, do not assert a bug".

README.md 4.9 KB

# sqs-queue-auditor

Configuration-audit skill for a single AWS SQS queue.

Parses the `GetQueueAttributes` output for one queue (and its referenced dead-letter queue), applies the judgment a senior engineer applies to that one source, and reports the misconfigurations that silently drop or re-deliver messages while every attribute reads as fine. Then it names the boundary: the questions a single queue's config cannot answer.

## Files in this skill

| File | What it is |
|---|---|
| [`SKILL.md`](./SKILL.md) | The methodology. This is what an AI agent loads. |
| [`examples/`](./examples/) | Eight worked examples, one per rule plus a clean control. |
| [`fixtures/`](./fixtures/) | Committed `GetQueueAttributes` snapshots that drive the replay tests. No external credentials required. |
| [`tests/`](./tests/) | Replay tests that exercise the audit against the fixtures. |
| [`FAILURE_MODES.md`](./FAILURE_MODES.md) | Where this skill is wrong and where the agent should escalate. |

## What it checks

| Code | Rule | Severity |
|---|---|---|
| R1 | No dead-letter queue on a processing queue | high |
| R2 | `maxReceiveCount` outside the 3-10 band | medium / low |
| R3 | DLQ retention not longer than source retention | critical |
| R4 | Poison messages age out before reaching the DLQ | critical |
| R5 | Visibility timeout at the 30s default | low |
| R6 | Retention shorter than a plausible outage | medium |
| R7 | Resource policy allows a wildcard principal with no condition | high |
| R8 | Server-side encryption at rest disabled | low |
| R9 | FIFO queue with content-based dedup off | low |

The two critical rules (R3, R4) are the ones a console read almost never catches: both turn a correctly-wired, correctly-sized dead-letter queue into one that silently never receives the messages it was built for.

## Quality bar (this skill passes all three)

- [x] Two worked examples required by the bar; this skill ships [eight](./examples/), one per rule plus a clean control that asserts no false positives.
- [x] Fixture-based replay tests, runnable with no external credentials. 48 assertions across the 8 tests (`for t in tests/replay_*.py; do python "$t" || exit 1; done`).
- [x] Explicit failure-modes section ([`FAILURE_MODES.md`](./FAILURE_MODES.md)).

## Measured lift

An LLM ablation eval is committed under [`tests/eval/`](./tests/eval/). A reference run with Claude Sonnet 4.6 (N=3, LLM-as-judge against the 7-item rubric, on the four most diagnostic fixtures) measured **+3.08 / 7 (+44%) lift** of an agent loaded with this `SKILL.md` over an agent given the same `GetQueueAttributes` JSON with no methodology. Treatment beats control on every fixture and sweeps 7.00 / 7 with zero variance.

The lift concentrates where it should: **no control output produced a boundary section** (every cold agent presented a config read as a full health verdict), and on the clean control the cold agent flagged an `aws:SourceArn`-scoped wildcard policy as a HIGH "public queue", the textbook false positive the skill's R7 precision exists to avoid. See [`tests/eval/README.md`](./tests/eval/README.md) for the per-fixture table, the per-rubric-item breakdown, the R4 over-fire regression the eval caught and the SKILL.md edit that closed it, and the caveats. Reproduce with `python tests/eval/run_eval.py --trials 3`.

## How to use

### As a Claude Code / Claude Skills user

Drop `skills/sqs-queue-auditor/` into your skills directory and invoke when reviewing or hardening a queue. The agent reads `SKILL.md`, parses the queue attributes, and reports findings plus the boundary. Point it at a real queue with `aws sqs get-queue-attributes --queue-url <url> --attribute-names All`, or run it against the committed fixtures first.

### As a contributor adding a new rule or example

1. Add a fixture directory under `fixtures/<example-slug>/` with `queue.json` (and `dlq.json` if the queue has a DLQ), following the `GetQueueAttributes` shape in [`tests/README.md`](./tests/README.md).
2. Add a worked example under `examples/` mirroring the existing eight.
3. Add a replay test under `tests/replay_NN_<slug>.py` asserting the expected findings and that the boundary is reported.
4. Update [`SKILL.md`](./SKILL.md) and this table if the rule is new.

See the top-level [`CONTRIBUTING.md`](../../CONTRIBUTING.md) for the repo-wide bar.

## Anyshift integration (opt-in)

The audit runs vendor-neutral by default. Every boundary note this skill emits is a join it cannot make from one queue's attributes: queue to its consumers, queue to its CloudWatch metrics over time, queue to the account's IAM graph, queue to the producers and consumers on either side. Opting in to the [Anyshift MCP](https://www.anyshift.io) resolves those joins from a versioned resource graph, so a deferred flag becomes a closed finding.

A measured "with vs without" delta will be published here once the MCP integration has been exercised against the replay fixtures.

## License

[Apache 2.0](../../LICENSE).

SKILL.md 13.3 KB

---
name: sqs-queue-auditor
description: Audit a single AWS SQS queue's configuration for the misconfigurations that silently drop or re-deliver messages while every attribute reads as fine. Parses the GetQueueAttributes output (and the referenced dead-letter queue), checks the redrive path (DLQ present, maxReceiveCount band, DLQ-vs-source retention ordering), the message lifecycle (poison messages aging out before they reach the DLQ, default visibility timeout, short retention), and exposure (open resource policy, encryption at rest, FIFO dedup contract). Reports findings with severity and a recommendation, then names the boundary: the questions a single queue's config cannot answer (consumer processing time, live behaviour, the IAM union, the producers and consumers on either side). Use when asked to review, harden, or sanity-check an SQS queue, or to explain why messages are going missing. Vendor-neutral; runs offline against the queue attributes with no Anyshift account.
---

# sqs-queue-auditor

Configuration-audit skill for a single AWS SQS queue. Takes the `GetQueueAttributes` output for one queue, applies the judgment a senior engineer applies to that one source (the thresholds, the known-bad combinations, the one arithmetic relationship that turns a correct-looking config into silent message loss), and returns a ranked list of findings with recommendations. Then it names exactly where a single queue's configuration stops being able to answer the question.

## When to invoke

- An agent is asked to review, harden, or sanity-check an SQS queue before or after it ships.
- Messages are going missing or being processed twice and nobody can see why from the console.
- A dead-letter queue is configured but empty during an incident, and the question is whether it is actually wired to catch what is failing.
- A queue is being added to a Terraform module or a CDK stack and the config should be checked against the known-bad combinations before apply.

## What this skill reads, and what it does not

It reads the static configuration of **one queue**, plus the attributes of the **dead-letter queue that queue's own RedrivePolicy points at**. Both are SQS control-plane reads (`GetQueueAttributes`). That is the entire input. The audit is correct and complete *for what a queue's configuration can tell you*, and it is explicit about the rest:

- It does **not** read CloudWatch. Live behaviour (redrive volume, age of the oldest message, in-flight count, empty-receive rate) is a time-series, not an attribute.
- It does **not** read the consumers. Whether the visibility timeout is actually long enough is a property of how long the consumer takes, which is not in the queue.
- It does **not** read account IAM. The effective set of principals that can act on the queue is the union of the resource policy (visible) and every identity policy in the account (not visible here).
- It does **not** read the producers. Whether the right services are writing to the queue, and whether anyone is draining the DLQ, needs the inventory on either side.

Every audit ends by naming these. The boundary is the same one every time: the join across resources, across sources, or across time.

## The methodology, in order

### 1. Parse the attributes

`GetQueueAttributes` returns every value as a string, and the compound attributes are JSON documents encoded *inside* those strings. Before any judgment:

- Parse `RedrivePolicy` (a JSON string) into `deadLetterTargetArn` and `maxReceiveCount`. A queue with no `RedrivePolicy` has no DLQ.
- Parse `MessageRetentionPeriod`, `VisibilityTimeout`, `DelaySeconds` as integer seconds (they arrive as strings).
- Parse `Policy` (a JSON string) into IAM statements, if present.
- Read `FifoQueue`, `ContentBasedDeduplication`, `SqsManagedSseEnabled`, `KmsMasterKeyId`.
- If a DLQ is referenced, load *its* attributes too. The retention-ordering check is impossible without them.

A naive read skips the embedded JSON entirely and never sees the redrive wiring. Parsing it is step zero of the judgment.

### 2. Audit the redrive path

The dead-letter path is where messages are supposed to go when processing fails. Three things break it:

- **No DLQ on a processing queue (R1).** Without a `RedrivePolicy`, a poison message is retried until `MessageRetentionPeriod` expires, then deleted with no signal. There is no quarantine.
- **maxReceiveCount out of band (R2).** Below 3, a transient downstream blip dead-letters good messages. Above 10, poison messages are retried many times before quarantine, delaying detection and feeding R4. The sane band is roughly 3 to 10.
- **DLQ retention not longer than the source (R3).** A message's age is measured from its original `SentTimestamp`, and SQS does **not** reset that timestamp when the message moves to the DLQ. If the DLQ's retention is less than or equal to the source's, a message that fails late in the source's window arrives in the DLQ already near its age limit and is deleted almost immediately. The DLQ looks wired and sized; the messages you most need to inspect are the ones it drops. This is the single most important non-obvious check in the skill.

### 3. Audit the message lifecycle

Three queue-side timing relationships, all derivable from the static config:

- **Poison messages age out before the DLQ (R4).** A poison message needs at least `maxReceiveCount x VisibilityTimeout` seconds of wall-clock to exhaust its receive count and dead-letter. If that product exceeds `MessageRetentionPeriod`, retention wins: the message is deleted by age before it ever reaches the DLQ. The DLQ is configured but unreachable for slow failures. This is pure arithmetic on three attributes and is almost never checked by hand. Fire R4 **only on the configured-value inequality** (`maxReceiveCount x VisibilityTimeout > MessageRetentionPeriod`); do not raise it speculatively because backlog or load "might" stretch the wall-clock. The product is already a lower bound, so a config that satisfies the inequality is safe by construction. Queue depth and receive cadence are behind the boundary, not inputs to this check.
- **Visibility timeout at the 30s default (R5).** A risk flag, not a proven bug. If a consumer takes longer than 30s, the message reappears mid-processing and is delivered twice. Whether that happens depends on consumer processing time, which is not a queue attribute. Surfaced as low severity and deferred to the boundary.
- **Retention shorter than a plausible outage (R6).** Retention below an hour means a brief consumer outage, deploy, or scaling lag silently drops every message still queued.

### 4. Audit exposure

- **Open resource policy (R7).** A `Policy` statement that allows a wildcard principal (`"*"`) with no narrowing `Condition` (`aws:SourceArn`, `aws:SourceAccount`, `aws:PrincipalOrgID`) authorises any AWS principal to act on the queue. This is the confused-deputy and public-queue exposure. A wildcard principal *with* a `SourceArn` condition (the standard SNS-to-SQS pattern) is fine and must not be flagged.
- **Encryption at rest disabled (R8).** Neither SQS-managed SSE nor a KMS key configured. Low severity, because whether it matters depends on the data classification, which the queue does not carry.

### 5. Audit FIFO invariants

- **FIFO with content-based dedup off (R9).** When `ContentBasedDeduplication` is off on a FIFO queue, every producer must supply an explicit `MessageDeduplicationId` or duplicate sends are accepted as distinct. Whether the producers actually do this is a property of the producers, not the queue. Flagged low and deferred to the boundary.

### 6. Rank and report, then name the boundary

Order findings by severity (critical, high, medium, low). For each: the rule, the attribute(s) it is grounded in, what breaks, and the fix. Then list the boundary: the joins this audit cannot make. A clean config still gets a boundary section, because a clean config is not a clean system.

## Severity model

| Severity | Meaning |
|---|---|
| **critical** | A configuration that silently loses messages. R3 and R4. |
| **high** | A configuration that loses messages on a poison input, or exposes the queue. R1, R7. |
| **medium** | A configuration that loses messages under an ordinary operational gap. R2 (too low), R6. |
| **low** | A risk flag whose confirmation needs something behind the boundary. R2 (too high), R5, R8, R9. |

The low band is deliberately honest: those findings depend on consumer processing time, data classification, or producer behaviour, none of which is a queue attribute. The skill flags them for verification rather than asserting a bug it cannot prove.

## Rule reference

| Code | Rule | Severity | Grounded in |
|---|---|---|---|
| R1 | No dead-letter queue on a processing queue | high | `RedrivePolicy` absent |
| R2 | `maxReceiveCount` outside the 3-10 band | medium / low | `RedrivePolicy.maxReceiveCount` |
| R3 | DLQ retention not longer than source retention | critical | source vs DLQ `MessageRetentionPeriod` |
| R4 | Poison messages age out before reaching the DLQ | critical | `VisibilityTimeout` x `maxReceiveCount` vs `MessageRetentionPeriod` |
| R5 | Visibility timeout at the 30s default | low | `VisibilityTimeout` |
| R6 | Retention shorter than a plausible outage | medium | `MessageRetentionPeriod` |
| R7 | Resource policy allows a wildcard principal with no condition | high | `Policy` |
| R8 | Server-side encryption at rest disabled | low | `SqsManagedSseEnabled` / `KmsMasterKeyId` |
| R9 | FIFO queue with content-based dedup off | low | `ContentBasedDeduplication` |

## Output format

The agent's final message in any invocation must include:

1. **Queue**: ARN, standard or FIFO, DLQ wired or not.
2. **Findings**: ranked by severity, each with the rule code, the attribute(s), what breaks, and the recommendation. Or "no findings" for a clean config.
3. **Boundary**: the joins this audit could not make, stated explicitly so the gap is visible instead of silent.

## Worked examples

Eight end-to-end examples are committed under `examples/`, each with fixtures (real `GetQueueAttributes` shape) and a runnable replay test. Each isolates one rule, except where two genuinely co-occur.

- [`examples/01-no-dlq.md`](./examples/01-no-dlq.md): a payments queue with no DLQ; poison messages are retried until retention expiry, then dropped (R1).
- [`examples/02-dlq-retention-shorter-than-source.md`](./examples/02-dlq-retention-shorter-than-source.md): the silent-loss bug; the DLQ retains for less time than the source, so late failures are deleted on arrival (R3).
- [`examples/03-maxreceivecount-too-low.md`](./examples/03-maxreceivecount-too-low.md): `maxReceiveCount=1` dead-letters good messages on the first transient failure (R2).
- [`examples/04-poison-ages-out-before-dlq.md`](./examples/04-poison-ages-out-before-dlq.md): the flagship; a 15-minute visibility timeout and `maxReceiveCount=1000` mean poison messages age out before reaching a correctly-wired DLQ (R4, plus R2).
- [`examples/05-default-visibility-short-retention.md`](./examples/05-default-visibility-short-retention.md): a 30s default visibility timeout and 5-minute retention; two soft flags that defer to the boundary (R5, R6).
- [`examples/06-public-queue-policy.md`](./examples/06-public-queue-policy.md): a resource policy with `Principal: "*"` and no condition, on an unencrypted queue (R7, R8).
- [`examples/07-fifo-dedup-off.md`](./examples/07-fifo-dedup-off.md): a FIFO queue with content-based dedup off, depending on a producer contract the queue cannot verify (R9).
- [`examples/08-clean-standard.md`](./examples/08-clean-standard.md): the control; a correctly-configured queue produces zero findings and still reports its boundary.

## Replay tests

Every example has a replay test in `tests/` that runs the audit against committed fixtures, with no external credentials. Run from the skill directory:

```bash
for t in tests/replay_*.py; do python "$t" || exit 1; done
```

The 8 tests cover all nine rules, the severity model, and the clean-control (no false positives), totalling 48 assertions. Tests exit non-zero if the audit produces the wrong findings or drops the boundary. See [`tests/README.md`](./tests/README.md) for the fixture schema and how to add a new replay test.

## Failure modes

This skill is wrong in predictable ways. Read [`FAILURE_MODES.md`](./FAILURE_MODES.md) before relying on it. Highlights:

- It audits configuration, not behaviour. A queue that passes every check can still be failing right now for a reason only CloudWatch shows.
- The R1 "is this a processing queue" judgment is supplied by the caller; a pure buffer queue may legitimately have no DLQ.
- The low-severity flags (R5, R8, R9) cannot be confirmed without the consumer, the data classification, or the producers. They are flags, not verdicts.

## Anyshift integration (opt-in)

The audit above runs end-to-end against the `GetQueueAttributes` output the user already has. No Anyshift dependency.

Every boundary note in this skill is a join: queue to its consumers, queue to its metrics over time, queue to the account's IAM graph, queue to the producers and consumers on either side. The Anyshift MCP can act as a context primer by resolving those joins from a versioned resource graph, so a finding like R5 ("visibility timeout at default, verify against consumer processing time") or R7 ("resource policy is half the access story") can be closed instead of deferred. A measured "with vs without" delta will be published here once the integration has been exercised against the replay fixtures.