# Event Delivery Failure Modes Guide

Use this reference for AWS event-driven architecture reviews involving EventBridge, SQS, SNS, Step Functions, Pipes, event schemas, filtering, cross-account routing, retries, DLQs, archive/replay, idempotency, and event-loop risk.

## What people get wrong

The lazy story is:

> Async decoupling makes the system more reliable.

Wrong. Async systems move failure into queues, retries, duplicates, ordering gaps, poison messages, and invisible backlog. Decoupling without contracts and observability is just delayed failure.

Common bad assumptions:

- EventBridge patterns are precise by default.
- DLQs prove no event is lost.
- SNS, SQS, and EventBridge are interchangeable.
- Replay is safe without idempotent consumers.
- Step Functions state machines automatically document business correctness.
- Cross-account event buses are safe if the policy allows the account.

## Event-driven failure modes

- Broad event patterns create fanout storms, loops, or unintended consumers.
- Consumer retries amplify downstream throttling or duplicate side effects.
- SQS visibility timeout, max receive count, FIFO ordering, or deduplication is misaligned with handlers.
- SNS filtering drops events silently from business perspective.
- EventBridge archive/replay reprocesses stale events into non-idempotent workflows.
- Pipes/enrichment target roles allow unintended read/write access.

## Minimum safe workflow

1. Map producers, buses/topics/queues/state machines, consumers, schemas, ownership, and account boundaries.
2. Define delivery guarantees: ordering, duplication, latency, retention, replay, and poison-message handling.
3. Review filtering and routing precision for EventBridge rules, SNS filters, Pipes, and cross-account bus policies.
4. Verify retry, DLQ, redrive, archive, replay, timeout, and idempotency behavior for every consumer.
5. Check observability: backlog, age, failures, throttles, retries, DLQ depth, business counters, and trace correlation.
6. Compare service choice: SNS, SQS, EventBridge, Step Functions, or Pipes based on coupling and failure semantics.
7. Require approval before live replay, redrive, rule broadening, or policy mutation.

## Verification targets

- EventBridge event buses, rules, patterns, targets, DLQs, archives, replays, global endpoints, and bus policies
- SQS queue type, visibility timeout, retention, redrive policy, DLQ, FIFO/dedup, and ApproximateAgeOfOldestMessage
- SNS topic policy, subscription filters, delivery status, DLQ, encryption, and cross-account subscribers
- Step Functions retries, catches, timeouts, compensation paths, execution history, and idempotency keys
- EventBridge Pipes source/filter/enrichment/target roles and failure destinations
- schema registry/contracts, producer ownership, consumer ownership, and version compatibility

## When to push back

Push back if the user asks to:

- replay or redrive events without idempotency and blast-radius proof
- broaden EventBridge rules to catch unknown events
- ignore DLQ growth because primary flow is healthy
- use async messaging to hide slow or unreliable dependencies
- grant cross-account bus access without source and detail-type constraints
- choose SNS/SQS/EventBridge without stating delivery semantics

# Official sources

Use this reference only when you need source grounding for AWS service behavior or the detailed source list.

## AWS documentation

Use these as starting points, not as proof of the user's live AWS state:
- https://docs.aws.amazon.com/decision-guides/latest/sns-or-sqs-or-eventbridge/sns-or-sqs-or-eventbridge.html
- https://docs.aws.amazon.com/lambda/latest/dg/concepts-event-driven-architectures.html
- https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

## Grounding rule

Official documentation explains AWS service behavior. It does not prove the user's current account, Region, quota, resource configuration, IAM boundary, pricing, entitlement, or operational state. Prefer read-only AWS MCP or CLI evidence, repository evidence, or sanitized user-provided evidence for current-state claims.

## Current MCP/documentation refresh (2026-06-02)

Service facts from official docs:
- AWS decision guidance differentiates SNS, SQS, and EventBridge by messaging pattern and integration need; choosing one is an architecture decision, not a default.
- Lambda event-driven architecture guidance calls out benefits, trade-offs, and anti-patterns; asynchronous flows need explicit retry, idempotency, and failure handling design.

Sampled live evidence:
- Read-only regional availability sampling reported Amazon EventBridge and AWS Lambda as `isAvailableIn` in `us-east-1`, `us-west-2`, `eu-west-1`, and `ap-southeast-1`.
- Sampled APIs `EventBridge+ListRules`, `SQS+GetQueueAttributes`, `SNS+GetTopicAttributes`, `Lambda+GetFunction`, and `SFN+DescribeStateMachine` were reported `isAvailableIn` in those regions.

Review implications:
- Require event contract, producer/consumer ownership, retry/DLQ policy, ordering/deduplication needs, idempotency, schema/versioning, observability, and replay strategy.
- Service/API availability does not prove that a specific event bus, queue, topic, rule, Lambda, or state machine is configured correctly.

# Safety checklist

Use this reference before privileged, destructive, traffic-changing, cost-changing, compliance-impacting, or production-impacting recommendations.

## Non-negotiables

- Never ask users to paste secrets, access keys, session tokens, private keys, customer identifiers, or sensitive account data into chat.
- Use read-only AWS MCP or read-only AWS CLI evidence for live state when available; otherwise use repository evidence, sanitized user evidence, or official documentation and label the evidence level.
- Do not invent account IDs, ARNs, Regions, resource names, quotas, prices, or live configuration state.
- Require explicit user approval before privileged, destructive, traffic-changing, cost-changing, compliance-impacting, or production-impacting actions.
- Use current official AWS documentation for service behavior when the answer depends on AWS service details.
- Keep remediation least-privilege, reversible, and scoped to the requested workload or account boundary.

## Stress checks

- What can expose data?
- What can escalate privilege?
- What can break production or block rollback?
- What can create unbounded cost?
- What compliance or audit evidence is missing?
- What rollback or validation path is unproven?

## Evidence labels

Use `live evidence`, `repo evidence`, `user-provided evidence`, `documentation-based`, or `inference`. Documentation alone never proves the user's live AWS state.

# Workflow and output contract

Use this reference only when performing the full review, implementation guidance, incident triage, or production-readiness pass.

## Review domains

Check these areas before giving a verdict:
- Producers, consumers, event contracts, schema/versioning, accounts/Regions, ownership, and ordering requirements
- Pattern precision, filtering, transformations, cross-account policies, targets, retries, DLQs, archives, replay, and idempotency
- Monitoring: TriggeredRules, FailedInvocations, throttling, latency, DLQ depth, workflow failures, and alarm thresholds
- Failure modes: loops, duplicate delivery, poison messages, consumer slowness, quota limits, and cost spikes

## Safe workflow

1. **Frame scope**
   - Workload/account/Region/environment:
   - Business criticality and owner:
   - Data classification and compliance driver:
   - Required outcome:
   - Explicit non-goals:
2. **Collect evidence**
   - Prefer read-only AWS MCP or read-only AWS CLI evidence for current-state claims when available.
   - Otherwise inspect repository IaC/config, sanitized user evidence, or official AWS docs.
   - Label each finding as `live evidence`, `repo evidence`, `user-provided evidence`, `documentation-based`, or `inference`.
3. **Stress-test risk**
   - What can expose data?
   - What can escalate privilege?
   - What can break production or block rollback?
   - What can create unbounded cost?
   - What evidence is missing?
4. **Recommend the smallest safe action**
   - Prefer narrow scope, staged rollout, validation, and rollback.
   - If the safest action is to stop and gather evidence, say that plainly.

## Output contract

Return this structure:
```markdown
# AWS Event Driven Architecture Review: <scope>
## Executive verdict
- Status: READY / READY WITH RISKS / NOT READY / NEEDS EVIDENCE
- Biggest risk:
- Evidence level:
## Scope and assumptions
- Confirmed:
- Unknown:
- Out of scope:
## Findings
| Severity | Finding | Evidence | Why it matters | Minimum safe action |
|---|---|---|---|---|
## Recommended actions
1. <action> — owner: <owner>, validation: <check>, rollback: <rollback>
## Validation
- Commands or checks:
- Expected result:
## Residual risk
- <risk or explicit none>
```