# 01 — Org, Accounts & Governance

Scope: organization structure, account/project strategy, landing zones, guardrails
(SCPs / org policies / Azure Policy), centralized logging and billing, tagging
standards. This is the layer everything else inherits from — get it wrong and every
downstream control is patchwork.

## 1. Accounts/projects are the unit of isolation

- **Use separate accounts (AWS) / projects (GCP) / subscriptions (Azure) as blast-radius
  boundaries.** IAM mistakes, quota exhaustion, credential theft, and cost overruns are
  contained by the account boundary; they are NOT contained by tags, VPCs, or naming
  conventions inside one account.
- Minimum viable structure, even for small teams:
  - **Management/org root** — billing and org administration only. No workloads, no
    users doing daily work, ever.
  - **Security/audit** — centralized logs, audit trails, security tooling. Write access
    from workload accounts, read access only for security team.
  - **Prod** — one per major system or per team at scale; one shared prod account is
    acceptable only below ~10 engineers.
  - **Non-prod** (dev/staging) — separate from prod, always. Staging in the prod
    account is the most common audit finding that enables lateral movement.
  - **Sandbox** — disposable experimentation with hard budget caps and auto-cleanup.
- Per-environment isolation beats per-application isolation when you must choose: a
  dev compromise must not be able to touch prod data under any IAM misconfiguration.
- Scale pattern: account-per-team-per-environment, vended automatically (Account
  Factory / project factory in Terraform). Manual account creation does not scale past
  ~10 accounts and produces snowflakes.

```hcl
# GOOD: environments as separate accounts under OUs (Terraform sketch)
resource "aws_organizations_organizational_unit" "workloads_prod" {
  name      = "workloads-prod"
  parent_id = aws_organizations_organization.org.roots[0].id
}
resource "aws_organizations_organizational_unit" "workloads_nonprod" {
  name      = "workloads-nonprod"
  parent_id = aws_organizations_organization.org.roots[0].id
}
resource "aws_organizations_organizational_unit" "security" {
  name      = "security"
  parent_id = aws_organizations_organization.org.roots[0].id
}

# BAD: one account, environments by tag
# tags = { Environment = "prod" }   <- this is a label, not a boundary
```

## 2. Organize by OU/folder, attach policy at the container

- Group accounts into OUs (AWS), folders (GCP), management groups (Azure) by **policy
  needs**, not by org chart. "workloads-prod", "workloads-nonprod", "security",
  "sandbox", "suspended" is a better top level than "team-alpha", "team-beta".
- Attach guardrails to OUs/folders so new accounts inherit them on creation. A
  guardrail applied per-account is a guardrail someone will forget.
- Keep a "suspended/quarantine" OU with a deny-all-but-investigation policy for
  compromised or decommissioning accounts.

## 3. Guardrails: preventive controls at the org layer

Guardrails are deny-rules that apply to everyone including account admins. They encode
"things that must never happen" — IAM inside the account encodes "who may do what."

Non-negotiable guardrail set (express as AWS SCPs / GCP org policy constraints / Azure
Policy deny assignments):

1. **Deny leaving the organization** (AWS: `organizations:LeaveOrganization`).
2. **Deny disabling/altering audit logging** (CloudTrail stop/delete/update; GCP audit
   config changes; Azure diagnostic settings deletion) outside the security account.
3. **Deny root user actions** in member accounts (AWS: deny all where
   `aws:PrincipalArn` is root) — combined with org-level root credential management /
   root access removal where available.
4. **Region restriction** — deny resource creation outside approved regions (data
   residency + reduces unwatched attack surface).
5. **Deny public object storage** at org level (S3 BPA account config protected by SCP;
   GCP `storage.publicAccessPrevention` enforced; Azure deny blob public access).
6. **Deny creation of IAM users / access keys** (AWS) except a tagged break-glass path;
   GCP: `iam.disableServiceAccountKeyCreation` enforced org-wide with per-project
   exceptions only via documented process.
7. **Deny unencrypted storage creation** (EBS/RDS/disks without encryption).
8. **Deny default-VPC usage or auto-creation** where the provider supports it (GCP:
   `compute.skipDefaultNetworkCreation`).

```json
// GOOD: SCP fragment — protect the audit trail
{
  "Effect": "Deny",
  "Action": ["cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail"],
  "Resource": "*",
  "Condition": { "StringNotEquals": { "aws:PrincipalArn": "arn:aws:iam::SECURITY_ACCT:role/org-audit-admin" } }
}
```

- **AWS: pair SCPs with Resource Control Policies (RCPs).** SCPs cap what
  principals in member accounts may do; RCPs cap what *any* principal — including
  ones outside the org — may do to **resources** in member accounts (S3, STS, KMS,
  SQS, Secrets Manager, DynamoDB, ECR, CloudWatch Logs, and more). Use an RCP for
  the org-wide data perimeter — deny access from principals outside the org,
  require TLS — instead of repeating those conditions in every bucket/key policy
  (rules/02 §3, rules/05 §1). RCPs don't affect the management account or
  service-linked roles.

```json
// GOOD: RCP fragment — org-wide S3 identity perimeter (external principals denied
// even where a bucket policy grants them access)
{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": "*",
  "Condition": {
    "StringNotEqualsIfExists": { "aws:PrincipalOrgID": "o-example" },
    "BoolIfExists": { "aws:PrincipalIsAWSService": "false" }
  }
}
```

- SCPs/RCPs/org policies do not grant anything; test them with org-policy dry-run /
  SCP simulation against real workflows before enforcing, and roll out OU-by-OU.
- Pair preventive guardrails with detective baseline: AWS Config / Security Hub, GCP
  Security Command Center, Azure Defender for Cloud / Policy compliance — deployed
  org-wide from the security account, findings centralized. (Alert routing and
  on-call: see sota-observability.)

## 4. Centralized logging and billing

- **One audit trail, org-wide, to a bucket the producers cannot touch.** AWS: an
  organization CloudTrail delivering to a bucket in the security/log-archive account
  with object lock + deny-delete bucket policy. GCP: org-level log sink to a project
  in the security folder. Azure: diagnostic settings to a central Log Analytics
  workspace / storage in a locked subscription.
- Workload accounts get *write* into the central store and *no* delete/modify. Admins
  of a compromised account must not be able to erase their tracks.
- Centralize: control-plane audit logs (always), DNS query logs, VPC flow logs
  (sampled where volume demands), object-storage access logs for sensitive buckets,
  LB access logs.
- **Consolidated billing under the management account** with cost data exported
  (CUR / BigQuery billing export / Cost Management exports) to a queryable store
  available to FinOps tooling. Per-account billing views delegated to team leads.
  Details: rules/06.

## 5. Tagging/labeling standard — non-negotiable

Untagged resources cannot be attributed, costed, or safely deleted. Enforce, don't
request.

Minimum mandatory tag set (adapt names to house style, keep the semantics):

| Tag | Meaning | Example |
|---|---|---|
| `owner` | Team or service owner (group, not person) | `payments-team` |
| `env` | Environment | `prod` / `staging` / `dev` / `sandbox` |
| `service` | System/application name | `checkout-api` |
| `cost-center` | Billing attribution | `cc-4012` |
| `data-class` | Highest data sensitivity touched | `public` / `internal` / `confidential` / `regulated` |
| `managed-by` | Provisioning source | `terraform:repo-name` / `manual` |

- Enforce at three layers: (1) provider policy — AWS tag policies + SCP requiring tags
  on create for taggable services, GCP/Azure label/tag policy; (2) IaC — `default_tags`
  in the Terraform AWS provider / module-level mandatory variables; (3) detective —
  scheduled report of noncompliant resources with auto-quarantine in sandbox.
- `managed-by: manual` is an explicit exception flag, reviewed monthly, not a default.
- Tag VALUES come from a controlled vocabulary (tag policy / validation in module),
  or you get `Prod`, `prod`, `production`, and `prd` and lose attribution anyway.

```hcl
# GOOD: provider-level default tags — applies to every resource in the config
provider "aws" {
  default_tags {
    tags = {
      owner      = "payments-team"
      env        = "prod"
      service    = "checkout-api"
      managed-by = "terraform:infra-payments"
    }
  }
}
```

## 6. Landing zone: bootstrap once, vend forever

A landing zone is the automated baseline every new account/project receives:

- Org placement (correct OU/folder) and guardrail inheritance.
- Baseline IAM: SSO permission sets mapped, no local users, break-glass role.
- Audit logging wired to central store; security tooling enrolled.
- Network: either a vended VPC pattern (rules/03) or explicit "no network" for
  serverless-only accounts.
- Budget + anomaly alert with an owner (rules/06).
- Mandatory tags applied at the account level.

Use the provider's framework as a starting point (AWS Control Tower / Landing Zone
Accelerator, GCP project factory blueprints, Azure landing zones) but keep the
definition in version-controlled IaC — pipeline and IaC-scanning concerns belong to
sota-devsecops. A landing zone you can't reproduce from code is a liability.

Bootstrap exceptions (the only acceptable console-click steps, documented in a runbook):
org creation, root MFA enrollment, initial SSO/identity-provider connection, billing
contacts.

## 7. Anti-patterns

- **The "one big account" with tag-based separation.** Every IAM policy becomes a
  condition-key puzzle; one wildcard ends the separation.
- **Workloads in the management account.** The management account can alter every
  guardrail; a compromise there is org-wide root.
- **Org chart as OU tree.** Reorgs then force account migrations; policy needs are
  stabler than reporting lines.
- **Guardrails only in dev** ("we'll enable in prod later"). Reverse it: guardrails in
  prod first; dev gets looser quotas, not looser security.
- **Shadow orgs**: a second cloud provider or personal accounts on the corporate card
  with no guardrails. Audit billing data for unknown payer lines.

## Audit checklist

- [ ] Org exists; all accounts/projects/subscriptions are members; none standalone.
- [ ] Management account/root project has no workloads, no daily-use identities.
- [ ] Prod and non-prod are separate accounts/projects (not tags in one).
- [ ] Security/log-archive account exists; org audit trail delivers there; producers
      cannot delete/modify logs (bucket policy / object lock / sink permissions).
- [ ] Root/owner credentials: MFA enforced, no access keys, usage alarmed (any root
      login pages someone).
- [ ] SCPs/org policies enforce, at minimum: no audit-log tampering, no public
      buckets, region restriction, no new IAM users/SA keys, no leaving org.
- [ ] AWS: RCPs enforce the org-wide data perimeter on supported services (deny
      principals outside the org, require TLS) — not left to per-resource policies.
- [ ] Guardrails attached at OU/folder level, inherited by new accounts automatically.
- [ ] Account vending is automated and IaC-defined; pick a recent account and verify
      it matches the baseline.
- [ ] Mandatory tag set defined, enforced on create, and a compliance report exists;
      sample 10 resources across accounts for tag presence and vocabulary compliance.
- [ ] Consolidated billing with cost export to queryable store; per-team visibility.
- [ ] Quarantine/suspended OU (or equivalent) exists with deny-most policy.
- [ ] No unknown accounts: reconcile org account list against billing and ownership
      records; every account has a responsive owner.

# 02 — IAM Design

Scope: human access, workload identity, OIDC federation, permission boundaries,
least-privilege iteration, cross-account patterns, break-glass, and the IAM findings
that dominate real audits. Secret storage/rotation mechanics: see
sota-secrets-management. CI pipeline identity specifics: see sota-devsecops.

## 1. Humans: federated SSO, short-lived sessions, zero static keys

- **All human access goes through the identity provider** (AWS IAM Identity Center,
  Google Cloud Identity / Workspace, Microsoft Entra ID) with MFA — phishing-resistant
  (FIDO2/passkeys) for admin roles. Humans assume short-lived roles; sessions ≤ 8h for
  standard, ≤ 1h for privileged.
- **Zero IAM users for humans.** No console passwords, no access keys. AWS IAM users
  exist only for the rare service that cannot do roles (legacy SMTP, some third-party
  integrations) — each one inventoried, key-rotated, and condition-restricted
  (source IP, single action).
- Access is granted to **groups in the IdP**, mapped to permission sets / role
  bindings. Direct user-to-role grants are an audit finding: they survive offboarding
  reviews invisibly.
- Tier the permission sets: `read-only` (default for everyone), `developer`
  (env-scoped write), `admin` (per-account, time-bound). Prefer just-in-time
  elevation (PIM in Entra, temporary elevated access tooling elsewhere) over standing
  admin: standing admin count per prod account should be ~0–2.
- Offboarding = disable in IdP only. If that doesn't kill all access (because a local
  user or shared credential exists), that's the finding.

## 2. Workloads: platform-issued identity, never embedded keys

Every workload gets identity from the platform it runs on; credentials are
short-lived and auto-rotated by the provider:

| Runtime | Mechanism |
|---|---|
| AWS EC2/ECS/Lambda | Instance profile / task role / execution role |
| EKS | EKS Pod Identity or IRSA — role per service account, never node-role inheritance for app permissions |
| GCP compute/run/functions | Attached service account (dedicated per workload, NOT default compute SA) |
| GKE | Workload Identity Federation for GKE |
| Azure compute/AKS | Managed identity (user-assigned per workload) / workload identity for AKS |
| External (CI, SaaS, other cloud) | OIDC/workload identity federation — exchange the external token for short-lived cloud creds |

- **A static cloud access key or service-account JSON key in an env var, file, or
  secret manager is a finding** (High), not a pattern. The fix is federation, not
  better hiding. Enforce with org policy: deny SA key creation (GCP), deny
  `iam:CreateAccessKey` (AWS SCP) except documented exceptions.
- One identity per workload. Shared "app-runner" roles across services destroy both
  least privilege and audit attribution.
- OIDC federation trust must pin **issuer AND subject**: a trust policy that accepts
  any repo/branch from a CI provider is public assumability with extra steps.

```json
// BAD: any GitHub repo in the org (or worse, any at all) can assume this role
"Condition": { "StringLike": { "token.actions.githubusercontent.com:sub": "*" } }

// GOOD: pinned to repo and environment
"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:my-org/infra-payments:environment:prod"
  }
}
```

## 3. Policy authoring rules

- **No wildcards in Action or Resource for write/admin permissions.** `s3:Get*` on a
  scoped bucket: acceptable. `s3:*` on `*`: finding. `iam:*`, `sts:AssumeRole` on `*`,
  `kms:*`: always findings outside admin roles.
- Watch **privilege-escalation primitives** as if they were admin: `iam:PassRole`
  (unscoped = become any role a service can wear), `iam:CreatePolicyVersion`,
  `iam:AttachUserPolicy`, `lambda:UpdateFunctionCode` on privileged functions, GCP
  `iam.serviceAccounts.actAs` / `getAccessToken`, Azure role-assignment write. Scope
  `PassRole` to specific role ARNs plus `iam:PassedToService`.
- Use **conditions as containment**: `aws:SourceVpce`/`SourceIp` for data-plane
  access, `aws:ResourceOrgID`/`PrincipalOrgID` to stop confused-deputy and
  cross-org exfiltration, `sts:ExternalId` for third-party assume-role. On AWS,
  enforce the org-perimeter conditions once, centrally, with an RCP (rules/01 §3)
  instead of hand-copying them into every resource policy.
- Resource policies (bucket/key/queue policies) are a second IAM system — audit them
  with the same rigor. A perfect identity policy is irrelevant if the bucket policy
  grants `Principal: "*"`.
- Write policies in IaC with comments stating *why* each statement exists. An
  uncommented broad grant cannot be safely narrowed later.

```json
// BAD: "deploy role" that is actually privilege escalation to any role
{
  "Effect": "Allow",
  "Action": ["iam:PassRole", "lambda:*"],
  "Resource": "*"
}

// GOOD: pass exactly the runtime role, to exactly the service, update exactly our functions
{
  "Effect": "Allow",
  "Action": "iam:PassRole",
  "Resource": "arn:aws:iam::111122223333:role/checkout-api-runtime",
  "Condition": { "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com" } }
},
{
  "Effect": "Allow",
  "Action": ["lambda:UpdateFunctionCode", "lambda:UpdateFunctionConfiguration"],
  "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:checkout-*"
}
```

## 4. Permission boundaries & delegated administration

- When teams self-manage IAM in their accounts, cap them: **permission boundaries**
  (AWS) on every role they create — boundary forbids IAM/org/billing/guardrail
  mutation and touching other teams' resources; deny role creation *without* the
  boundary attached. GCP/Azure: restrict grantable roles via
  `iam.allowedPolicyMemberDomains` + custom-role discipline / Azure
  `roleDefinitionIds` constraints on owners.
- Separate **control-plane admin** from **data access** in role design: the person
  who can change KMS key policy should not be the role that decrypts production data
  (see rules/05 §encryption).

## 5. Least privilege is iterative — wire the loop

Nobody writes least-privilege first try. Ship slightly-scoped, then tighten on
evidence:

1. Start from activity, not imagination: generate policies from access logs
   (IAM Access Analyzer policy generation from CloudTrail; GCP role recommendations
   from Policy Intelligence; Entra access reviews).
2. Run **unused-access detection** continuously (IAM Access Analyzer unused-access
   findings, GCP IAM Recommender, Azure access reviews): unused roles, unused keys,
   unused granted permissions ≥ 90 days → remove, with an owner-notified grace path.
3. Run **external/public-access analysis** continuously (Access Analyzer external
   findings, SCC, Defender CSPM): any resource or role reachable from outside the
   org zone of trust is reviewed or removed.
4. Quarterly access review for privileged grants; automated diff of who-has-admin
   between quarters.

## 6. Cross-account / cross-project access

- Pattern: **hub identity, spoke roles.** Humans and CI authenticate once (IdP /
  identity account), then assume scoped roles in target accounts. No duplicated users
  per account.
- Every cross-account trust policy must pin the exact principal ARN (not account
  root unless deliberate), and for third parties add `sts:ExternalId` (confused
  deputy) — better: require their OIDC federation instead of an account-wide trust.
- Maintain an inventory of all trust relationships pointing outside the org;
  unknown account IDs in trust policies are Critical until identified.
- GCP: prefer service-account impersonation with
  `roles/iam.serviceAccountTokenCreator` over keys; Azure: cross-tenant via Entra
  B2B/Lighthouse with scoped delegations, never shared SP secrets.

## 7. Break-glass

- Two break-glass paths per cloud, tested quarterly:
  1. **Org level:** management-account root (or equivalent) credentials —
     hardware-MFA, credentials split/sealed (e.g., password in one vault, MFA token in
     a safe), any use alarms the security channel.
  2. **Account level:** a pre-provisioned `break-glass-admin` role assumable by a
     tiny named group via a path that does NOT depend on the IdP (IdP outage is a
     primary break-glass scenario).
- Break-glass use requires: alert fires automatically, post-use review within 24h,
  credential rotation after use. A break-glass account that has ever been used for
  routine work is just an admin account with worse logging.

## 8. Common audit findings (what to actually look for)

| Finding | Severity (typical) | Detection |
|---|---|---|
| Role/bucket/topic assumable or readable by `*` / `allUsers` | Critical | Access Analyzer external findings; policy grep for `"Principal":"*"` without conditions |
| OIDC trust with wildcard subject | Critical | Read every federation trust policy |
| Human IAM users with active access keys | High | Credential report; keys > 90d unrotated |
| `Action:*` / `iam:PassRole` on `*` in non-admin roles | High | Policy lint (Access Analyzer policy checks, custom rules) |
| CI using stored long-lived cloud keys instead of OIDC | High | Inspect CI secret stores + key last-used |
| Unused roles/keys/permissions > 90 days | Medium | Unused-access analyzers |
| Direct user grants bypassing groups | Medium | IdP + cloud mapping diff |
| No permission boundary on delegated-admin-created roles | Medium | List roles missing boundary in self-service accounts |
| Standing admin > 2 humans per prod account | Medium | Enumerate admin-equivalent bindings |
| Break-glass untested / undocumented | Medium | Ask for last test record |

## Audit checklist

- [ ] Credential report / key inventory: zero human IAM users with keys or passwords
      (exceptions documented, conditioned, rotated).
- [ ] SSO enforced with MFA; admin roles require phishing-resistant MFA; sessions
      time-bound.
- [ ] Access granted via IdP groups → permission sets; no direct user bindings.
- [ ] Every workload identity is platform-issued or federated; zero static SA
      keys/access keys in apps or CI (check key-creation org policy is enforced).
- [ ] All OIDC federation trusts pin issuer + audience + exact subject.
- [ ] No `Action:*`/`Resource:*` writes outside admin roles; PassRole/actAs scoped;
      escalation primitives enumerated and justified.
- [ ] Resource policies reviewed: no `Principal:"*"` without strong conditions;
      org-ID conditions on shared resources.
- [ ] Cross-account trusts inventoried; every external account ID identified;
      ExternalId or federation for third parties.
- [ ] Permission boundaries (or provider equivalent) on self-service IAM.
- [ ] Unused-access analyzer running; findings older than 90 days are zero or
      ticketed.
- [ ] Break-glass: exists, IdP-independent, alarmed, tested within last quarter.
- [ ] Offboarding test: pick a recent leaver; verify zero residual access.

# 03 — Networking, DNS, TLS & Edge

Scope: VPC/VNet design, CIDR planning, egress control, private connectivity,
hub-spoke topology, DNS architecture, TLS/certificate automation, load balancing,
CDN, DDoS posture, IPv6.

## 1. VPC design: three tiers, multi-AZ, deny-by-default

- **Standard tier model per VPC/VNet:**
  - **Public subnets** — only internet-facing entry/exit points: load balancers, NAT
    gateways, bastion-replacement endpoints. No application instances. No databases,
    ever.
  - **Private subnets** — application compute. Outbound internet only via controlled
    egress (NAT/proxy); inbound only from LB tier.
  - **Isolated subnets** — data stores and internal-only services. No route to the
    internet in either direction; reach cloud APIs via private endpoints.
- **Span ≥ 2 (prefer 3) availability zones** with one subnet per tier per AZ. A
  single-AZ subnet layout silently forces single-AZ workloads later.
- Security groups / firewall rules: deny-by-default, reference **security groups (or
  tags/service accounts in GCP), not CIDRs**, for internal flows — `sg-app → sg-db:5432`
  survives re-IPs; CIDR rules rot. No `0.0.0.0/0` ingress anywhere except 80/443 on
  the edge LB tier. SSH/RDP from the internet is a finding even "temporarily" — use
  SSM Session Manager / IAP / Azure Bastion instead, and you usually don't need a
  bastion subnet at all.
- Keep NACLs/subnet-level rules coarse (subnet-tier intent) and do fine-grained
  control in SGs; duplicating every rule in both layers guarantees drift.
- Flow logs on (sampled where cost-sensitive), delivered to central logging (rules/01).

```hcl
# BAD: CIDR-based, internet-wide "temporary" access
resource "aws_security_group_rule" "db_in" {
  type        = "ingress"
  from_port   = 5432
  to_port     = 5432
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"] # finding: Critical
}

# GOOD: SG-to-SG, intent readable, survives re-IP
resource "aws_security_group_rule" "db_in" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.db.id
  source_security_group_id = aws_security_group.app.id # only the app tier
}
```

## 2. CIDR planning: allocate like you'll have 50 VPCs

- Reserve a **non-overlapping supernet plan up front** (e.g., carve 10.0.0.0/8 into
  per-region /12s, per-VPC /16–/20s), tracked in IPAM (provider IPAM service or a
  versioned registry). Overlapping CIDRs are nearly unfixable once peered/VPN'd —
  they block peering, hybrid connectivity, and mergers.
- Avoid 192.168.0.0/16 and the most common 10.0.0.0/24-style defaults for anything
  that might ever connect to a partner/VPN — collisions are guaranteed.
- Size for the orchestrator: Kubernetes eats IPs (pod-per-IP models). Give cluster
  VPCs/subnets generous space or use secondary ranges (GKE alias IPs, EKS custom
  networking) from day one.
- Don't bridge overlap with NAT hacks; renumber the smaller side or use IPv6.

## 3. Egress control

- **Egress is a control point, not a default-open pipe.** Data exfiltration and
  C2 traffic leave through egress; treat `0.0.0.0/0` outbound from data tiers as a
  finding.
- Tiered approach:
  1. Isolated tier: no egress at all; private endpoints for needed cloud services.
  2. Private tier: egress via NAT gateway; restrict with L7 egress filtering
     (AWS Network Firewall / GCP Secure Web Proxy / Azure Firewall FQDN rules, or a
     self-managed proxy) allowing named domains for high-sensitivity environments.
  3. Public tier: only LBs/NAT; they originate nothing themselves.
- NAT gateways: one per AZ for prod (cross-AZ NAT = paid cross-AZ hop + AZ
  coupling); they bill per-hour AND per-GB processed — high-volume flows to cloud
  services should bypass NAT via private/gateway endpoints (see rules/06).

## 4. Private connectivity to managed services

- Reach provider services privately: **gateway endpoints** (AWS S3/DynamoDB — free,
  use always), **interface endpoints / PrivateLink**, **GCP Private Google Access /
  Private Service Connect**, **Azure Private Endpoints / service endpoints**.
- Default for prod: data stores, secret managers, container registries, and logging
  endpoints reachable without traversing the internet or NAT. This is both a
  security control (no public exposure, org-ID conditions on endpoint policies) and
  a cost control (NAT per-GB).
- PrivateLink/PSC is also the SOTA pattern for **service-to-service across
  accounts/VPCs without merging networks** — expose one service, not your whole CIDR.

## 5. Topology: flat → peered → hub-spoke

- < ~3 VPCs: peering is fine. Peering is non-transitive — beyond a handful, the mesh
  (n²) becomes unauditable.
- At scale: **hub-spoke** via AWS Transit Gateway / GCP Network Connectivity Center /
  Azure vWAN-Virtual-hub. Centralize in the hub: hybrid links (VPN/Direct
  Connect/Interconnect/ExpressRoute), inspection/egress firewalls, shared endpoints.
  Spokes (workload VPCs) cannot reach each other unless route tables say so —
  segment prod from non-prod at the routing layer, not just SGs.
- Don't share one VPC across unrelated teams as a topology shortcut (GCP Shared VPC
  is the deliberate, IAM-governed exception when used with per-team subnets).

## 6. DNS architecture

- **Registrar hygiene:** domains in a corporate registrar account (not an
  employee's), registrar MFA + transfer lock + registry lock for crown-jewel
  domains, auto-renew with monitored payment, contact = team alias. Expired domains
  and dangling delegations are takeover vectors.
- **Split-horizon:** public zones contain only public entry points; internal records
  live in private zones (Route 53 private hosted zones / Cloud DNS private zones /
  Azure Private DNS) attached to VPCs. Internal hostnames in public DNS leak
  topology. Private endpoints (above) require matching private DNS zones —
  the most common "why is it still going over the internet" bug.
- **Dangling records are the top DNS finding:** CNAMEs/A records pointing at
  released cloud resources (deleted buckets, old LB names, deprovisioned PaaS apps)
  enable subdomain takeover. Make DNS records lifecycle-coupled to the resources in
  IaC; scan zones for danglers regularly.
- **DNSSEC stance:** sign zones where the registrar+provider support is solid and
  you have rotation automation (managed DNSSEC on Route 53/Cloud DNS/Azure DNS);
  skip hand-rolled key management. Always enable it for domains used as identity
  anchors (email/SPF/DKIM-bearing zones). CAA records on all public zones limiting
  issuance to your CAs.
- Low TTLs (60–300s) on records you'll need to move in an incident; long TTLs on
  stable apex/MX.

## 7. TLS and certificate automation

- **Certificate lifetimes are collapsing by CA/Browser Forum schedule: max 200 days
  from 2026-03-15, 100 days from 2027-03-15, 47 days from 2029-03-15.** Manual
  renewal is now an outage generator, period. Every cert must be issued and renewed
  automatically.
- Prefer **provider-managed certs** terminated on managed LBs/CDN (ACM, Google-managed
  certs, Azure-managed) — auto-renewed, no private key you can leak. Where you must
  hold certs (self-managed ingress, on-prem), use **ACME** (Let's Encrypt or your
  CA's ACME endpoint) with DNS-01 for wildcards/internal, cert-manager on Kubernetes.
- Expiry monitoring as a backstop (alert at 30/14/7 days) even with automation —
  automation fails silently; see sota-observability for alert wiring.
- Internal/mTLS: use a private CA service (AWS Private CA, GCP CA Service, or
  service-mesh-issued identities) with short lifetimes; never a long-lived wildcard
  cert copied between services.
- TLS policy on LBs/CDN: modern policy (TLS 1.2 minimum, 1.3 preferred), HSTS on
  web origins.

## 8. Load balancing

- **L7 (ALB / GCP HTTP(S) LB / Azure App Gateway or Front Door)** for HTTP: routing,
  WAF attachment, OIDC auth offload, gRPC. **L4 (NLB / GCP passthrough / Azure LB)**
  for raw TCP/UDP, extreme connection rates, static IPs, PrivateLink targets.
- Health checks must hit a **meaningful endpoint** (checks downstream dependency
  readiness, not just "200 on /"), with sane thresholds; a health check on `/` that
  always 200s converts partial outages into full ones by keeping dead nodes in
  rotation. Distinguish liveness (process up) from readiness (can serve).
- Enable cross-zone/multi-AZ distribution deliberately (know the cross-AZ data cost),
  connection draining/deregistration delay ≥ app's longest request, and LB access
  logs to central storage.
- LBs are the only public compute-adjacent surface: attach WAF for L7 apps exposed
  to the internet; origins behind a CDN must not be directly reachable (see §9).

## 9. CDN and edge

- Put a CDN (CloudFront / Cloud CDN or Media CDN / Azure Front Door) in front of any
  public static or cacheable content, and in front of dynamic apps when you want
  edge TLS, DDoS absorption, and WAF at the edge.
- **Lock the origin:** origin accepts traffic only from the CDN — S3 via Origin
  Access Control (no public bucket behind CloudFront, ever), custom origins via
  origin-auth headers/managed prefix lists/private connectivity. An origin reachable
  directly bypasses your WAF, cache, and DDoS layer.
- **Origin shielding / tiered caching on** for origin-cost-sensitive backends.
- **Cache keys minimal:** include only headers/cookies/query params that change the
  response. Default-everything cache keys = near-0% hit ratio; forgetting a varying
  header = serving user A's response to user B (cache poisoning class).
- Private content: **signed URLs/cookies** (CloudFront signed URLs, GCS/S3 presigned,
  Front Door + token auth) with short expiry; never security-by-obscure-URL.
- Cache invalidation is a deploy step (versioned asset filenames preferred over
  purges).

## 10. DDoS posture

- Baseline (free/default): provider always-on L3/4 mitigation (AWS Shield Standard,
  Google/Azure network defenses), CDN absorbing edge traffic, autoscaling with hard
  caps so an attack can't scale your bill infinitely.
- Internet-facing L7 apps: WAF with rate-limiting rules + managed rule sets.
- Paid tiers (Shield Advanced / Cloud Armor Adaptive / Azure DDoS Protection) when
  you have revenue-critical public endpoints — they add response teams and cost
  protection. Decide explicitly and record the stance; "we never considered DDoS"
  is the finding.
- Don't expose what doesn't need exposure: the best DDoS surface is none (private
  endpoints, CDN-only origins).

## 11. IPv6 stance

- Decide explicitly; default for new builds: **dual-stack at the edge** (LB/CDN
  accept IPv6), IPv4 or dual-stack internally as provider support allows.
- IPv6 relieves IPv4 exhaustion (large pod networks, many VPCs) and avoids growing
  per-IPv4-address charges; egress-only internet gateways give outbound-only IPv6
  semantics like NAT-without-NAT-cost.
- IPv6 has **no NAT safety blanket**: every IPv6 address is globally routable, so
  SG/firewall discipline must be airtight before enabling on private tiers; audit
  for `::/0` rules exactly like `0.0.0.0/0`.

## Audit checklist

- [ ] Subnet tiers exist (public/private/isolated); no compute or data stores in
      public subnets; databases have no internet route.
- [ ] ≥ 2 AZs per tier; NAT per AZ in prod.
- [ ] No `0.0.0.0/0` or `::/0` ingress except 80/443 on edge LBs; no SSH/RDP from
      internet; internal rules reference SGs/tags, not broad CIDRs.
- [ ] CIDR plan documented/IPAM-tracked; no overlaps among connected networks.
- [ ] Egress controlled: isolated tier has none; sensitive envs have domain-level
      egress filtering; flow logs centralized.
- [ ] Private/gateway endpoints for storage, secrets, registries, logging in prod;
      matching private DNS zones attached.
- [ ] Topology: peering count sane or hub-spoke; prod/non-prod not mutually routable.
- [ ] Registrar: corporate account, MFA, transfer locks, auto-renew, team contacts.
- [ ] Split-horizon: no internal records in public zones; zones scanned for dangling
      records (sample-check CNAME targets exist and are yours).
- [ ] CAA records present; DNSSEC stance decided and recorded.
- [ ] All public certs auto-issued/renewed (ACM/ACME/managed); expiry alerts as
      backstop; nothing renewed by hand or living past current CA/B lifetime caps.
- [ ] LB health checks meaningful; draining configured; access logs on; WAF on
      internet-facing L7.
- [ ] CDN origins not directly reachable (OAC/origin auth verified by hitting origin
      directly); cache keys reviewed for poisoning/varying headers; private content
      uses signed URLs.
- [ ] DDoS stance recorded; rate limiting on public APIs; autoscale caps set.
- [ ] IPv6: stance recorded; if enabled, `::/0` rules audited.

# 04 — Compute Selection: Serverless, Containers, Kubernetes, VMs

Scope: choosing the compute layer, serverless patterns, Kubernetes architecture
essentials. Container image hardening: see sota-sandboxing. Deploy pipelines and
GitOps: see sota-devsecops.

## 1. The decision tree — simplest thing that meets requirements

Work down; stop at the first fit. Each step down adds operational surface you must
staff.

1. **Fully managed / no compute at all.** Static site → object storage + CDN. API
   that's pure CRUD → consider managed API + database integrations before writing
   glue compute.
2. **Serverless functions** (Lambda / Cloud Functions / Azure Functions) when:
   event-driven or spiky traffic, short tasks (AWS Lambda caps at 15 min/invocation,
   10 GB memory — verify other providers' current limits), team wants zero
   infrastructure ops, per-request pricing beats idle provisioning. Wrong when:
   long-lived connections (websockets at scale), sustained high constant load
   (always-on container is cheaper), heavy local state, > a few GB memory/GPU needs,
   or latency budgets that can't absorb cold starts. Two AWS options shift these
   break-evens: Lambda Managed Instances (GA since re:Invent 2025) runs functions on
   EC2-backed capacity with multi-request execution environments and Savings
   Plans/RI pricing — steady-load functions may now beat migrating to containers
   (decide with measured cost numbers); Lambda MicroVMs (2026) add suspendable
   isolated sandboxes up to 8 h for code-execution/agent workloads past the 15-min
   cap.
3. **Containers on managed runners** (Cloud Run / Fargate-on-ECS / Azure Container
   Apps) — **the default for standard web services and workers in 2026.** You bring
   an image; provider runs, scales, patches hosts. Choose when: HTTP services,
   background workers, anything that fits "stateless container + autoscale" without
   needing the K8s API.
4. **Kubernetes (managed: EKS/GKE/AKS)** only with a written justification, e.g.:
   you need the ecosystem (operators, service mesh, custom controllers, ML platforms
   like Ray/Kubeflow), multi-team platform with namespace tenancy, portability is a
   real requirement (not a vibe), or workload shapes managed runners can't express
   (DaemonSets, stateful sets with custom topology, GPU bin-packing). **And** you have
   ≥ 1 FTE-equivalent of platform capacity for upgrades (3 minor releases/year,
   ~14-month support window — being > 2 versions behind is an audit finding).
5. **VMs** for: lift-and-shift, licensed/legacy software, kernel/hardware control,
   stateful systems not yet on managed equivalents. VMs demand the patching, AMI
   pipeline, and autoscaling-group discipline everything above gives you for free.

Anti-pattern: **resume-driven Kubernetes** — a 5-service startup on a 3-node cluster
spends platform time it doesn't have; Cloud Run/Fargate would carry it to millions of
requests/day. Reverse anti-pattern: 40 microservices duct-taped across function
sprawl when the team actually needs an orchestrator.

Mixed estates are normal: functions for event glue, managed containers for services,
one K8s cluster for the platform workloads that justify it. Pick per workload, not
per company.

## 2. Serverless patterns

- **Idempotent handlers, always.** Every major trigger (queues, streams, schedulers)
  delivers at-least-once. Key side effects on an idempotency key (request/message
  ID) stored conditionally (DynamoDB conditional put / Firestore txn) — dedupe at
  the effect, not the entry point.
- **DLQs/failure destinations on every async consumer** (queue redrive policies,
  Lambda failure destinations, Pub/Sub dead-letter topics) with alerting on DLQ
  depth, and a documented redrive procedure. An async function without a DLQ deletes
  failures silently.

```hcl
# GOOD: queue with bounded retries into an alarmed DLQ
resource "aws_sqs_queue" "orders" {
  name                       = "orders"
  visibility_timeout_seconds = 90 # > 6x consumer timeout, per AWS guidance
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.orders_dlq.arn
    maxReceiveCount     = 5 # then it parks, visibly, instead of looping forever
  })
}
# BAD: no redrive_policy — poison messages retry until expiry, then vanish
```
- **Cold starts:** keep packages small, init outside the handler reused across
  invocations, avoid VPC-attach unless needed (it's cheap now but still adds
  config/ENI considerations), use provisioned concurrency / min instances only for
  measured latency-critical paths (it converts serverless pricing into
  always-on pricing — decide with numbers).
- **Concurrency is a real limit and a real weapon.** Account/regional concurrency is
  shared: one runaway function can starve the rest — set per-function reserved
  concurrency caps for anything triggered by unbounded sources. Also use concurrency
  caps to protect downstreams (DB connection limits — or use RDS Proxy/serverless
  drivers).
- **Orchestration: state machines for state, code for logic.** Multi-step workflows
  with retries/waits/human-approval/sagas → Step Functions / GCP Workflows / Azure
  Durable Functions, or AWS Lambda durable functions (GA Dec 2025: checkpointed
  steps/waits inside Lambda code, suspensions up to a year) — not hand-rolled
  retry loops sleeping inside a 15-minute invocation. If you're paying a function to
  `sleep()`, you chose the wrong orchestrator.
- Timeouts tuned to p99 + margin, not max — a 15-minute timeout on a 2-second
  function turns retries of a hung dependency into a cost and concurrency incident.
- Event contracts versioned; consumers tolerant readers (see sota-api-design).

## 3. Kubernetes architecture essentials

(Current upstream: v1.36, supported window v1.34–v1.36 as of mid-2026 — the latest
three minors; v1.33 reached EOL ~2026-06; verify your managed-provider version
offerings at design time.)

- **Managed control plane only** (EKS/GKE/AKS). Self-hosted control planes need a
  dedicated platform team and a reason.
- **Cluster topology:** separate prod and non-prod clusters (cheaper than perfect
  multi-tenancy isolation); within a cluster, namespace-per-team/service with
  ResourceQuotas + LimitRanges. Regional/multi-AZ control plane and node placement
  for prod.
- **Node pools by workload shape:** general on-demand pool for baseline,
  spot/preemptible pools for interruption-tolerant work (taint them; workloads
  opt in via tolerations), dedicated pools for GPU/memory-heavy. Prefer
  provider-managed provisioning (Karpenter on EKS, GKE NAP/Autopilot, AKS node
  autoprovisioning) over hand-tuned static ASGs.

### Autoscaling — three layers, configure all deliberately

| Layer | Tool | Rule |
|---|---|---|
| Pod count | HPA (or KEDA for event/queue-driven scaling to zero) | Scale on a metric that tracks load (RPS, queue depth, CPU as fallback); set sane min/max; behavior stabilization to stop flapping |
| Pod size | VPA / in-place resize | In-place pod resize is GA since v1.35 (resize CPU/mem without restart); use VPA in recommendation mode at minimum to ground requests in reality |
| Nodes | Cluster autoscaler / Karpenter / Autopilot | Must be on — HPA without node scaling = Pending pods at the worst time; set max nodes (cost cap) |

Don't point HPA and VPA at the same metric (CPU) in active mode simultaneously —
they fight; HPA on a throughput metric + VPA for sizing is the stable combo.

### Workload discipline — every production manifest

```yaml
# GOOD: the minimum acceptable production Deployment fragment
resources:
  requests: { cpu: 250m, memory: 512Mi }   # measured, not guessed; scheduler currency
  limits:   { memory: 512Mi }              # memory limit = request (predictable OOM);
                                           # CPU limit often omitted to avoid throttling — decide per workload
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector: { matchLabels: { app: checkout } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2          # or maxUnavailable — but NEVER minAvailable == replicas
  selector: { matchLabels: { app: checkout } }
```

- **No requests = no capacity math.** Missing requests is an audit finding: the
  scheduler bin-packs blind, evictions hit randomly, autoscaling lies.
- **PDB on everything with > 1 replica** — node upgrades and spot reclaim drain
  nodes; without PDBs, voluntary disruption takes out all replicas at once. A PDB
  requiring all replicas (`minAvailable` = replica count) blocks node upgrades
  forever — equally a finding.
- **Topology spread across zones** (and hosts) for anything claiming HA; replicas
  on one node are one failure domain.
- Probes: readiness ≠ liveness; liveness must not check dependencies (dependency
  blip → restart storm).
- Graceful shutdown: handle SIGTERM, `terminationGracePeriodSeconds` ≥ drain time,
  preStop hook to de-register before exit.

### Cluster operations

- **Upgrade cadence is a standing commitment:** upstream ships 3 minors/year with a
  ~14-month support window; managed providers add extended-support fees for
  laggards. Track ≤ 1 version behind your provider's default; test upgrades in
  non-prod with the same add-ons.
- Add-on sprawl is the hidden K8s cost: every controller (mesh, ingress, cert
  manager, secrets operator) is software you now operate. Adopt the minimal set;
  prefer provider-managed add-on variants.
- Cluster API endpoint: private (or IP-restricted) for prod; authn via cloud IAM
  (no static kubeconfig certs in CI — see sota-devsecops); RBAC mapped to IdP groups.

## 4. VMs (when you must)

- VMs live in autoscaling groups / MIGs / VMSS even at count=1 (self-healing,
  recreate-from-image), built from pipeline-produced images (Packer or provider
  image builder), no SSH mutation in place — access via SSM/IAP for debugging only.
- Patch via image rebake + rolling replace, not in-place fleet patching.
- Stateful VM workloads: pin per-AZ, snapshot schedules (rules/05), and a written
  failover story — an ASG won't save your database.

## Audit checklist

- [ ] Compute choice per workload has a justification; K8s clusters have a written
      reason + named platform owner; no orchestrator running 3 trivial services.
- [ ] Serverless: every async consumer has DLQ + alert + redrive runbook; handlers
      idempotent (check for idempotency keys on at-least-once triggers).
- [ ] Function timeouts/memory tuned vs p99; reserved concurrency caps on
      unbounded-trigger functions; provisioned concurrency justified by latency data.
- [ ] Multi-step workflows in a workflow engine (or durable functions), not
      sleep/retry loops in handler code.
- [ ] K8s version within provider standard support, ≤ 1 behind default; upgrade
      runbook exists and was exercised.
- [ ] All prod pods have resource requests; memory limits set; no
      requests-vs-usage gap > 2x sustained (check VPA recs / metrics).
- [ ] PDBs present on multi-replica workloads and none block drains
      (minAvailable < replicas).
- [ ] Topology spread or anti-affinity across zones on HA-claimed services.
- [ ] All three autoscaling layers configured; cluster/node autoscaler max set;
      spot pools tainted with tolerating workloads only.
- [ ] Probes sane (liveness dependency-free); SIGTERM handled; grace period ≥ drain.
- [ ] Cluster API endpoint private/IP-restricted; RBAC via IdP groups; quotas per
      namespace.
- [ ] VMs in ASGs/MIGs from pipeline-built images; no snowflakes (check instance
      age + provenance); no inbound SSH from internet.

# 05 — Data & Storage Architecture

Scope: object storage design, block/file/object selection, backup architecture,
encryption and KMS key strategy. Database engine selection, schema, replication
internals: see sota-databases. This file covers where data lives, how it's protected
at rest, and how it survives deletion — malicious or accidental.

## 1. Object storage design

- **Bucket-per-purpose, not bucket-as-filesystem.** One bucket = one data class +
  one access pattern + one lifecycle + one policy. Mixing public assets, internal
  exports, and PII in one bucket under prefix conventions means the loosest policy
  wins. Bucket names are global/guessable — never security-relevant.
- **Public access blocked, belt and suspenders.** Provider defaults now help (S3
  Block Public Access on + ACLs disabled for all new buckets since April 2023; GCP
  public access prevention; Azure blob public access disable) — but defaults protect
  only new resources. Enforce at account/org level (S3 BPA account setting guarded
  by SCP; GCP org policy `storage.publicAccessPrevention`; Azure Policy), and audit
  legacy buckets explicitly. Genuinely public content goes behind a CDN with origin
  access control (rules/03 §9), not a public bucket.
- **ACLs disabled everywhere** (S3 Object Ownership = bucket owner enforced).
  Policy-only access control; ACL grants found on legacy buckets are migration debt.
- **Versioning on** for any bucket whose objects you'd miss (protects against
  overwrite/delete), paired with lifecycle rules expiring noncurrent versions —
  versioning without expiry is an unbounded bill.
- **Lifecycle policy on every bucket, by design not retrofit:** transition to
  infrequent-access/archive tiers on access-pattern evidence (storage class
  analysis / intelligent tiering for unknown patterns), expire what has a retention
  end, always abort incomplete multipart uploads (silent cost leak), expire stale
  delete markers.

```hcl
# GOOD: baseline private bucket (AWS flavor; mirror on GCS/Azure)
resource "aws_s3_bucket_public_access_block" "b" {
  bucket                  = aws_s3_bucket.b.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
resource "aws_s3_bucket_versioning" "b" {
  bucket = aws_s3_bucket.b.id
  versioning_configuration { status = "Enabled" }
}
resource "aws_s3_bucket_lifecycle_configuration" "b" {
  bucket = aws_s3_bucket.b.id
  rule {
    id     = "baseline"
    status = "Enabled"
    abort_incomplete_multipart_upload { days_after_initiation = 7 }
    noncurrent_version_expiration { noncurrent_days = 90 }
  }
}
```

- Bucket policies: require TLS (`aws:SecureTransport`), pin org
  (`aws:PrincipalOrgID`) on shared buckets, restrict to VPC endpoints
  (`aws:SourceVpce`) for internal data planes. The TLS and org-perimeter
  conditions are better enforced org-wide once via an RCP (rules/01 §3); keep
  bucket policies for the bucket-specific conditions. Access logging on sensitive
  buckets to the central log account.
- Cross-account writes (log delivery, partner drops): bucket-owner-enforced
  ownership + explicit service principals with source-account/org conditions —
  never `Principal:"*"` with a prefix "restriction".

## 2. Block vs file vs object

| Use | Pick | Notes |
|---|---|---|
| Boot volumes, databases on VMs, low-latency random IO | **Block** (EBS / PD / Managed Disks) | Single-instance attach (mostly); size + IOPS/throughput are separate dials — provision from measurements; snapshots ≠ backup until copied out (see §3) |
| Shared POSIX across instances/pods, lift-and-shift NFS | **File** (EFS / Filestore / Azure Files) | Pay premium for shared semantics; check throughput mode pricing; don't use as a default because "it mounts everywhere" |
| Everything else — artifacts, media, exports, data lake, backups, static sites | **Object** | Default choice; design per §1 |

- Choosing file storage to share state between app replicas is usually an
  architecture smell — externalize state to object storage or a database.
- Block storage hygiene: delete-on-termination set deliberately, snapshots
  lifecycle-managed, unattached volumes reaped (cost finding, rules/06), encryption
  by default at the account/org setting.

## 3. Backup architecture: survive the account, not just the disk

Design backups against four failure classes: hardware/AZ loss, bad deploy/data bug,
accidental deletion, and **malicious actor with prod credentials** (ransomware). Most
backup setups handle the first two only.

- **3-2-1 translated to cloud: primary + same-region backup + cross-account (and
  cross-region per DR tier, rules/07) copy.** Replication is not backup —
  replication faithfully copies the corruption and the deletes. Versioning is not
  backup either (a principal with delete rights removes versions).
- **Cross-account is the ransomware control:** backup vault in a dedicated backup
  account (AWS Backup cross-account vault copy; GCS bucket in a backup project with
  distinct IAM; Azure Backup vault in isolated subscription) that prod admin
  credentials cannot reach.
- **Immutability where supported:** S3 Object Lock (compliance mode for regulated
  retention, governance otherwise), AWS Backup vault lock, GCS retention policy +
  bucket lock, Azure immutable blob storage / vault immutability. An admin-deletable
  backup fails the threat model.
- Coverage by data-tier policy, not per-team improvisation: every stateful resource
  is tagged with a backup tier; org-wide backup plans select by tag; an untagged
  stateful resource is a finding.
- **Restore is the product; backups are a means.** Tested restores (at least
  quarterly for tier-1 data) with measured restore time vs RTO; an unrestored backup
  is Schrödinger's backup. Verify backup of the *whole* unit of recovery (DB +
  config + KMS access), not just one piece.
- Retention matches policy/regulation explicitly — both minimum (compliance) and
  maximum (privacy/GDPR deletion duties). Infinite retention is a liability, not
  diligence.

## 4. Encryption defaults & KMS key strategy

Encryption-at-rest with provider-managed keys is table stakes (mostly default-on).
The real design decisions are about **key control and blast radius**:

- **Three key tiers — pick deliberately per data class:**
  1. **Provider-managed keys** (SSE-S3-style / Google default / Microsoft-managed):
     fine for low-sensitivity data; zero ops; no access separation — anyone with
     data-read IAM reads plaintext.
  2. **Customer-managed keys, CMK** (KMS / Cloud KMS / Key Vault): the default for
     confidential/regulated data. You get: key policy as a *second, independent*
     authorization layer, per-key CloudTrail/audit usage logs, rotation control,
     and a kill switch (disable key = data unreadable everywhere, including in
     stolen snapshots).
  3. **Hold-your-own/external key stores** (XKS, Cloud EKM, HYOK): only under
     explicit regulatory mandate — you inherit an availability dependency on your
     key store; treat as exceptional.
- **Key segmentation = blast radius:** key per environment per data domain (e.g.,
  `prod/payments-data`), not one org-wide key. One key for everything means one key
  grant reads everything and key compromise is total.
- **Separate key admins from data readers.** Key policy: security/platform team
  administers (no Decrypt); workload roles get Encrypt/Decrypt via grants; nobody
  holds both `kms:PutKeyPolicy`-class admin and broad decrypt. This is the control
  that makes stolen-snapshot exfiltration fail: copying an encrypted snapshot
  cross-account requires key access, not just data access.

```json
// BAD: one statement, everyone in the account, full key control + use
{ "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": "kms:*", "Resource": "*" }

// GOOD: admin and use split into distinct principals
{ "Sid": "KeyAdmins",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/platform-kms-admin" },
  "Action": ["kms:Put*", "kms:Update*", "kms:Enable*", "kms:Disable*",
              "kms:TagResource", "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion"],
  "Resource": "*" },
{ "Sid": "DataUsers",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/payments-api-runtime" },
  "Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey*"],
  "Resource": "*" }
```
- Enable **automatic key rotation** where supported (yearly rotation of backing key
  material); deletion only via scheduled deletion windows (and alarms on
  `ScheduleKeyDeletion` — it's a destruction primitive).
- KMS calls cost per-request at volume: use S3 Bucket Keys / data-key caching
  (envelope encryption) for high-request-rate paths instead of dropping to
  provider-managed keys for cost reasons.
- Secrets (API keys, passwords) do not live in object storage at all — see
  sota-secrets-management. Field-level/application-layer encryption for the most
  sensitive fields: see sota-code-security.

## 5. Data placement hygiene

- Residency: data-class tags (rules/01) + region restrictions (org guardrails) keep
  regulated data in approved regions; cross-region replication of regulated data is
  a compliance decision, not a convenience default.
- Minimize copies: every export/"temp" bucket/analytics dump is an unguarded replica.
  Data flows into the lake/warehouse through governed pipelines, with the same
  data-class controls as the source.
- Snapshot/AMI sharing: shared-to-public snapshots are a recurring breach class —
  audit for any snapshot/image shared outside the org; block publicly shared
  snapshots via org guardrail where available.

## Audit checklist

- [ ] Account/org-level public-access prevention enforced (not just per-bucket);
      zero public buckets/containers outside an approved, documented list.
- [ ] ACLs disabled (bucket-owner-enforced) on all buckets; legacy ACL grants gone.
- [ ] Every bucket has: versioning decision, lifecycle rules (incl. multipart
      abort + noncurrent expiry), TLS-required policy; sensitive buckets have
      access logging and endpoint/org conditions.
- [ ] No `Principal:"*"` in bucket/queue/key resource policies without strong
      conditions.
- [ ] Stateful resources carry backup-tier tags; org backup plans select by tag;
      sample-verify an actual recovery point exists for each tier-1 system.
- [ ] Backups copied cross-account; immutability/vault-lock on tier-1; verify prod
      admin role genuinely cannot delete backup copies.
- [ ] Restore tested within the last quarter for tier-1 (ask for the record and
      the measured time); retention matches stated policy, both min and max.
- [ ] Unattached volumes, orphaned snapshots, stale AMIs reaped or scheduled;
      no snapshots/images shared public or to unknown accounts.
- [ ] Default encryption on for block storage and DBs account-wide; CMKs used for
      confidential/regulated data classes.
- [ ] Key-per-domain segmentation (no single god key); key admins ≠ data readers in
      key policies; rotation on; ScheduleKeyDeletion alarmed.
- [ ] Regulated data regions match residency policy; exports/temp copies of
      sensitive data inventoried and governed.

# 06 — Cost Engineering (FinOps)

Scope: cost visibility, the big optimization levers, unit economics, anomaly
detection, and cost as a review gate. Cost is an architecture fitness function: a
design that meets every functional requirement at 4x the necessary spend is a wrong
design, the same way one that misses latency budgets is.

Pricing numbers change constantly — this file names the *mechanisms and traps*;
verify current rates against provider pricing pages before committing numbers to a
design.

## 1. Visibility before optimization

You cannot optimize what you cannot attribute. Sequence: allocate → show → set
targets → optimize. Optimizing an unattributed bill produces one heroic month and a
relapse.

- **Allocation foundation = account structure + tags.** Account/project-per-team
  (rules/01) gives you coarse attribution for free; mandatory `owner`, `service`,
  `env`, `cost-center` tags (activated as cost-allocation tags / labels) give
  per-workload resolution. Shared costs (network hubs, clusters, support plans,
  data transfer) need an explicit, documented split rule — even a crude one beats
  "platform absorbs it".
- **Export raw billing data** to a queryable store (AWS CUR/Data Exports → Athena;
  GCP billing export → BigQuery; Azure cost exports) — console dashboards don't
  answer real questions; tooling (Cost Explorer / Looker dashboards / Cloud
  Intelligence dashboards / third-party FinOps platforms; FOCUS-format exports for
  multi-cloud normalization) sits on top of the export.
- **Showback per team monthly, visible to the team and its management.** Chargeback
  only when the org's accounting supports it; showback captures most of the
  behavioral effect.
- **Kubernetes needs its own allocation layer** (OpenCost or provider cost
  allocation for EKS/GKE/AKS): the cluster is one line on the bill but many
  tenants; per-namespace/label cost with idle-cost assignment, or your "platform"
  line hides everyone's waste.
- Budgets with alerts on every account — including (especially) sandboxes; a
  forgotten sandbox GPU instance is a classic four-figure surprise.

## 2. The big levers, in order of typical yield

1. **Delete idle.** Unattached volumes/IPs, stopped-but-billed instances, idle LBs
   and NAT gateways, empty clusters, dev environments running nights/weekends
   (schedule them off — ~65% off for 12x5 usage), stale snapshots, never-queried
   logs at premium retention. Run an idle-sweep report monthly; auto-reap in
   sandbox.
2. **Rightsize.** Use provider recommenders (Compute Optimizer / GCP recommender /
   Azure Advisor) and utilization data; act on the recs (the finding is usually not
   "no data" but "recommendations ignored for a year"). In K8s, requests are the
   cost driver — requests >> usage bills you for air (rules/04). Also rightsize
   storage: gp2→gp3-class migrations, overprovisioned IOPS, premium tiers on dev
   disks.
3. **Commitment discounts for the stable floor.** Measure a baseline over ≥ 1
   quarter, then cover ~60–80% of it with Savings Plans / Committed Use Discounts /
   Reservations (start with compute-flexible commitments; instance-pinned only for
   truly static fleets). Review coverage and utilization quarterly. Committing to
   an unoptimized baseline locks in waste — rightsize first, commit second.
4. **Spot/preemptible for interruption-tolerant work** (batch, CI, stateless
   horizontally-scaled services with headroom): steep discounts in exchange for
   reclaim. Requirements: graceful SIGTERM handling, checkpointing for long jobs,
   diversified instance types, and never for singleton stateful workloads.
5. **Storage lifecycle** (rules/05): tiering + expiry + multipart abort. Logs and
   backups are the classic unbounded growers.
6. **Architecture-level moves** (the biggest but slowest lever): serverless for
   spiky loads, managed containers instead of an underutilized K8s cluster,
   batch aggregation instead of per-event processing, caching/CDN offload of
   origin traffic, ARM-based instances (Graviton/Axion/Cobalt-class) for
   compatible workloads.

## 3. The traps: egress and per-GB processing

Data transfer is the bill's dark matter — invisible in design diagrams, dominant in
some bills.

- **Egress to internet** bills per-GB; **cross-region** per-GB; **cross-AZ**
  typically per-GB in both directions (AWS); same-AZ private traffic free. Know
  which arrows in your architecture diagram cost money.
- **NAT gateway processing:** NAT bills per-hour AND per-GB processed. The classic
  incident: high-volume traffic to object storage routed through NAT — fix with
  free gateway endpoints (S3/DynamoDB) or private endpoints; an interface endpoint's
  per-hour+per-GB is usually far below NAT data processing for the same flow.
  "NAT data processing > NAT hourly cost" on a bill = misrouted traffic, go look.

```hcl
# GOOD: S3 gateway endpoint — removes S3 traffic from NAT, costs nothing
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.eu-west-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}
# BAD: no endpoint — every GET/PUT from private subnets pays NAT per-GB processing
```
- **Cross-AZ chatter:** chatty service meshes, K8s services hopping zones
  (use topology-aware routing for high-volume internal traffic), DB replicas
  crossing AZs by design (fine — that one's purchased availability).
- **CDN as cost control:** CDN egress rates undercut origin egress and absorb
  origin compute; cache-hit ratio is a cost metric (rules/03 §9).
- Cross-cloud / to-on-prem flows: per-GB both directions adds up — co-locate
  chatty components; move compute to data, not data to compute.
- Watch per-request pricing at volume: KMS calls (use bucket keys/data-key
  caching), object-storage PUT/GET on tiny-object workloads (aggregate), LB LCU
  dimensions, log ingestion per-GB (see sota-observability for telemetry cost
  discipline).

## 4. Unit economics

- Absolute spend is noise; **cost per unit of value** (per request, per active
  user, per tenant, per job, per GB processed) is signal. Rising spend with flat
  unit cost is growth; flat spend with rising unit cost is decay.
- Pick 1–3 unit metrics per service, compute monthly from the billing export +
  usage metrics, trend them on the team dashboard next to latency/error SLOs.
- Per-tenant cost matters for pricing and abuse: a tenant whose serving cost
  exceeds their revenue is a business bug; meter the expensive dimensions
  (storage, egress, compute-heavy API calls) per tenant where the architecture
  allows.

## 5. Anomaly detection and guardrails

- Enable provider anomaly detection (AWS Cost Anomaly Detection, GCP/Azure
  anomaly alerts) with alerts routed to the owning team, not a central inbox that
  rubber-stamps.
- Budgets: hard caps where the platform supports enforcement (sandbox), alert
  thresholds (50/80/100% forecast) elsewhere.
- Architectural cost-bombs need *technical* caps, not just alerts: autoscaling
  max sizes, function concurrency caps, log ingestion quotas, lifecycle rules.
  An alert fires after the money is gone; a cap prevents it. Recursive patterns
  (function writing to the bucket that triggers it; log-processing that logs)
  deserve explicit review.

## 6. Cost in PR review for infra changes

- Infra PRs state expected cost delta. Automate the estimate (Infracost or
  equivalent in CI — wiring belongs to sota-devsecops) so the number appears in
  review; reviewer checks the number like they check a migration.
- Cost review prompts for any infra change: What's the monthly steady-state? What
  scales with traffic, and what's the cap? What data crosses AZ/region/internet
  boundaries? What's the lifecycle/teardown story? Spot/commitment applicable?
- New-resource definition of done includes: tags (rules/01), lifecycle/expiry,
  autoscale caps, and an owner who will see its cost line.

## 7. Operating cadence

- Monthly: showback review per team; idle-sweep report; anomaly postmortems.
- Quarterly: commitment coverage/utilization review; rightsizing batch; unit-cost
  trend review; renegotiate/retier support plans as spend grows.
- Cost incidents get lightweight postmortems like outages: what spent, why no
  cap, which control was missing.

## Audit checklist

- [ ] Billing export to a queryable store exists and is used (ask for the last
      analysis it answered); dashboards per team.
- [ ] Cost-allocation tags activated; >90% of spend attributable to a team/service;
      shared-cost split rule documented; K8s per-namespace allocation in place.
- [ ] Budgets + anomaly alerts on every account, routed to owners; sandbox has
      hard caps/auto-reap.
- [ ] Idle inventory near zero: unattached volumes/IPs, idle LBs/NAT, off-hours
      schedules for non-prod (check a Tuesday-2am snapshot of dev spend).
- [ ] Rightsizing recommender findings reviewed within last quarter; K8s
      requests-vs-usage gap measured and bounded.
- [ ] Commitment coverage 60–80% of stable baseline; utilization > 90%; coverage
      reviewed quarterly; no commitments bought against un-rightsized baseline.
- [ ] Spot used for batch/CI/stateless where tolerable (with SIGTERM handling);
      justification where it is not.
- [ ] No NAT-processing-dominated bills (gateway/private endpoints for high-volume
      service traffic); cross-AZ/region flows known and intentional; CDN hit
      ratio tracked.
- [ ] Storage lifecycle rules on logs/backups/buckets (rules/05); log ingestion
      quotas/retention tiers set.
- [ ] Unit-cost metrics defined and trended for the top services.
- [ ] Autoscale maxes, concurrency caps, and quota ceilings set on elastic
      resources (the bill cannot scale unbounded).
- [ ] Infra PRs carry cost deltas (sample recent PRs for evidence).

# 07 — Resilience & Disaster Recovery

Scope: RTO/RPO-driven design, DR strategy tiers, multi-AZ vs multi-region decisions,
dependency mapping, quota management, graceful degradation, DR testing. Backup
mechanics: rules/05. Alerting/SLO machinery: sota-observability.

## 1. Start from RTO/RPO, not from architecture

- Every system gets two numbers assigned by the business owner, recorded next to
  the system inventory:
  - **RTO** — max tolerable time to restore service.
  - **RPO** — max tolerable data loss window.
- Without declared numbers, engineers silently assume either "whatever
  backup-restore gives" (and the business assumes zero-downtime) or gold-plate
  everything multi-region. Both are failures. "Undeclared RTO/RPO for a
  production system" is itself an audit finding (Medium).
- Tier the portfolio — typically 3–4 tiers — and map each tier to a DR strategy
  and a test cadence. Most systems belong in lower tiers; the conversation that
  puts them there is the deliverable.

## 2. DR strategy tiers

| Strategy | Typical RTO | Typical RPO | Cost shape | Mechanics |
|---|---|---|---|---|
| **Backup & restore** | hours–day+ | hours (last backup) | Storage only | Cross-region/account backups (rules/05) + IaC to rebuild; restore runbook |
| **Pilot light** | tens of min–hours | minutes | Data replication + dormant minimal core | Data continuously replicated; core infra (DB replica, AMIs/images, network) exists but scaled to ~0; scale up on declare |
| **Warm standby** | minutes | seconds–minutes | Scaled-down full copy running | Full stack live at reduced capacity in second region; scale + cut traffic over |
| **Active-active** | ~seconds–minutes | ~0 | ≥ 2x + engineering complexity | Both regions serve; data layer is the hard part (conflict resolution or partitioned writes); failover = weight shift |

- RTO/RPO of the **data layer dominates**: stateless compute redeploys in minutes
  from IaC + registries; the database replica lag, promotion time, and backup
  granularity set your real numbers. Design data first.
- **Failover must be tested-automatic or runbook-manual — decide.** Auto-failover
  that's never been exercised will surprise you (split-brain, flapping); manual
  failover needs a decision tree (who declares, on what evidence) or you'll lose
  your RTO to meetings. Record the declare-authority by name/role.
- DR region requirements: IaC fully region-parameterized; images/artifacts
  replicated; secrets/KMS available in target region (multi-region keys or
  per-region keys provisioned — a backup you can't decrypt in-region is
  decoration); DNS/traffic-management plan (low TTLs, health-checked routing or
  global LB); quotas pre-raised in the standby region (cold quotas are the classic
  pilot-light failure — everyone's DR plan targets the same region).

```hcl
# GOOD: DNS failover wired before the incident (Route 53 sketch)
resource "aws_route53_health_check" "primary" {
  fqdn              = "api-primary.example.com"
  type              = "HTTPS"
  resource_path     = "/healthz" # meaningful readiness, see rules/03 §8
  failure_threshold = 3
}
resource "aws_route53_record" "api_primary" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = "api.example.com"
  type            = "CNAME"
  ttl             = 60 # low TTL: failover applies in ~minutes, not cache-lifetime
  set_identifier  = "primary"
  records         = ["api-primary.example.com"]
  health_check_id = aws_route53_health_check.primary.id
  failover_routing_policy { type = "PRIMARY" }
}
# BAD: TTL 86400 on the record you plan to repoint during a disaster
```

## 3. Multi-AZ is the default; multi-region is a justified exception

- **Multi-AZ, always, for prod:** LBs spanning zones, ASGs/MIGs across ≥ 2 (prefer
  3) AZs, multi-AZ managed databases, K8s topology spread (rules/04). AZ failure
  is common enough to design for by default and cheap enough to absorb (the main
  cost is cross-AZ traffic — accept it for prod).
- **Multi-region active-active is justified by:** regulatory mandate, genuinely
  global latency requirements, or revenue-per-minute that dwarfs the 2x+ cost and
  the permanent complexity tax (data consistency, double the deploy surface,
  config drift between regions). For most systems, the honest answer is multi-AZ +
  pilot-light/warm-standby DR.
- The most common resilience lie: "multi-region" where region B has never served
  production traffic. Untested standby = backup-restore with extra steps and extra
  cost. If you pay for warm standby, route a trickle of real traffic or fail over
  on schedule.
- Account for **control-plane vs data-plane** behavior in regional incidents:
  during a region's bad day, creating new resources (control plane) often fails
  while existing ones (data plane) keep running — DR designs that require
  mass-creating resources mid-incident are betting on the part most likely to be
  degraded. Pilot light pre-creates the skeleton for exactly this reason. Prefer
  static stability: pre-provisioned capacity over launch-on-failure.

## 4. Dependency mapping

- You cannot state an RTO without knowing the dependency graph. For each tier-1
  system, maintain the list of: cloud services used (per region), internal
  upstreams, third-party SaaS (auth provider! payment! email!), and DNS/CDN/cert
  chain — each annotated with "what happens here if it's down" and "does our DR
  region remove this dependency or duplicate it".
- **Your availability ceiling is the weakest hard dependency.** A multi-region app
  with single-region auth, a single payment provider, or one CDN has that
  provider's availability, not yours. Decide per dependency: accept (document),
  degrade (see §6), or dual-source (rarely worth it; auth and DNS are the usual
  candidates).
- Hidden circulars to hunt in audits: deploy pipeline hosted on the infrastructure
  it deploys; secrets manager needed to boot the secrets manager's dependencies;
  SSO required to reach the console during an SSO outage (break-glass, rules/02
  §7); runbooks stored on the wiki that's down.

## 5. Quotas, limits, and capacity

- Cloud quotas are soft until they're not — at 2am during failover. For every prod
  account: know the top 10 quotas you actually consume (instances per family,
  vCPUs, EIPs, LB count, API rates, function concurrency), monitor utilization
  against limits (provider quota dashboards + alerts at ~70%), and pre-request
  headroom in DR regions for tier-1 capacity.
- Quota increases take hours–days and need a support tier that can answer —
  factor both into RTO. Anything that "requests quota on failover" has already
  failed.
- Also bound the other direction: autoscale maxes and concurrency caps (rules/06)
  so one system's incident can't exhaust shared account limits and become
  everyone's incident — or isolate noisy systems into their own accounts
  (rules/01).

## 6. Graceful degradation architecture

Total failure should be the last stop, not the first. Build the intermediate
states:

- **Classify features critical vs sheddable** per service; expose kill switches /
  feature flags ops can flip without a deploy (flag system itself must fail open
  to defaults).
- **Standard patterns:** timeouts on every remote call (no default-infinite
  clients); retries with backoff + jitter and a budget (retry storms turn brownout
  into blackout); circuit breakers around flaky dependencies; load shedding /
  admission control when saturated (fast 429/503 beats slow death); queues as
  buffers between tiers (accept writes, defer processing); serve stale
  cache/read-only mode when the write path is down.
- Degraded modes are product decisions — "checkout works, recommendations blank"
  needs a product sign-off before the incident, not during.
- Implementation detail of these patterns in code: see sota-async-concurrency and
  sota-api-design; this file owns the requirement that the modes exist.

## 7. Test it: game days and DR exercises

An untested DR plan is a document, not a capability. Cadence by tier:

- **Tier-1:** full DR exercise (actual regional failover or full restore-and-serve)
  at least annually; component-level chaos (AZ evacuation, instance/pod kill, DB
  failover, dependency blackhole) quarterly; backup restore test quarterly
  (rules/05).
- **Game day discipline:** written scenario + hypothesis; defined blast radius and
  abort criteria; run in prod-like (or prod, with leadership sign-off and
  controlled scope — start in staging, graduate); measure actual RTO/RPO vs
  declared; file and fix the gaps. The measured numbers replace the aspirational
  ones in the inventory.
- Use fault-injection tooling where available (AWS FIS, chaos engineering tools,
  Azure Chaos Studio) instead of hand-run destruction; tooling gives repeatability
  and stop conditions.
- Also exercise the humans: paging works, runbooks current, declare-authority
  known, status comms templates exist. Half of blown RTOs are coordination, not
  technology.

## Audit checklist

- [ ] System inventory exists with declared RTO/RPO per system, business-owner
      signed; tiers mapped to DR strategies.
- [ ] Prod workloads multi-AZ end to end: LB, compute groups/topology spread, DB
      multi-AZ; no tier-1 singleton in one zone.
- [ ] Multi-region claims verified: standby actually serves traffic in tests;
      IaC region-parameterized; images, secrets, and KMS keys available in DR
      region; DNS/traffic failover defined with low TTLs.
- [ ] Data layer numbers known: replica lag, promotion time, backup cadence —
      consistent with declared RPO/RTO (do the math; flag fantasy numbers).
- [ ] Failover mode decided (auto vs manual) and exercised; declare-authority
      named; runbook accessible during an outage of the primary (not on the
      affected wiki).
- [ ] Backups cross-account/cross-region per tier with tested restores (rules/05);
      restore time measured against RTO.
- [ ] Dependency map for tier-1 systems incl. third parties; each hard dependency
      has accept/degrade/dual-source decision; no hidden circulars (deploy, auth,
      secrets, DNS).
- [ ] Quota utilization monitored with alerts; DR-region quotas pre-raised for
      tier-1 capacity; support plan adequate for incident-time escalation.
- [ ] Degradation built: timeouts/retry budgets/circuit breakers/load shedding on
      critical paths; kill switches exist and were flipped in a test; degraded
      modes product-approved.
- [ ] Last DR exercise within cadence for each tier; measured RTO/RPO recorded;
      gap actions closed. Ask for evidence, not assurances.