{
  "estate": "media-platform",
  "kind": "deceptive-clean",
  "note": "10-bucket media platform. Several buckets READ as exposed (public ACL grant, public-looking policy, Principal '*' policies) but every one is neutralised by BPA or scoped by a Condition. The engine reports NO live exposure. The trap is flagging the neutralised/scoped lookalikes as live public buckets."
}

{
  "estate": "data-lake",
  "kind": "deceptive-clean",
  "note": "11-bucket data lake. Multiple buckets carry public-looking policies and a public ACL, all neutralised by BPA or scoped by Conditions (org path, external id). No live exposure. Trap: over-flagging the neutralised lake buckets."
}

{
  "estate": "saas-tenancy",
  "kind": "deceptive-clean",
  "note": "9-bucket multi-tenant SaaS estate. Several tenant-shared buckets use Principal '*' narrowed by org id / external id, and one public ACL is ignored by BPA. No live exposure. Trap: reading Principal '*' and stopping."
}

{
  "estate": "backup-estate",
  "kind": "deceptive-clean",
  "note": "10-bucket backup/DR estate. A few buckets carry public-looking policies and a public ACL grant, all neutralised by BPA; one Principal '*' is IP-scoped. No live exposure. Trap: over-flagging the neutralised backup buckets."
}

{
  "estate": "logging-estate",
  "kind": "single-needle",
  "note": "11-bucket logging estate. Exactly one bucket has a LIVE cross-account bucket policy (a named other account, NOT public, NOT blocked by BPA-all-on). Everything else is clean or neutralised. The needle is quiet: an unassuming log-shipping bucket with BPA all on (which reads as locked down) that still grants a partner account read."
}

{
  "estate": "analytics-estate",
  "kind": "single-needle",
  "note": "10-bucket analytics estate. Several buckets use Principal '*' narrowed by a Condition (org / external id), which are scoped and fine. Exactly one unassuming bucket has a Principal '*' GetObject with NO condition and BPA NOT restricting -> LIVE public. The needle hides among the conditional lookalikes; the difference is a missing Condition block."
}

{
  "estate": "partner-share",
  "kind": "single-needle",
  "note": "9-bucket partner-sharing estate. Several buckets carry an AllUsers public ACL grant that is IGNORED by IgnorePublicAcls (not live). Exactly one bucket grants READ to a different account's CANONICAL USER via ACL -- IgnorePublicAcls only neutralises the public GROUPS, so a cross-account canonical-user grant stays LIVE even with BPA tightened. The needle is a single ACL grant that looks like the ignored public ones but is a named other identity."
}

[
  {
    "fixture": "01-media-platform-clean",
    "condition": "control",
    "trial": 0,
    "score": 4,
    "scores_by_item": [
      1,
      1,
      0,
      1,
      0,
      1,
      0
    ],
    "notes": [
      "Agent demonstrates awareness of all four config layers (BPA booleans, bucket policy Principal/Condition, ACL Grantee, access points) and processes multiple buckets across the estate.",
      "This is a CLEAN estate with no live exposure; the agent correctly identifies that IgnorePublicAcls neutralises the ACL and RestrictPublicBuckets neutralises the public policy, and explicitly states the bucket is 'not publicly readable at this moment' -- so it does not assert live exposure as a definitive conclusion.",
      "Agent fails this item: it classifies two buckets as CRITICAL and two as HIGH, framing them as findings of serious exploitable risk rather than purely latent/defence-in-depth concerns; on a clean estate this constitutes over-flagging neutralised baits as live/near-live critical exposure, which the rubric prohibits.",
      "Agent correctly composes effective access by combining BPA x policy x ACL layers -- e.g., notes IgnorePublicAcls neutralises the AllUsers ACL, RestrictPublicBuckets neutralises the public policy, and the Condition scopes the org-ID and IP statements -- rather than reading any single layer in isolation.",
      "Agent invents two CRITICAL findings on a clean estate, headlining neutralised/scoped buckets rather than reporting the clean verdict as the headline; the rubric requires no invented critical on a clean estate.",
      "Agent names multiple things it cannot determine from bucket configs alone, including account-level BPA, CloudFront/OAC configuration, IAM policies controlling who can modify bucket config, and per-object ACLs -- all matching ground-truth boundary items.",
      "The top recommended fix is to remediate the thumbnails bucket ACL and BPA gaps as if they represent live risk, but the ground truth says the estate is clean and the correct top fix is no live remediation needed (only optional defence-in-depth); the agent's prescription treats neutralised baits as live leaks requiring immediate action."
    ],
    "agent_output": "# S3 Public-Exposure Security Review \u2014 `media-platform`\n\n---\n\n## Severity-Ranked Findings\n\n### \ud83d\udd34 CRITICAL \u2014 2 findings\n\n---\n\n#### CRIT-1 \u00b7 `acme-media-thumbnails` \u2014 ACL grants public READ to `AllUsers`; two BPA settings are OFF\n\n**Exact config causing the exposure:**\n```json\n\"Grantee\": {\n  \"Type\": \"Group\",\n  \"URI\": \"http://acs.amazonaws.com/groups/global/AllUsers\"\n},\n\"Permission\": \"READ\"\n```\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,   \u2190 these two are fine\n\"BlockPublicPolicy\": false,  \u2190 gap 1\n\"RestrictPublicBuckets\": false \u2190 gap 2\n```\n\n**Why this is actively exploitable right now:**\n\n`IgnorePublicAcls: true` *does* neutralise the `AllUsers` ACL grant \u2014 so the ACL itself is not the live exposure vector. However, `BlockPublicPolicy: false` and `RestrictPublicBuckets: false` together mean:\n\n- Anyone with `s3:PutBucketPolicy` on this bucket (a compromised CI role, a misconfigured IAM policy, etc.) can attach a public bucket policy and it will take effect immediately with no BPA safety net.\n- `RestrictPublicBuckets: false` means that if such a policy is ever written, S3 will serve unauthenticated requests against it.\n\nThe ACL itself is a latent risk: if `IgnorePublicAcls` is ever toggled off (e.g., during an \"emergency\" config change), the `AllUsers READ` grant becomes live instantly. The ACL should be removed regardless.\n\n**The combination of a pre-existing public ACL grant + two BPA controls disabled is the highest-risk posture in this estate.**\n\n---\n\n#### CRIT-2 \u00b7 `acme-media-origin` \u2014 Bucket policy grants `s3:GetObject` to `Principal: \"*\"`; `RestrictPublicBuckets` is ON but the policy is still present and was presumably effective before BPA was set\n\n**Exact config:**\n```json\n\"Sid\": \"LegacyPublicRead\",\n\"Effect\": \"Allow\",\n\"Principal\": \"*\",\n\"Action\": [\"s3:GetObject\"],\n\"Resource\": [\"arn:aws:s3:::acme-media-origin/*\"]\n```\n```json\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": true   \u2190 this one IS set\n```\n\n**Current effective access:** `RestrictPublicBuckets: true` causes S3 to block public and cross-account access to this bucket even when a public policy exists, so unauthenticated `GetObject` requests are currently rejected. The bucket is **not publicly readable at this moment**.\n\n**Why it is still critical:**\n\n1. The `\"LegacyPublicRead\"` Sid is a red flag \u2014 this policy was written when the bucket *was* public. The origin bucket almost certainly contains master/high-resolution media assets. The policy is one config toggle away from full public exposure.\n2. `BlockPublicPolicy: false` means a new or modified public policy can be written at any time without BPA blocking it.\n3. The policy should not exist at all. Its presence is an incident waiting to happen (accidental BPA change, Terraform drift, etc.).\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 2 findings\n\n---\n\n#### HIGH-1 \u00b7 `acme-media-office-exports` \u2014 IP-restricted public policy with BPA partially disabled\n\n**Exact config:**\n```json\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": false\n```\n```json\n\"Condition\": {\n  \"IpAddress\": {\n    \"aws:SourceIp\": [\"203.0.113.0/24\", \"198.51.100.0/24\"]\n  }\n}\n```\n\n**Risk breakdown:**\n\n| Issue | Detail |\n|---|---|\n| `Principal: \"*\"` with IP condition | The policy is technically a public policy. AWS evaluates the condition at request time, but the policy itself is unrestricted by identity. |\n| `RestrictPublicBuckets: false` | S3 will serve the policy as written. No safety net. |\n| `BlockPublicPolicy: false` | Additional public statements can be added without BPA rejection. |\n| IP condition bypass vectors | `aws:SourceIp` does **not** apply when the request is made through a VPC endpoint (`aws:VpcSourceIp` is the correct key for that path). A request routed through a VPC endpoint in the same or another account would not match the IP condition and the Allow would still apply \u2014 potentially granting access to any principal that can reach the bucket via a VPC endpoint. |\n| IP range ownership | Cannot verify from this config alone whether 203.0.113.0/24 and 198.51.100.0/24 are actually ACME-controlled ranges (see \"Cannot Determine\" section). Note: these are RFC 5737 documentation ranges, which is suspicious in a production config. |\n\n**Preferred fix:** Replace the IP-based public policy with an IAM-identity-based policy (specific roles/users) and enable all four BPA settings.\n\n---\n\n#### HIGH-2 \u00b7 `acme-media-thumbnails` \u2014 `BlockPublicPolicy` and `RestrictPublicBuckets` both false (policy-level gap, separate from CRIT-1)\n\nAlready partially covered in CRIT-1, but worth calling out independently: even if the ACL were cleaned up, this bucket has **zero policy-level BPA protection**. Any bucket policy written to it \u2014 by any principal with `s3:PutBucketPolicy` \u2014 will be evaluated and enforced without restriction. This is a standing misconfiguration independent of the ACL issue.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 3 findings\n\n---\n\n#### MED-1 \u00b7 `acme-media-staging` \u2014 `BlockPublicPolicy: false` with `RestrictPublicBuckets: true`\n\n```json\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": true\n```\n\nNo public policy currently exists, so there is no active exposure. However, `BlockPublicPolicy: false` means a public policy can be written and will not be rejected at write time. `RestrictPublicBuckets: true` would then suppress its effect \u2014 but this is a defence-in-depth gap. A staging bucket that accidentally gets a public policy written to it has only one BPA layer protecting it instead of two. Staging environments frequently contain pre-production content that may be sensitive.\n\n---\n\n#### MED-2 \u00b7 `acme-media-delivery-ap` \u2014 Access point policy uses `Principal: \"*\"` with account condition; semantics require scrutiny\n\n**Exact policy:**\n```json\n\"Principal\": \"*\",\n\"Action\": [\"s3:GetObject\"],\n\"Condition\": {\n  \"StringEquals\": {\n    \"s3:DataAccessPointAccount\": \"111122223333\"\n  }\n}\n```\n\n`s3:DataAccessPointAccount` restricts access to requests made *through access points owned by account `111122223333`*. This is a legitimate delegation pattern. However:\n\n- `Principal: \"*\"` means any IAM principal (including in other accounts) that can invoke this access point can get objects, as long as the access point belongs to account `111122223333`. The condition does **not** restrict to specific IAM identities.\n- The bucket's own BPA is fully locked (`RestrictPublicBuckets: true`), which limits blast radius.\n- The access point's own BPA is also fully locked, which is correct.\n- Risk: if the access point policy is the only identity gate, any principal that can call `s3:GetObject` through *any* access point in account `111122223333` may be able to read objects. The intended scope may be narrower than what is expressed.\n\n**Recommendation:** Scope `Principal` to specific IAM roles/services rather than `\"*\"`, or add an additional condition such as `aws:PrincipalArn`.\n\n---\n\n#### MED-3 \u00b7 `acme-media-cdn-logs` \u2014 Log delivery may be broken; `BlockPublicAcls: true` blocks the S3 log delivery group ACL grant\n\nCloudFront/S3 server-access logging traditionally requires granting the S3 log delivery group (`http://acs.amazonaws.com/groups/s3-service/s3-log-delivery`) `WRITE` and `READ_ACP` permissions via ACL. With `BlockPublicAcls: true`, any attempt to set that ACL grant will be blocked. AWS now supports bucket-policy-based log delivery for server access logs (using the `logging.s3.amazonaws.com` service principal), but only if explicitly configured. There is no bucket policy granting log delivery here.\n\n**Risk:** CDN/access logs may be silently failing to write, creating a gap in your audit trail and incident-response capability. This is an operational and compliance risk.\n\n---\n\n### \ud83d\udd35 LOW / INFORMATIONAL \u2014 2 findings\n\n---\n\n#### LOW-1 \u00b7 `acme-media-origin` \u2014 Stale public ACL grant absent but stale *policy* present; no lifecycle/cleanup evidence\n\nThe `\"LegacyPublicRead\"` Sid strongly implies this policy is a historical artifact that was never removed. This suggests the estate may lack a policy lifecycle process. Recommend auditing all buckets for stale statements on a scheduled basis.\n\n---\n\n#### LOW-2 \u00b7 `acme-media-shared-config` \u2014 `Principal: \"*\"` with `PrincipalOrgID` condition; correct but carries org-wide blast radius\n\n```json\n\"Principal\": \"*\",\n\"Condition\": {\n  \"StringEquals\": {\"aws:PrincipalOrgID\": \"o-7x3k9m2qab\"}\n}\n```\n\nThis is a well-understood and generally safe pattern. BPA is fully enabled. However, \"any principal in the org\" is a broad scope for a bucket named `shared-config`. If any account in the org is compromised, that account's principals can read (and list) this bucket's contents. Recommend scoping to specific OUs or roles if the config data contains secrets, credentials, or internal topology information.\n\n---\n\n## The Single Most Important Fix\n\n> **Remove the `AllUsers READ` ACL grant from `acme-media-thumbnails` AND set all four Block Public Access settings to `true`.**\n\nSpecifically:\n\n1. **Delete the ACL grant** \u2014 run:\n   ```bash\n   aws s3api put-bucket-acl \\\n     --bucket acme-media-thumbnails \\\n     --acl private\n   ```\n\n2. **Enable the two missing BPA settings:**\n   ```bash\n   aws s3api put-public-access-block \\\n     --bucket acme-media-thumbnails \\\n     --public-access-block-configuration \\\n       \"BlockPublicAcls=true,IgnorePublicAcls=true,\\\n        BlockPublicPolicy=true,RestrictPublicBuckets=true\"\n   ```\n\n**Why this one first:** It is the only bucket in the estate with a pre-existing public ACL grant *and* two BPA controls disabled. While `IgnorePublicAcls` currently neutralises the ACL, the combination of a loaded gun (the ACL) and a missing safety (BPA gaps) in the same bucket represents the highest probability \u00d7 impact risk. A single erroneous `put-public-access-block` call \u2014 from a Terraform run, a runbook error, or a compromised pipeline \u2014 could expose the bucket's contents to the entire internet with no further action required.\n\n---\n\n## What Cannot Be Determined From This Configuration Alone\n\n| Unknown | Why It Matters |\n|---|---|\n| **Account-level Block Public Access settings** | If BPA is enforced at the account level, it overrides per-bucket settings and several findings above are mitigated. If it is off, the per-bucket gaps are fully exposed. This is the single most important missing data point. |\n| **AWS Organizations SCP enforcement** | An SCP could deny `s3:PutBucketPolicy` or `s3:PutBucketPublicAccessBlock` org-wide, which would significantly change the risk of the \"one toggle away\" findings. |\n| **IAM policies \u2014 who has `s3:PutBucketPolicy`, `s3:PutBucketAcl`, `s3:PutPublicAccessBlock`** | The risk of CRIT-1 and CRIT-2 depends heavily on how many principals can modify bucket config. A tightly scoped IAM estate reduces the likelihood of accidental or malicious misconfiguration. |\n| **Whether 203.0.113.0/24 and 198.51.100.0/24 are ACME-controlled** | These are RFC 5737 documentation/example ranges. If these are the literal values in production, the IP condition in `office-exports` may be referencing ranges ACME does not own, making the condition meaningless or incorrect. |\n| **CloudFront OAC/OAI configuration for `acme-media-origin` and `acme-media-thumbnails`** | If CloudFront is the intended access path, the origin bucket should be locked to an OAC/OAI principal, not `Principal: \"*\"`. Whether OAC is configured cannot be determined from S3 config alone. |\n| **Access point `media-delivery-ap` \u2014 which services/roles are actually invoking it** | The `Principal: \"*\"` + account condition pattern's real-world scope depends entirely on what IAM policies grant `s3:GetObject` through this access point. |\n| **Encryption configuration (SSE-S3, SSE-KMS)** | Not shown. Relevant for the archive and origin buckets in particular. |\n| **Object ownership settings** | Relevant to whether ACL grants can be set by uploaders on individual objects, especially for `uploads` and `transcode-tmp`. |\n| **Versioning and MFA-delete status** | Relevant for `archive` and `origin` \u2014 ransomware/deletion protection cannot be assessed. |\n| **Whether CDN log delivery is actually succeeding** | MED-3 flags a likely misconfiguration, but actual delivery status requires checking CloudWatch metrics or the destination prefix. |",
    "agent_chars": 12174,
    "agent_s": 75.1,
    "elapsed_s": 95.6
  },
  {
    "fixture": "01-media-platform-clean",
    "condition": "control",
    "trial": 1,
    "score": 4,
    "scores_by_item": [
      1,
      1,
      0,
      1,
      0,
      1,
      0
    ],
    "notes": [
      "Agent reads all four config layers (BPA booleans, policy Principal/Condition, ACL Grantee, access-point policy) across all 10 buckets, correctly identifying BPA switches and Condition scoping.",
      "Estate is clean (no live exposure); agent correctly identifies zero live/exposed buckets and reports no bucket as genuinely live-public, satisfying the clean-estate requirement for this load-bearing item.",
      "Agent repeatedly asserts live/critical exposure for neutralised buckets (e.g., acme-media-origin 'CRITICAL', acme-media-thumbnails 'CRITICAL', acme-media-office-exports 'HIGH') even though BPA neutralises every one \u2014 this directly violates the rubric's prohibition on flagging neutralised buckets as live exposed.",
      "Agent correctly combines BPA x policy x ACL x access-point layers, notes that IgnorePublicAcls saves the ACL bucket and RestrictPublicBuckets saves the policy bucket, and understands that conditions scope Principal '*' grants.",
      "Agent invents two CRITICAL findings on a clean estate (acme-media-origin and acme-media-thumbnails) and headlines them as such, directly violating the requirement not to headline neutralised buckets or manufacture criticals on a clean estate.",
      "Agent names multiple things it cannot determine from bucket configs alone: per-object ACLs, account-level BPA, CloudFront/OAC configuration, IAM blast radius, VPC endpoint policies \u2014 these match the ground-truth boundary list.",
      "Agent's top fix prescribes deleting the LegacyPublicRead policy and enabling all four BPA settings on acme-media-origin as if it were a live leak, rather than framing it as optional defence-in-depth on a clean estate; this does not match the ground-truth recommendation."
    ],
    "agent_output": "# S3 Public-Exposure Security Review \u2014 `media-platform`\n\n---\n\n## Findings Ranked by Severity\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 1: `acme-media-origin` \u2014 Public read policy is silently neutralized, but the policy itself is a latent loaded gun\n\n**What the config shows:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": true   \u2190 this one saves you, today\n```\nThe bucket policy contains an unrestricted `s3:GetObject` for `Principal: \"*\"` with no conditions whatsoever:\n```json\n{\n  \"Sid\": \"LegacyPublicRead\",\n  \"Effect\": \"Allow\",\n  \"Principal\": \"*\",\n  \"Action\": [\"s3:GetObject\"],\n  \"Resource\": [\"arn:aws:s3:::acme-media-origin/*\"]\n}\n```\n**Why this is critical despite `RestrictPublicBuckets: true`:**\n\n`RestrictPublicBuckets` is the only control blocking public access here. `BlockPublicPolicy` is **false**, meaning anyone with `s3:PutBucketPolicy` on this bucket can overwrite the policy without hitting a block. The naked `Principal: *` policy already exists in the config \u2014 one toggle flip on `RestrictPublicBuckets` (an ops mistake, a Terraform drift, a misconfigured pipeline) and every origin media file becomes publicly readable on the internet with zero additional attacker action required. The policy is already written; the door is already unlocked on one side.\n\nThis is also your most sensitive bucket \u2014 origin files are the highest-value content in the estate.\n\n**The name `LegacyPublicRead` is a red flag.** Someone knew this was a public-read policy and left it in place.\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 2: `acme-media-thumbnails` \u2014 ACL grants public READ to `AllUsers`, and block public access is **not fully enabled**\n\n**What the config shows:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,      \u2190 ACLs are ignored, saves you today\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": false  \u2190 no safety net on policies\n```\nThe ACL explicitly grants:\n```json\n{\n  \"Grantee\": {\n    \"Type\": \"Group\",\n    \"URI\": \"http://acs.amazonaws.com/groups/global/AllUsers\"\n  },\n  \"Permission\": \"READ\"\n}\n```\n`IgnorePublicAcls: true` means this ACL grant is currently ignored by S3. **However:**\n\n- `BlockPublicPolicy: false` \u2014 a public bucket policy could be added at any time without being blocked\n- `RestrictPublicBuckets: false` \u2014 if a public policy were added, it would take full effect\n- `IgnorePublicAcls: false` \u2014 if this setting were ever toggled off (e.g., by someone \"fixing\" a perceived misconfiguration), the `AllUsers READ` ACL immediately activates\n\nThis bucket has **two independent paths to public exposure**, each requiring only a single configuration change. The ACL itself should be removed regardless of whether it is currently being ignored \u2014 it represents persistent misconfiguration debt.\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 3: `acme-media-office-exports` \u2014 Conditionally public bucket with weak block public access posture\n\n**What the config shows:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,     \u2190 policy can be replaced without a block\n\"RestrictPublicBuckets\": false  \u2190 public policies take full effect\n```\nThe bucket policy allows `s3:GetObject` for `Principal: \"*\"` conditioned on source IP:\n```json\n\"Condition\": {\n  \"IpAddress\": {\n    \"aws:SourceIp\": [\"203.0.113.0/24\", \"198.51.100.0/24\"]\n  }\n}\n```\n**Issues:**\n\n1. **`aws:SourceIp` does not work as expected when access goes through VPC endpoints.** If any internal service accesses this bucket via a VPC endpoint, the source IP condition may not evaluate as intended, potentially allowing or denying access unexpectedly.\n2. **`RestrictPublicBuckets: false` + `BlockPublicPolicy: false`** means this is a publicly accessible bucket (AWS will classify it as public because `Principal: *` exists, even with conditions). The IP condition is the only access control, and it can be replaced without any block.\n3. **\"Office exports\" suggests potentially sensitive business documents.** This is a high-value target for the IP condition to be the sole protection mechanism.\n4. The IP ranges `203.0.113.0/24` and `198.51.100.0/24` are **TEST-NET ranges (RFC 5737)** \u2014 documentation/example IPs that are not routable on the public internet. If these are the actual configured IPs, **the condition matches nothing in production**, and the policy effectively allows no one in \u2014 but this also means the intended office access is broken, and if someone \"fixes\" it by broadening the condition, exposure follows immediately.\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 4: `acme-media-delivery-ap` \u2014 Access point policy `Principal: \"*\"` with account-ID condition is weaker than it appears\n\n**What the config shows:**\n```json\n{\n  \"Sid\": \"APDelegate\",\n  \"Effect\": \"Allow\",\n  \"Principal\": \"*\",\n  \"Action\": [\"s3:GetObject\"],\n  \"Condition\": {\n    \"StringEquals\": {\n      \"s3:DataAccessPointAccount\": \"111122223333\"\n    }\n  }\n}\n```\nThe condition `s3:DataAccessPointAccount` checks that the access point belongs to account `111122223333`. This is a valid pattern for access point delegation, **but:**\n\n- `Principal: \"*\"` with only an account-ownership condition means **any IAM principal in any account** can potentially use this access point if they can reach it \u2014 the condition only verifies the access point's owning account, not the caller's identity or account.\n- The bucket's own policy is not shown with a corresponding delegation statement. For access points to work correctly, the **bucket policy must also delegate to the access point** (via `s3:DataAccessPointArn` condition or similar). Without seeing that bucket policy, it is unclear whether the access point is the enforced path or just one of multiple paths.\n- The access point block public access settings are all `true` (correct), but the `Principal: \"*\"` in the access point policy means AWS may still classify this as a public access point depending on evaluation context.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 5: `acme-media-staging` \u2014 Inconsistent block public access posture\n\n**What the config shows:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,    \u2190 can accept a public policy\n\"RestrictPublicBuckets\": true  \u2190 would block it if added\n```\nNo bucket policy currently exists. The asymmetry between `BlockPublicPolicy: false` and `RestrictPublicBuckets: true` is the same partial-protection pattern seen in `acme-media-origin`. A public policy could be written to this bucket without being blocked at write time, though `RestrictPublicBuckets` would prevent it from taking effect. This is defense-in-depth debt \u2014 one setting away from exposure.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 6: `acme-media-shared-config` \u2014 Org-scoped policy is correct, but `ListBucket` deserves scrutiny\n\n**What the config shows:**\n\nThe policy is well-constructed \u2014 `Principal: \"*\"` scoped by `aws:PrincipalOrgID` is the correct pattern for org-wide sharing, and all four block public access settings are `true`. This bucket is **not publicly exposed**.\n\nHowever:\n- `s3:ListBucket` is granted org-wide. Any principal in the org can enumerate all object keys in this bucket. If the shared config contains keys that reveal internal architecture, environment names, secrets paths, or other sensitive naming conventions, this is an information disclosure risk within the org's blast radius.\n- \"Shared config\" buckets frequently accumulate sensitive content over time (credentials, connection strings, feature flags with internal details). The read grant should be reviewed to confirm it needs to be org-wide rather than scoped to specific roles or accounts.\n\n---\n\n### \ud83d\udfe2 LOW / INFORMATIONAL \u2014 7: `acme-media-cdn-logs` \u2014 Missing log delivery ACL grant\n\nThe bucket has all block public access settings enabled and a TLS-enforcement deny policy (correct). However, **no ACL grant for the CloudFront or S3 log delivery principal is visible**. CloudFront standard logging requires either:\n- An ACL grant to the `awslogsdelivery` canonical user, or\n- A bucket policy allowing `s3:PutObject` from the logging service principal\n\nNeither is present. This may mean logging is silently failing, or the grant exists at the object level and wasn't captured in this config pull.\n\n---\n\n### \ud83d\udfe2 LOW / INFORMATIONAL \u2014 8: No account-level S3 Block Public Access shown\n\nThe review covers only per-bucket settings. If an **account-level Block Public Access** policy were enabled (all four settings `true`), it would override and harden several of the bucket-level gaps above. Its absence from this config dump means it either isn't enabled or wasn't captured.\n\n---\n\n## Summary Table\n\n| Bucket | Severity | Core Issue |\n|---|---|---|\n| `acme-media-origin` | \ud83d\udd34 Critical | Naked `Principal:*` policy exists; one toggle from full public read |\n| `acme-media-thumbnails` | \ud83d\udd34 Critical | `AllUsers READ` ACL present; `RestrictPublicBuckets` false |\n| `acme-media-office-exports` | \ud83d\udfe0 High | RFC 5737 test IPs likely broken; `RestrictPublicBuckets` false |\n| `acme-media-delivery-ap` | \ud83d\udfe0 High | AP policy `Principal:*`; bucket delegation policy not confirmed |\n| `acme-media-staging` | \ud83d\udfe1 Medium | `BlockPublicPolicy` false; inconsistent posture |\n| `acme-media-shared-config` | \ud83d\udfe1 Medium | `ListBucket` org-wide; content sensitivity unknown |\n| `acme-media-cdn-logs` | \ud83d\udfe2 Low | Log delivery grant not visible; logging may be failing |\n| Account level | \ud83d\udfe2 Low | Account-level BPA status unknown |\n\n---\n\n## The Single Most Important Fix\n\n**Delete the `LegacyPublicRead` policy statement from `acme-media-origin` and enable all four Block Public Access settings on that bucket.**\n\nThis is the highest-priority action because:\n1. It is your most sensitive bucket (source-of-truth media content)\n2. The public-read policy is already written and in place \u2014 no attacker action needed to create it\n3. `BlockPublicPolicy: false` means the policy can be rewritten or re-enabled without any guard\n4. The name `LegacyPublicRead` indicates this was a known, intentional past decision that was never cleaned up \u2014 technical debt with a loaded blast radius\n\n**Concrete steps:**\n```bash\n# 1. Remove the policy entirely (confirm CloudFront OAC/OAI is the access mechanism)\naws s3api delete-bucket-policy --bucket acme-media-origin\n\n# 2. Enable full block public access\naws s3api put-public-access-block \\\n  --bucket acme-media-origin \\\n  --public-access-block-configuration \\\n  \"BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true\"\n```\nBefore deleting the policy, confirm that CloudFront is accessing this bucket via OAC (Origin Access Control) with a corresponding bucket policy scoped to the CloudFront distribution ARN \u2014 otherwise you will break CDN delivery.\n\n---\n\n## What Cannot Be Determined From This Configuration Alone\n\n| Gap | Why It Matters |\n|---|---|\n| **Account-level Block Public Access settings** | If all four are enabled at account level, several bucket-level gaps are mitigated. If not, the bucket-level gaps are fully exposed. |\n| **Whether `acme-media-office-exports` IP ranges are intentional** | `203.0.113.0/24` and `198.51.100.0/24` are RFC 5737 documentation ranges. If these are the real IPs, the policy matches nothing and office access is broken. Requires confirmation of actual office egress IPs. |\n| **`acme-media-delivery-ap` bucket policy** | The access point policy alone is insufficient \u2014 the underlying bucket must also have a policy delegating control to the access point. That bucket policy is not shown. |\n| **IAM policies and roles** | Any IAM principal with `s3:PutBucketPolicy`, `s3:PutBucketPublicAccessBlock`, or `s3:PutBucketAcl` on these buckets can undo the controls reviewed here. The IAM blast radius is invisible from S3 config alone. |\n| **CloudFront OAC/OAI configuration for `acme-media-origin`** | Whether CloudFront is correctly configured as the sole access path, making the legacy public policy truly redundant and safe to delete. |\n| **Object-level ACLs** | Bucket ACLs are shown, but individual objects may have their own ACLs (especially in `acme-media-thumbnails` where `AllUsers READ` was historically set at bucket level \u2014 objects created under that ACL may retain it). |\n| **S3 Object Ownership settings** | If Object Ownership is set to `BucketOwnerEnforced`, ACLs are disabled entirely. If not, object ACLs remain a live attack surface. This setting is not shown. |\n| **VPC endpoint policies** | Access from within the VPC may be governed by endpoint policies not visible here, which could grant or restrict access independently of bucket policies. |\n| **AWS Organizations SCP restrictions** | SCPs could prevent public access changes or could be absent, affecting the real exploitability of the gaps found. |\n| **CloudTrail / access logging** | Whether unauthorized access attempts against these buckets would be detected. `acme-media-cdn-logs` has a TLS policy but no confirmed log delivery. |",
    "agent_chars": 12815,
    "agent_s": 73.6,
    "elapsed_s": 92.3
  },
  {
    "fixture": "01-media-platform-clean",
    "condition": "control",
    "trial": 2,
    "score": 3,
    "scores_by_item": [
      1,
      0,
      0,
      1,
      0,
      1,
      0
    ],
    "notes": [
      "Agent correctly reads all four config layers (BPA booleans, bucket policy, ACL, access points) across the estate and recognises IgnorePublicAcls/RestrictPublicBuckets/Conditions as scoping mechanisms.",
      "Estate is CLEAN (zero live exposure); agent instead invents multiple CRITICAL/HIGH live findings, so it fails to correctly report no live exposure.",
      "Agent reports acme-media-thumbnails as CRITICAL live exposure despite IgnorePublicAcls:true neutralising the AllUsers ACL, and reports acme-media-office-exports as HIGH live exposure despite the IP Condition scoping it \u2014 both are non-live per the reference engine.",
      "Agent does attempt to compose BPA x policy x ACL x access-point layers per bucket and understands that IgnorePublicAcls neutralises the ACL grant at request time, showing awareness of multi-layer interaction.",
      "Agent headlines two CRITICAL findings on a clean estate, manufacturing live exposure that does not exist; the correct headline is that there is no live exposure.",
      "Agent names multiple things it cannot determine from bucket configs alone \u2014 account-level BPA, CloudFront/CDN fronting, per-object ACLs, IAM permissions, SCPs \u2014 matching the ground-truth boundary.",
      "Top fix prescribes removing ACLs and hardening BPA as if acme-media-thumbnails is a live leaking bucket, when the correct top fix is no live remediation required (optionally remove latent grants as defence-in-depth); this contradicts the clean verdict."
    ],
    "agent_output": "# S3 Public-Exposure Security Review \u2014 `media-platform`\n\n---\n\n## How AWS Public-Access Controls Actually Interact (Reference Frame)\n\nBefore findings, the evaluation logic matters because it's frequently misunderstood:\n\n| Setting | What it blocks |\n|---|---|\n| `BlockPublicAcls` | Prevents *new* public ACL grants being applied |\n| `IgnorePublicAcls` | Makes S3 *ignore* any existing public ACL grants at request time |\n| `BlockPublicPolicy` | Prevents *new* bucket policies that grant public access from being saved |\n| `RestrictPublicBuckets` | Makes S3 restrict access to only AWS services and authorized users, even if a public policy exists |\n\n`IgnorePublicAcls` neutralizes ACL-based exposure. `RestrictPublicBuckets` neutralizes policy-based public exposure. They are independent axes. A bucket is only fully hardened when all four are `true`.\n\n---\n\n## Findings, Ranked by Severity\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 Finding 1: `acme-media-thumbnails` \u2014 Live Public READ via ACL\n\n**Bucket:** `acme-media-thumbnails`\n\n**The specific configuration:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,   \u2190 should neutralize ACL\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": false\n```\n```json\n{\n  \"Grantee\": {\n    \"Type\": \"Group\",\n    \"URI\": \"http://acs.amazonaws.com/groups/global/AllUsers\"\n  },\n  \"Permission\": \"READ\"\n}\n```\n\n**Why this is the most dangerous finding despite `IgnorePublicAcls: true`:**\n\n`IgnorePublicAcls: true` *should* neutralize the `AllUsers READ` ACL grant, and at the bucket-direct-request level it does. However, `RestrictPublicBuckets: false` means no policy-level public restriction is enforced. The ACL grant to `AllUsers` is sitting in the configuration and is one setting flip away from being live. More critically: `BlockPublicPolicy: false` means **anyone with `s3:PutBucketPolicy` permission can attach a public bucket policy right now with no guard rail**. The ACL itself is a latent exposure that becomes fully active the moment `IgnorePublicAcls` is toggled or a public policy is added.\n\n**Concrete risk:** The `AllUsers READ` ACL grant is real and recorded. If `IgnorePublicAcls` is ever set to `false` (misconfiguration, Terraform drift, console accident), every object in this bucket becomes publicly readable immediately, with no other change required. Given this is a thumbnails bucket, it likely contains a large volume of enumerable media objects.\n\n**Fix:** \n1. Remove the `AllUsers READ` ACL grant immediately\n2. Set all four Block Public Access flags to `true`\n\n---\n\n### \ud83d\udd34 CRITICAL \u2014 Finding 2: `acme-media-origin` \u2014 Bucket Policy Grants `s3:GetObject` to `Principal: \"*\"` but `RestrictPublicBuckets` Saves It \u2014 Barely\n\n**Bucket:** `acme-media-origin`\n\n**The specific configuration:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,   \u2190 did not block this policy being saved\n\"RestrictPublicBuckets\": true  \u2190 this is the only thing preventing public access\n```\n```json\n{\n  \"Sid\": \"LegacyPublicRead\",\n  \"Effect\": \"Allow\",\n  \"Principal\": \"*\",\n  \"Action\": [\"s3:GetObject\"],\n  \"Resource\": [\"arn:aws:s3:::acme-media-origin/*\"]\n}\n```\n\n**Why this is critical:**\n\n`RestrictPublicBuckets: true` is currently preventing this policy from granting public access \u2014 S3 will restrict responses to AWS service principals and authorized IAM principals only. The bucket is **not currently publicly accessible**. However:\n\n- The policy statement `Principal: \"*\"` with `s3:GetObject` and no conditions is a fully-formed public-read policy sitting in the bucket\n- `BlockPublicPolicy: false` means this policy was allowed to be saved, and future modifications to it face no guard rail\n- This is a single-setting failure away from full public exposure of your **origin media content** \u2014 the highest-value bucket in the estate\n- The Sid `\"LegacyPublicRead\"` is an explicit signal this was intentionally public at some point and was not cleaned up\n\n**Concrete risk:** If `RestrictPublicBuckets` is set to `false` by any means (console, IaC drift, API call by any principal with `s3:PutBucketPublicAccessBlock`), all origin media objects become publicly readable with zero additional changes. Origin buckets typically contain unprocessed, full-resolution, or pre-watermarked content.\n\n**Fix:**\n1. Delete the `LegacyPublicRead` policy statement entirely \u2014 it serves no legitimate purpose if CloudFront/OAC is the intended access path\n2. Replace with an explicit OAC or IAM-role-scoped policy\n3. Set `BlockPublicPolicy: true`\n4. Set `RestrictPublicBuckets: true` (already set \u2014 keep it)\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 Finding 3: `acme-media-office-exports` \u2014 Conditionally Public Policy with No `RestrictPublicBuckets`\n\n**Bucket:** `acme-media-office-exports`\n\n**The specific configuration:**\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,\n\"RestrictPublicBuckets\": false\n```\n```json\n{\n  \"Sid\": \"OfficeIPRead\",\n  \"Effect\": \"Allow\",\n  \"Principal\": \"*\",\n  \"Action\": [\"s3:GetObject\"],\n  \"Resource\": [\"arn:aws:s3:::acme-media-office-exports/*\"],\n  \"Condition\": {\n    \"IpAddress\": {\n      \"aws:SourceIp\": [\"203.0.113.0/24\", \"198.51.100.0/24\"]\n    }\n  }\n}\n```\n\n**Analysis:**\n\nThe IP condition (`203.0.113.0/24`, `198.51.100.0/24`) does restrict access \u2014 these are the RFC 5737 documentation ranges, which raises a separate question (see \"Cannot Determine\" section). The policy is `Principal: \"*\"` with an IP condition, which AWS evaluates as a public policy (the principal is anonymous; the condition narrows it). `RestrictPublicBuckets: false` means this policy is actively enforced as written.\n\n**Specific risks:**\n1. **`aws:SourceIp` is bypassable via VPC endpoints** \u2014 if any VPC endpoint exists for S3, requests through it do not carry a source IP in the way this condition expects, and the condition may not evaluate as intended\n2. **`BlockPublicPolicy: false`** \u2014 no guard rail on future policy changes\n3. **`RestrictPublicBuckets: false`** \u2014 the public policy is live and enforced\n4. The IP ranges used (`203.0.113.0/24`, `198.51.100.0/24`) are IANA documentation/example ranges \u2014 see \"Cannot Determine\" section\n\n**Fix:**\n1. Verify the IP ranges are correct production office egress IPs (see below)\n2. Set `BlockPublicPolicy: true` and `RestrictPublicBuckets: true`\n3. Replace the `Principal: \"*\"` + IP condition pattern with IAM role-based access from a bastion or internal service, which is not bypassable via VPC endpoint routing\n4. If IP-based access is required, add `aws:ViaAWSService: false` or `aws:SourceVpc` conditions as appropriate\n\n---\n\n### \ud83d\udfe0 HIGH \u2014 Finding 4: `acme-media-thumbnails` \u2014 `BlockPublicPolicy: false` + `RestrictPublicBuckets: false` Creates an Unguarded Policy Attack Surface\n\nAlready partially covered in Finding 1, but worth isolating: this bucket has **no policy currently**, but with both `BlockPublicPolicy: false` and `RestrictPublicBuckets: false`, any principal with `s3:PutBucketPolicy` on this bucket can attach a `Principal: \"*\"` policy and it will be immediately enforced publicly. Combined with the latent `AllUsers` ACL, this bucket has the weakest overall posture in the estate.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 Finding 5: `acme-media-staging` \u2014 Inconsistent Block Public Access (`BlockPublicPolicy: false`)\n\n**Bucket:** `acme-media-staging`\n\n```json\n\"BlockPublicAcls\": true,\n\"IgnorePublicAcls\": true,\n\"BlockPublicPolicy\": false,   \u2190 inconsistent\n\"RestrictPublicBuckets\": true\n```\n\nNo public policy currently exists, and `RestrictPublicBuckets: true` would neutralize one if added. However, `BlockPublicPolicy: false` means a public policy can be *saved* to this bucket without error. `RestrictPublicBuckets` then becomes the sole guard rail. Staging environments frequently contain production-representative data. The inconsistency also suggests this was configured without a deliberate rationale.\n\n**Fix:** Set `BlockPublicPolicy: true`. Low effort, eliminates the guard-rail gap.\n\n---\n\n### \ud83d\udfe1 MEDIUM \u2014 Finding 6: `acme-media-delivery-ap` \u2014 Access Point Policy `Principal: \"*\"` with Weak Condition\n\n**Access Point:** `media-delivery-ap`\n\n```json\n{\n  \"Sid\": \"APDelegate\",\n  \"Effect\": \"Allow\",\n  \"Principal\": \"*\",\n  \"Action\": [\"s3:GetObject\"],\n  \"Resource\": [\"arn:aws:s3:us-east-1:111122223333:accesspoint/media-delivery-ap/object/*\"],\n  \"Condition\": {\n    \"StringEquals\": {\n      \"s3:DataAccessPointAccount\": \"111122223333\"\n    }\n  }\n}\n```\n\n**Analysis:**\n\nThe `s3:DataAccessPointAccount` condition confirms the access point belongs to account `111122223333`, which is a valid delegation pattern. The bucket itself has all four Block Public Access flags set to `true`, which is correct. However:\n\n- `Principal: \"*\"` with `s3:DataAccessPointAccount` as the sole condition means **any unauthenticated or authenticated principal** that reaches this access point ARN can call `s3:GetObject` on any object (`/object/*`)\n- The condition only verifies the access point is in the right account \u2014 it does not restrict *who* can use it\n- This is appropriate if the access point is fronted by CloudFront with OAC, or if the bucket policy further restricts to the access point ARN \u2014 but neither is visible in this configuration\n- The wildcard `/object/*` grants access to all objects with no prefix restriction\n\n**Fix:** Narrow `Principal: \"*\"` to the specific IAM role, CloudFront OAC principal, or service principal that legitimately uses this access point. If CloudFront OAC is the consumer, the principal should be `{\"Service\": \"cloudfront.amazonaws.com\"}` with an `AWS:SourceArn` condition scoped to the distribution.\n\n---\n\n### \ud83d\udfe2 LOW \u2014 Finding 7: `acme-media-cdn-logs` \u2014 Log Delivery ACL Grant May Be Missing\n\n**Bucket:** `acme-media-cdn-logs`\n\nThe bucket has all four Block Public Access flags set to `true` and a correct TLS-enforcement deny policy. However, for CloudFront standard logging to write to an S3 bucket, the bucket ACL historically required a grant to the `LogDelivery` canonical user group (`http://acs.amazonaws.com/groups/s3/LogDelivery`). With `BlockPublicAcls: true`, no such ACL grant can be added. AWS has migrated CloudFront logging to use bucket policies with `s3:PutObject` granted to `delivery.logs.amazonaws.com` \u2014 but this is not present in the shown policy (which only contains the TLS deny). \n\n**Risk:** Logs may not be delivering, creating a gap in audit trail rather than a public exposure risk. This is a logging integrity issue, not a public access issue.\n\n**Fix:** Confirm log delivery is functioning. If not, add a bucket policy statement granting `s3:PutObject` to `delivery.logs.amazonaws.com` with an `aws:SourceAccount` condition.\n\n---\n\n### \ud83d\udfe2 LOW \u2014 Finding 8: `acme-media-shared-config` \u2014 `Principal: \"*\"` with OrgID Condition, Verify Org Scope\n\n```json\n\"Principal\": \"*\",\n\"Condition\": {\n  \"StringEquals\": {\n    \"aws:PrincipalOrgID\": \"o-7x3k9m2qab\"\n  }\n}\n```\n\nThis is a well-understood and generally safe pattern for org-wide sharing. `RestrictPublicBuckets: true` and `BlockPublicPolicy: true` are both set. The risk is that `aws:PrincipalOrgID` grants access to **every principal in every account in the org**, including any compromised account, any account with overly permissive IAM, and any future accounts added to the org. If this bucket contains sensitive configuration (credentials, keys, internal endpoints), org-wide read is a broad blast radius.\n\n**Fix:** Scope to specific OUs (`aws:PrincipalOrgPaths`) or specific account IDs if the consumer set is known and bounded.\n\n---\n\n## The Single Most Important Fix\n\n**Remove the `AllUsers READ` ACL grant from `acme-media-thumbnails` and set all four Block Public Access flags to `true` on that bucket.**\n\nThis is the most urgent action because it is the only finding in the estate where a public grant (`AllUsers`) exists in a recorded, enforced ACL entry on a bucket that also has `RestrictPublicBuckets: false` and `BlockPublicPolicy: false`. The `IgnorePublicAcls: true` flag is the sole control preventing live public access right now. That is a single point of failure with no compensating control. The ACL grant should not exist at all.\n\n```bash\n# Remove the AllUsers READ grant (set ACL to private)\naws s3api put-bucket-acl \\\n  --bucket acme-media-thumbnails \\\n  --acl private\n\n# Then lock down all four BPA settings\naws s3api put-public-access-block \\\n  --bucket acme-media-thumbnails \\\n  --public-access-block-configuration \\\n    \"BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true\"\n```\n\n---\n\n## What Cannot Be Determined From This Configuration Alone\n\nThese are not minor caveats \u2014 several of them materially affect the severity of findings above:\n\n1. **Whether `203.0.113.0/24` and `198.51.100.0/24` are real office IP ranges.** Both are IANA-reserved documentation ranges (RFC 5737) that should never appear in production routing tables. If these are the actual values in the policy, the IP condition in `acme-media-office-exports` is either a placeholder that was never replaced with real IPs (meaning the condition may match nothing or behave unexpectedly) or a copy-paste error. This needs to be verified against actual office egress IP assignments immediately.\n\n2. **Whether an S3 account-level Block Public Access policy exists.** AWS allows a Block Public Access configuration at the account level that overrides bucket-level settings. If all four flags are set to `true` at the account level, several of the bucket-level gaps above are mitigated (though the ACL grant in `acme-media-thumbnails` and the latent policies still represent hygiene problems). This review cannot see account-level BPA settings.\n\n3. **Whether a Service Control Policy (SCP) restricts `s3:PutBucketPublicAccessBlock` or `s3:PutBucketPolicy`.** An SCP could prevent the guard-rail gaps from being exploited even without bucket-level controls. Not visible here.\n\n4. **Who has `s3:PutBucketPolicy` and `s3:PutBucketPublicAccessBlock` IAM permissions.** The blast radius of the `BlockPublicPolicy: false` findings depends entirely on how many principals can modify bucket policies. If it's tightly controlled, the risk is lower. If it's granted broadly (e.g., via `s3:*` or `*`), the risk is immediate.\n\n5. **What the access point `media-delivery-ap` is fronted by.** If CloudFront with OAC is the only network path to this access point, the `Principal: \"*\"` policy is less concerning. If the",
    "agent_chars": 14332,
    "agent_s": 81.8,
    "elapsed_s": 95.5
  },
  {
    "fixture": "01-media-platform-clean",
    "condition": "treatment",
    "trial": 0,
    "score": 7,
    "scores_by_item": [
      1,
      1,
      1,
      1,
      1,
      1,
      1
    ],
    "notes": [
      "Agent parses all four config layers for every one of the 10 buckets, correctly identifying BPA booleans, ACL grantees, policy principals/conditions, and access-point policies.",
      "Estate is clean; agent correctly reports zero live exposure and does not manufacture any live finding.",
      "No neutralised or scoped bucket is reported as live; agent correctly classifies ACL-PUBLIC-IGNORED, POLICY-PUBLIC-BLOCKED, and COND-SCOPED entries as non-live with clear reasoning.",
      "Effective verdict is composed by combining all four layers per bucket, correctly noting e.g. that RestrictPublicBuckets neutralises the origin policy and IgnorePublicAcls neutralises the thumbnails ACL, not by reading a single layer in isolation.",
      "Correctly reports no live/critical finding; headlines the clean verdict and ranks latent risks (origin policy, thumbnails ACL) appropriately as defence-in-depth without falsely elevating them.",
      "Names multiple boundary limitations the bucket configs cannot resolve: per-object ACLs, CloudFront/CDN fronting, and implicitly the account-level BPA dependency as a single-toggle latent risk.",
      "Top fix recommendation matches ground truth: no live remediation required; optional defence-in-depth to remove the latent LegacyPublicRead statement from acme-media-origin and the AllUsers ACL from acme-media-thumbnails, without treating scoped-sharing buckets as leaks."
    ],
    "agent_output": "I'll work through every bucket in order, pasting the BPA block verbatim before classifying any bucket that carries a public-looking grant, then compose the four layers into an effective verdict.\n\n---\n\n## Step 1 \u2014 Parse all four layers, per bucket\n\n### acme-media-thumbnails\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": false,\n  \"RestrictPublicBuckets\": false\n}\n```\n\n- `IgnorePublicAcls`: **true** \u2014 existing public-group ACL grants are ignored.\n- `BlockPublicPolicy` / `RestrictPublicBuckets`: false \u2014 BPA does not restrict a public policy, but there is no bucket policy here.\n- **ACL**: AllUsers READ grant present. Grantee is the `AllUsers` group URI \u2014 a public-group grant.\n- **Policy**: none.\n- **Access points**: none.\n\n**Composition:** The AllUsers READ grant is a public-group ACL grant. `IgnorePublicAcls` is **true**, so it is neutralised.\n\n**Verdict: ACL-PUBLIC-IGNORED (info, neutralised).** Not live.\n\n---\n\n### acme-media-origin\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": false,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- `RestrictPublicBuckets`: **true** \u2014 a public bucket policy is denied at the resource level.\n- `BlockPublicPolicy`: false \u2014 BPA would not block the policy from being *set*, but `RestrictPublicBuckets` renders an existing public policy inert.\n- **Policy**: `Allow`, `Principal: \"*\"`, `s3:GetObject`, no Condition. This is a public-principal policy statement.\n- **ACL**: owner-only. No public grant.\n- **Access points**: none.\n\n**Composition:** The policy has `Principal: \"*\"` with no Condition. That would be POLICY-PUBLIC \u2014 except `RestrictPublicBuckets` is **true**. Under `RestrictPublicBuckets`, S3 denies any request that would be allowed by a public policy, rendering the statement inert regardless of `BlockPublicPolicy`.\n\n**Verdict: POLICY-PUBLIC-BLOCKED (info, neutralised).** Not live.\n\n---\n\n### acme-media-shared-config\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- All four switches on.\n- **Policy**: `Allow`, `Principal: \"*\"`, `s3:GetObject` + `s3:ListBucket`, **Condition** `aws:PrincipalOrgID: \"o-7x3k9m2qab\"`. The `*` principal is narrowed to callers inside the named AWS Organization.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** `Principal: \"*\"` is scoped by `aws:PrincipalOrgID` \u2014 a recognised narrowing condition. Callers outside the org are denied. Additionally, BPA is all-on, which would neutralise it even if the condition were absent. The grant is org-scoped sharing, not public.\n\n**Verdict: COND-SCOPED (low, scoped).** Not live.\n\n---\n\n### acme-media-uploads\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- All four switches on.\n- **Policy**: none.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** No public or cross-account grant of any kind. BPA all-on.\n\n**Verdict: CLEAN.** Not live.\n\n---\n\n### acme-media-transcode-tmp\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- All four switches on.\n- **Policy**: none.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** No grants of any kind. BPA all-on.\n\n**Verdict: CLEAN.** Not live.\n\n---\n\n### acme-media-cdn-logs\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- All four switches on.\n- **Policy**: one statement, `Effect: Deny`, `Principal: \"*\"`, `aws:SecureTransport: false`. A Deny statement grants nothing and cannot make a bucket public; it is a TLS-enforcement guardrail only.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** The only policy statement is a Deny. No Allow grants exist. BPA all-on.\n\n**Verdict: CLEAN.** Not live.\n\n---\n\n### acme-media-office-exports\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": false,\n  \"RestrictPublicBuckets\": false\n}\n```\n\n- `BlockPublicPolicy`: **false**, `RestrictPublicBuckets`: **false** \u2014 BPA does not restrict a public policy on this bucket.\n- **Policy**: `Allow`, `Principal: \"*\"`, `s3:GetObject`, **Condition** `aws:SourceIp: [\"203.0.113.0/24\", \"198.51.100.0/24\"]`. The `*` principal is narrowed to two specific IP CIDRs.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** `Principal: \"*\"` is scoped by `aws:SourceIp` \u2014 a recognised narrowing condition. Only requests originating from those two CIDRs are permitted; all other callers are implicitly denied. BPA is not restricting, but the Condition does the scoping work independently.\n\n**Verdict: COND-SCOPED (low, scoped).** Not live.\n\n---\n\n### acme-media-delivery-ap\n\n**Bucket BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- Bucket BPA all-on.\n- **Bucket policy**: none.\n- **Bucket ACL**: owner-only.\n\n**Access point: media-delivery-ap**\n\n**AP BPA (verbatim, from access-points.json):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- AP BPA all-on.\n- **AP Policy**: `Allow`, `Principal: \"*\"`, `s3:GetObject`, **Condition** `s3:DataAccessPointAccount: \"111122223333\"`. The `*` principal is narrowed by `s3:DataAccessPointAccount` \u2014 a recognised access-point delegation key that restricts the grant to requests routed through an access point owned by account `111122223333` (the same account as the bucket owner, per the ARN).\n\n**Composition:** The AP policy's `Principal: \"*\"` is scoped by `s3:DataAccessPointAccount`, a recognised narrowing condition. Callers not routing through an access point in that account are denied. Additionally, the AP's own BPA is all-on, which would neutralise a public AP policy independently. The grant is access-point-delegated, not public.\n\n**Verdict: COND-SCOPED (low, scoped) at the access point.** Not live.\n\n---\n\n### acme-media-archive\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": true,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- All four switches on.\n- **Policy**: none.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** No grants of any kind. BPA all-on.\n\n**Verdict: CLEAN.** Not live.\n\n---\n\n### acme-media-staging\n\n**BPA (verbatim):**\n```json\n{\n  \"BlockPublicAcls\": true,\n  \"IgnorePublicAcls\": true,\n  \"BlockPublicPolicy\": false,\n  \"RestrictPublicBuckets\": true\n}\n```\n\n- `RestrictPublicBuckets`: **true** \u2014 a public policy is denied at the resource level.\n- `BlockPublicPolicy`: false \u2014 BPA would not block the policy from being set, but `RestrictPublicBuckets` neutralises any existing public policy.\n- **Policy**: none.\n- **ACL**: owner-only.\n- **Access points**: none.\n\n**Composition:** No policy exists. ACL is owner-only. Even if a public policy were added, `RestrictPublicBuckets` would neutralise it.\n\n**Verdict: CLEAN.** Not live.\n\n---\n\n## Step 2 \u2014 Estate roll-up\n\n| Bucket | Code | Severity | Live? |\n|---|---|---|---|\n| acme-media-thumbnails | ACL-PUBLIC-IGNORED | info | No |\n| acme-media-origin | POLICY-PUBLIC-BLOCKED | info | No |\n| acme-media-shared-config | COND-SCOPED | low | No |\n| acme-media-uploads | CLEAN | \u2014 | No |\n| acme-media-transcode-tmp | CLEAN | \u2014 | No |\n| acme-media-cdn-logs | CLEAN | \u2014 | No |\n| acme-media-office-exports | COND-SCOPED | low | No |\n| acme-media-delivery-ap | COND-SCOPED (AP) | low | No |\n| acme-media-archive | CLEAN | \u2014 | No |\n| acme-media-staging | CLEAN | \u2014 | No |\n\n**No bucket carries a live finding.** Every exposed-looking grant is either neutralised by a BPA switch or scoped by a Condition. This is a deceptive-clean estate.\n\n---\n\n## Final Report\n\n### 1. Estate\n\n10 buckets reviewed. Question: is any bucket in this estate effectively publicly or cross-account exposed?\n\n---\n\n### 2. Findings \u2014 NO LIVE EXPOSURE\n\nThere is no live public or cross-account exposure anywhere in this estate. The two buckets that read most alarmingly are both inert:\n\n**acme-media-origin \u2014 POLICY-PUBLIC-BLOCKED (info, neutralised)**\nThe bucket policy contains `\"Principal\": \"*\"` with `s3:GetObject` and no Condition \u2014 a statement that would be a critical public exposure if BPA were not restricting. It is not live because `RestrictPublicBuckets: true`. S3 denies any request that would be permitted by that public statement, regardless of the fact that `BlockPublicPolicy` is false. The statement is latent: if `RestrictPublicBuckets` were toggled off, this bucket would become publicly readable immediately, with no other change required.\n\n**acme-media-thumbnails \u2014 ACL-PUBLIC-IGNORED (info, neutralised)**\nThe ACL grants `AllUsers` READ \u2014 a public-group grant that would be live if `IgnorePublicAcls` were off. It is not live because `IgnorePublicAcls: true`. S3 ignores the grant entirely. Same latent-risk note: toggling `IgnorePublicAcls` off would expose the bucket via ACL.\n\n**acme-media-shared-config \u2014 COND-SCOPED (low, scoped)**\n`Principal: \"*\"` narrowed by `aws:PrincipalOrgID: \"o-7x3k9m2qab\"`. Callers outside that AWS Organization are denied. This is intentional org-internal sharing, not public exposure. BPA is also all-on as a second layer.\n\n**acme-media-office-exports \u2014 COND-SCOPED (low, scoped)**\n`Principal: \"*\"` narrowed by `aws:SourceIp` to two CIDRs (`203.0.113.0/24`, `198.51.100.0/24`). Only requests from those ranges are permitted. This is an IP-allowlisted office read, not public exposure. Note that `BlockPublicPolicy` and `RestrictPublicBuckets` are both false here \u2014 the Condition is the *only* guardrail. If the Condition were removed or the CIDRs widened to `0.0.0.0/0`, this bucket would become publicly readable with no BPA to catch it.\n\n**acme-media-delivery-ap \u2014 COND-SCOPED (low, scoped) at access point**\nThe access point policy has `Principal: \"*\"` narrowed by `s3:DataAccessPointAccount: \"111122223333\"`, restricting access to requests r

# Judge prompt (reference)

This file documents the judge contract. The runner (`run_eval.py`) builds the live judge
prompt programmatically from this template + the deterministic ground truth from
`scenarios.expected_estate()`. Kept here so the rubric anchoring is reviewable without
reading Python.

## System

> You are an expert AWS / cloud-security evaluator grading an S3 estate review against a
> 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial
> credit. You will be given a known-good answer from a deterministic reference audit, the
> agent's review output, and the 7 rubric items. Return JSON only (no prose):
>
> ```json
> {"scores": [int, int, int, int, int, int, int], "notes": ["one sentence", ...]}
> ```

## User (per trial)

Provides:

- **FIXTURE** id and **CONTEXT** (the generic estate description the agent saw).
- **KNOWN-GOOD ANSWER** from the reused deterministic engine (verbatim per-bucket
  `s3-access-auditor` resolution, aggregated across the estate):
  - whether the estate is clean (no LIVE exposure anywhere);
  - the LIVE finding codes (the only thing that counts as exposure) + top severity;
  - the LIVE needle bucket(s), and how many of the estate's buckets are live;
  - the NON-LIVE bait codes also present (`POLICY-PUBLIC-BLOCKED`, `ACL-PUBLIC-IGNORED`,
    `COND-SCOPED`) -- buckets that READ as exposed but are neutralised by BPA or scoped by a
    Condition, which must NOT be flagged as live;
  - the expected headline (what the live defect actually is, or that it is clean);
  - the correct top fix; and
  - the boundary the audit cannot cross.
- **AGENT REVIEW OUTPUT** (verbatim).
- The **7 RUBRIC ITEMS** (see `rubric.md`), with the two load-bearing items called out:
  item 2 (surfaces the buried live needle as a primary finding, on the needle estates) and
  item 3 (does not over-flag the neutralised / scoped baits, on the deceptive-clean estates).

## Anchoring rules for the judge

- The agent prompt was GENERIC ("review this for problems"); it did NOT name public exposure,
  cross-account access, BPA neutralisation, or the buried needle. Reward the agent only for
  what it surfaces on its own. Do not penalize phrasing; grade substance.
- LIVE vs NEUTRALISED is the whole point. A `Principal '*'` ACL/policy that BPA neutralises
  (`IgnorePublicAcls` / `RestrictPublicBuckets` / `BlockPublicPolicy`), and a `Principal '*'`
  narrowed by a `Condition` (org / IP / external-id / access-point delegation), are NOT live
  exposure. Asserting they are live fails items 3 and 5. Noting them as latent / defence-in-depth
  is fine.
- On the NEEDLE estates the live bucket reads just like its neutralised/scoped siblings. The pass
  is naming exactly that one bucket and why it is live (cross-account survives BPA; a missing
  Condition makes a `Principal '*'` policy public; a cross-account canonical user survives
  `IgnorePublicAcls`). Missing it, or burying it under the baits, fails items 2 and 4.
- On the CLEAN estates the correct answer is "no live exposure." Manufacturing a live finding, or
  headlining a neutralised bucket as a public leak, fails items 2/3/5.
- Match the top fix and the boundary by substance, not wording.

Return JSON only.

# Eval: s3-estate-calibration-auditor (control-vs-treatment lift)

Measures the lift the `s3-estate-calibration-auditor` `SKILL.md` gives over a cold agent on the
effective-exposure calibration across an S3 estate. Control is the cold agent on a generic
prompt; treatment prepends `SKILL.md` as the methodology. Both arms are graded against the same
deterministic engine ground truth, so the score delta is attributable to the skill.

## The experiment

The base model aces the obvious S3 needles (a loud public bucket, a clear cross-account grant:
~7.0). It is WEAK in a narrow region we located empirically: it **over-flags** BPA-neutralised /
scoped-clean estates (~2.67-3.67) and **misses** one subtly-buried public / cross-account needle
hidden among neutralised lookalikes (~3.67). This harness scopes every fixture to exactly that
region. In **control**, a cold agent gets the raw config for EVERY bucket in an 8-12 bucket
ESTATE and a GENERIC "review this for problems" prompt that does **not** name public exposure,
cross-account access, BPA neutralisation, or the buried needle. In **treatment**, the same
estate and ask are prefixed with `SKILL.md`. The question is the lift on getting the
effective-exposure CALIBRATION right -- not flagging the buckets that LOOK exposed but are
neutralised/scoped, and still catching the one quiet live needle.

Seven fixtures, all hard-region:

| Fixture | Verdict | What it tests |
|---|---|---|
| `01-media-platform-clean` | CLEAN (10 buckets) | ignored ACL + BPA-restricted policy + org/IP/AP-delegation scoping, no live |
| `02-data-lake-clean` | CLEAN (11) | neutralised policies + ignored ACL + org-path/external-id scoping |
| `03-saas-tenancy-clean` | CLEAN (9) | org/external-id scoped sharing + ignored ACL |
| `04-backup-estate-clean` | CLEAN (10) | BPA-neutralised policies + ignored ACL + SourceIp scoping |
| `05-logging-estate-needle` | NEEDLE: `XACCT-POLICY` | one cross-account policy (BPA-all-on bucket) buried among clean/neutralised |
| `06-analytics-estate-needle` | NEEDLE: `POLICY-PUBLIC` | one unconditional public policy hidden among conditional lookalikes |
| `07-partner-share-needle` | NEEDLE: `XACCT-ACL` | one cross-account canonical-user ACL among ignored public-ACL lookalikes |

4 deceptive-clean, 3 single-needle. No loud/obvious public bucket (the model already aces those).

## Run

```bash
export ANTHROPIC_API_KEY=...
pip install anthropic
python tests/eval/run_eval.py --trials 3                              # treatment arm (~42 LLM calls)
python tests/eval/run_eval.py --conditions control,treatment --trials 3   # both arms, full lift
python tests/eval/run_eval.py --conditions control --trials 1 --fixtures 01,06  # smoke test
python tests/eval/run_eval.py --trials 3 --fresh                      # ignore prior results
```

`--conditions` defaults to `treatment` (control cells from the original screening run are
reused from `eval_results.json`); pass `control,treatment` to run both arms in one pass.

Defaults: `--trials 3`, agent + judge `claude-sonnet-4-6`, results in `eval_results.json`.
Each trial is persisted atomically; re-run the same command to resume after an interrupt.

## Ground truth offline (no key)

```bash
python tests/eval/scenarios.py   # prints clean/needle verdict, live codes, needle bucket, baits
```

The judge is anchored to `scenarios.expected_estate()`, which runs the reused deterministic
engine (verbatim per-bucket `_resolve.py`, aggregated across the estate by `_estate.py`). The two
load-bearing rubric items are **item 2** (surfaces the buried live needle as a primary finding, on
the needle estates) and **item 3** (does not over-flag the neutralised / scoped baits, on the
deceptive-clean estates). See `rubric.md` and `judge_prompt.md`.

## Reading the result

The summary prints, per fixture, the control mean, the treatment mean, and the lift, plus the
two load-bearing item rates (item 2 surfaces the buried live needle, item 3 does not over-flag
the neutralised / scoped baits) for each arm. The skill is working when treatment lifts the
weak-region fixtures toward 6-7/7 and the item-2 / item-3 pass rates climb under treatment. The
weakest treatment fixture is the next one to close with a `SKILL.md` edit.

# Rubric: s3-estate-calibration-auditor (control-only screening)

Seven binary items. Each is 1 (pass) or 0 (fail); no partial credit. The judge grades the
cold agent's audit of an S3 ESTATE (8-12 buckets) against the deterministic ground truth from
the reused per-bucket engine, aggregated across the estate (`_estate.py` -> verbatim
`_resolve.py`).

The agent was given a GENERIC prompt: the raw `public-access-block.json` / `bucket-policy.json`
/ `bucket-acl.json` / `access-points.json` for EVERY bucket in the estate, and "review this for
security/risk problems, rank by severity, give the top fix, and say what you cannot determine."
The prompt did NOT name public exposure, cross-account access, Block Public Access neutralisation,
the deceptive-clean baits, or the buried needle.

Every fixture is from the HARD region we located empirically. The cold agent aces obvious public
needles, so there are none here. Instead: 4 DECEPTIVE-CLEAN estates (several buckets LOOK exposed
but are genuinely neutralised by `IgnorePublicAcls` / `RestrictPublicBuckets` / a narrowing
`Condition`, so the engine reports NO live exposure) and 3 estates with exactly ONE quiet LIVE
needle buried among many neutralised lookalikes. The screening question is whether the cold agent
gets the effective-exposure CALIBRATION right: neither over-flagging the neutralised baits nor
missing the buried live needle.

1. **Parse.** Reads the four config layers per bucket (the BPA booleans in
   `public-access-block.json`, the Principal/Condition in `bucket-policy.json`, the Grantee in
   `bucket-acl.json`, `access-points.json`), and processes EVERY bucket in the estate, not just a
   couple. Recognises `IgnorePublicAcls` / `BlockPublicPolicy` / `RestrictPublicBuckets` as the BPA
   switches and a narrowing `Condition` (`aws:PrincipalOrgID`, `sts:ExternalId`, `aws:SourceIp`,
   `s3:DataAccessPoint*`) as scoping a `Principal '*'`.

2. **Surfaces the buried live needle (LOAD-BEARING).** On a NEEDLE estate, names the one genuinely
   live bucket (a cross-account policy, an unconditional public policy, or a cross-account
   canonical-user ACL) as A (the) PRIMARY finding, with the reason it is live, rather than burying
   it among the neutralised lookalikes or missing it. On a CLEAN estate, correctly reports NO live
   exposure. This is the item the cold agent most often misses on the needle estates: the live
   bucket reads just like its neutralised/scoped siblings.

3. **Does not over-flag the neutralised / scoped baits (LOAD-BEARING).** Does NOT report a bucket
   that is neutralised by BPA (`IgnorePublicAcls` / `RestrictPublicBuckets` / `BlockPublicPolicy`)
   or scoped by a narrowing `Condition` as a LIVE public/exposed bucket. A public-looking ACL with
   `IgnorePublicAcls` on, a `Principal '*'` policy with `RestrictPublicBuckets` on, and a
   `Principal '*'` narrowed by org / IP / external-id are NOT live exposure. Calling them live --
   or, on a clean estate, manufacturing any live finding -- fails this item. Noting them as latent
   / defence-in-depth is fine; asserting live public exposure is not. This is the item the cold
   agent most often fails on the deceptive-clean estates (the empirically-measured 2.67-3.67 region).

4. **Effective-access composition.** Resolves each bucket's EFFECTIVE verdict by combining BPA x
   policy x ACL x access points, not by reading one layer in isolation. Understands BPA neutralises
   PUBLIC grants but NOT cross-account grants (so BPA-all-on does not clear a cross-account
   policy/ACL), and that `IgnorePublicAcls` neutralises public GROUPS but not a cross-account
   canonical user. Does not clear a bucket on "BPA is all on" alone, nor condemn it on
   "`Principal '*'` is present" alone.

5. **Criticality.** Ranks the live needle as the headline (critical for a public policy, high for
   cross-account), and does NOT headline a neutralised/scoped bucket or (on a clean estate) invent a
   critical. Does not drown the real finding or the clean verdict in a wall of nitpicks about the
   correctly-neutralised buckets.

6. **Boundary.** Names at least one thing it cannot determine from the bucket configs alone,
   matching the ground-truth join: per-object ACLs, CloudFront/CDN fronting, the trusted account's
   identity policies, the account-level BPA dependency, or data sensitivity.

7. **Recommendation.** The top fix matches the ground truth in substance: fix / scope the one live
   bucket; or on a clean estate, no live fix beyond optional defence-in-depth and confirming the
   boundary. Does not prescribe ripping out the intentional scoped-sharing or the BPA-neutralised
   buckets as if they were live leaks.

## Verdict (computed from CONTROL means only)

- Aggregate control mean **< 4.0/7**, or a **majority** of fixtures below 4.0 -> **BUILD**
  (cold agent is weak here; the skill is worth writing).
- Aggregate **< 5.5/7** -> **MAYBE** (mixed; inspect per-fixture, especially items 2 and 3).
- Otherwise -> **SKIP** (cold agent already strong; the skill adds little).

"""
Control-vs-treatment lift eval for the s3-estate-calibration-auditor skill.

This harness measures the LIFT the SKILL.md gives over a cold agent on the EFFECTIVE-exposure
calibration across an S3 ESTATE (8-12 buckets). The thing under test is the HARD region we
located empirically: the cold agent ACES the obvious cross-account/public needles (7.0) but
OVER-FLAGS BPA-neutralised / scoped-clean estates (2.67-3.67) and MISSES one subtly-buried
public/cross-account needle among many neutralised lookalikes (3.67). Every fixture in this
harness is scoped to that region: 4 deceptive-clean estates (several buckets LOOK exposed but
are genuinely neutralised by IgnorePublicAcls / RestrictPublicBuckets / a narrowing Condition,
so the engine reports NO live exposure) and 3 estates with exactly ONE quiet live needle buried
among neutralised lookalikes. There is no loud, obvious public bucket -- the model already aces those.

Two conditions:
- Control: the agent gets the raw config (public-access-block.json / bucket-policy.json /
  bucket-acl.json / access-points.json) for EVERY bucket in the estate and a GENERIC "review
  this for problems" prompt that does NOT name public exposure, cross-account access, BPA
  neutralisation, the deceptive-clean baits, or the buried needle. It uses only what it brings
  from training.
- Treatment: the SAME estate config and the SAME ask, with the skill's SKILL.md prepended as
  the methodology to apply. The only variable between arms is the skill, so the lift is
  attributable to it.

Each output is graded against the 7-item rubric (rubric.md) by an LLM judge, anchored to the
deterministic reference verdict (_estate.py via scenarios.py) as ground truth. The two
load-bearing items are item 2 (surfaces the buried live needle as a primary finding) and item 3
(does not over-flag the neutralised / scoped baits).

Requirements:
- ANTHROPIC_API_KEY environment variable.
- `pip install anthropic` (the only non-stdlib dependency in the repo; isolated to tests/eval/).

Usage:
    python tests/eval/run_eval.py --trials 3                              # treatment (default)
    python tests/eval/run_eval.py --conditions control,treatment --trials 3
    python tests/eval/run_eval.py --conditions control --trials 1 --fixtures 01,06  # smoke test
    python tests/eval/run_eval.py --trials 3 --fresh                      # ignore prior results

Resume: each completed trial is persisted immediately (atomic temp+rename), and a re-run
reloads what is already on disk and fills only the missing (fixture, condition, trial) cells.
A crash, an interrupt, or an API overload mid-run therefore never throws away completed work --
just re-run the same command to finish. Pass --fresh to start clean.

Cost note: 7 fixtures x 1 condition x 3 trials, plus a judge call per output, is ~21 agent
calls + ~21 judge calls = ~42 LLM calls. Run both conditions to double that. Expect a few
dollars on Sonnet.
"""

from __future__ import annotations

import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path

try:
    from anthropic import Anthropic
except ImportError:
    print("ERROR: anthropic SDK not installed. Run: pip install anthropic", file=sys.stderr)
    sys.exit(1)

sys.path.insert(0, str(Path(__file__).parent))
from scenarios import SCENARIOS, load_fixture_text, expected_estate  # noqa: E402

DEFAULT_AGENT_MODEL = os.environ.get("EVAL_AGENT_MODEL", "claude-sonnet-4-6")
DEFAULT_JUDGE_MODEL = os.environ.get("EVAL_JUDGE_MODEL", "claude-sonnet-4-6")
MAX_TOKENS = 4096

# SKILL.md lives at the skill root (three levels up from tests/eval/).
SKILL_MD = Path(__file__).resolve().parents[2] / "SKILL.md"


def build_control_prompt(scenario: dict) -> str:
    """A GENERIC cloud-security-engineer prompt: the raw full-estate bucket config JSON and
    a plain "review this for problems" ask. No skill.

    This is the whole point of the control arm: the prompt deliberately does NOT name public
    exposure, cross-account access, Block Public Access neutralisation, the deceptive-clean
    baits, or the buried needle. It does not even tell the agent what kind of issue to look
    for. It asks the agent to review the configuration, rank issues by severity, give the
    single most important fix, and state what it cannot determine. Whether the cold agent
    gets the EFFECTIVE-exposure calibration right -- not over-flagging the buckets that LOOK
    exposed but are neutralised by BPA / scoped by a Condition, and still catching the one
    quiet live needle buried among them -- is exactly what we are measuring.
    """
    return f"""You are a cloud security engineer doing a review of an AWS account's S3 configuration.

Context: {scenario['context']}

Here is the configuration, exactly as returned by the S3 API:

{load_fixture_text(scenario)}

Review this for security and risk problems. Rank what you find by severity, give the single
most important fix, and state clearly what you cannot determine from this configuration alone.
Be specific and concrete about anything you flag."""


def build_treatment_prompt(scenario: dict, skill_md_text: str) -> str:
    """Treatment arm: the SAME review ask and the SAME estate config JSON as control, plus
    SKILL.md prepended as the methodology to apply. The only variable between arms is the
    skill, so the lift is attributable to it."""
    return f"""You are a cloud security engineer doing a review of an AWS account's S3 configuration. Apply the methodology below to the estate that follows it.

==== METHODOLOGY (SKILL.md) ====
{skill_md_text}
==== END METHODOLOGY ====

Context: {scenario['context']}

Here is the configuration, exactly as returned by the S3 API:

{load_fixture_text(scenario)}

Apply the methodology above. Resolve each bucket's effective verdict, name the one live bucket
if there is one (or report no live exposure), rank what you find by severity, give the single
most important fix, and state clearly what you cannot determine from this configuration alone.
Be specific and concrete about anything you flag."""


JUDGE_SYSTEM = """You are an expert AWS / cloud-security evaluator grading an S3 estate review against a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial credit.

You will be given a known-good answer from a deterministic reference audit, the agent's review output, and the 7 rubric items. The agent was asked to review the estate ("review this for problems"); reward the agent only for what it surfaces by substance, and grade substance, not phrasing.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}"""


def build_judge_prompt(scenario: dict, agent_output: str) -> str:
    exp = expected_estate(scenario)
    needle = ", ".join(exp["needle_buckets"]) if exp["needle_buckets"] else "(none -- the estate is clean)"
    baits = sorted(set(exp["all_codes"]) - set(exp["codes"]))
    return f"""FIXTURE: {scenario['id']}
CONTEXT (what the agent saw): {scenario['context']}

KNOWN-GOOD ANSWER (from the deterministic reference engine -- verbatim per-bucket s3-access-auditor resolution, aggregated across the estate):
- Estate clean (no LIVE exposure anywhere): {exp['clean']}
- LIVE finding codes (the only thing that counts as exposure): {exp['codes']}  (top severity: {exp['top_severity']})
- LIVE needle bucket(s): {needle}  ({exp['live_bucket_count']} of {exp['bucket_count']} buckets live)
- NON-LIVE bait codes also present (buckets that READ as exposed but are neutralised by BPA / scoped by a Condition -- these must NOT be flagged as live): {baits}
- What the defect actually is (or that it is clean): {scenario['expected_headline']}
- Correct top fix: {scenario['expected_top_fix']}
- Boundary the audit cannot cross: {scenario['expected_boundary_join']}

AGENT REVIEW OUTPUT:
{agent_output}

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: reads the four config layers per bucket (public-access-block.json BPA booleans, bucket-policy.json Principal/Condition, bucket-acl.json Grantee, access-points.json), and processes EVERY bucket in the estate rather than a couple. Recognises IgnorePublicAcls / BlockPublicPolicy / RestrictPublicBuckets as the BPA switches and a narrowing Condition (aws:PrincipalOrgID, sts:ExternalId, aws:SourceIp, s3:DataAccessPoint*) as scoping a Principal '*'.
2. Surfaces the buried live needle (LOAD-BEARING): on a NEEDLE estate, names the one genuinely-live bucket (cross-account policy / unconditional public policy / cross-account canonical-user ACL) as A (the) PRIMARY finding, with the reason it is live, rather than burying it among the neutralised lookalikes or missing it. On a CLEAN estate, correctly reports NO live exposure.
3. Does not over-flag the neutralised / scoped baits (LOAD-BEARING): does NOT report a bucket that is neutralised by BPA (IgnorePublicAcls / RestrictPublicBuckets / BlockPublicPolicy) or scoped by a narrowing Condition as a LIVE public/exposed bucket. A public-looking ACL with IgnorePublicAcls on, a Principal '*' policy with RestrictPublicBuckets on, and a Principal '*' narrowed by org/IP/external-id are NOT live exposure. Calling them live -- or, on a clean estate, manufacturing any live finding -- fails this item. (Noting them as latent / defence-in-depth is fine; asserting live public exposure is not.)
4. Effective-access composition: resolves each bucket's EFFECTIVE verdict by combining BPA x policy x ACL x access points, not by reading one layer in isolation. Specifically: understands BPA neutralises PUBLIC grants but NOT cross-account grants (so BPA-all-on does not clear a cross-account policy/ACL), and that IgnorePublicAcls neutralises public GROUPS but not a cross-account canonical user. Does not clear a bucket on "BPA is all on" alone, nor condemn it on "Principal '*' is present" alone.
5. Criticality: ranks the live needle as the headline (critical for a public policy, high for cross-account), and does NOT headline a neutralised/scoped bucket or (on a clean estate) invent a critical. Does not drown the real finding or the clean verdict in a wall of nitpicks about the correctly-neutralised buckets.
6. Boundary: names at least one thing it cannot determine from the bucket configs alone, matching the ground-truth join (per-object ACLs, CloudFront/CDN fronting, the trusted account's identity policies, the account-level BPA dependency, data sensitivity).
7. Recommendation: top fix matches the ground truth in substance (fix/scope the one live bucket; or on a clean estate, no live fix beyond optional defence-in-depth and confirming the boundary). Does not prescribe ripping out the intentional scoped-sharing or the BPA-neutralised buckets as if they were live leaks.

Return JSON only."""


RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 529}
MAX_RETRIES = 6


def _with_retries(fn, *args, **kwargs):
    """Call fn with exponential backoff on transient API errors (429/5xx/529/overloaded).

    The Anthropic SDK already retries a couple of times; this widens the window so a
    multi-minute overload spell drops far fewer trials. Re-raises on non-retryable errors
    or once retries are exhausted.

    Returns (result, call_seconds) where call_seconds is the wall-time of the SUCCESSFUL
    attempt only -- backoff sleeps and failed attempts are excluded.
    """
    delay = 2.0
    last_exc = None
    for attempt in range(MAX_RETRIES):
        try:
            t_call = time.time()
            return fn(*args, **kwargs), time.time() - t_call
        except Exception as e:  # noqa: BLE001 - inspect, then decide retryable
            status = getattr(e, "status_code", None)
            msg = str(e).lower()
            retryable = status in RETRYABLE_STATUS or "overloaded" in msg or "rate" in msg or "timeout" in msg
            if not retryable:
                raise
            last_exc = e
            if attempt < MAX_RETRIES - 1:
                time.sleep(delay)
                delay = min(delay * 2, 60.0)
    raise last_exc


def run_agent(client: Anthropic, model: str, prompt: str) -> tuple[str, float]:
    """Returns (agent_output_text, review_seconds). Seconds excludes retry backoff."""
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=MAX_TOKENS,
            # Deterministic decode: an audit must read each config value faithfully, not
            # sample it. At the default temperature the agent occasionally hallucinates a
            # BPA boolean (e.g. IgnorePublicAcls) to match a suggestively-named bucket; a
            # greedy decode removes that sampling variance.
            temperature=0,
            messages=[{"role": "user", "content": prompt}],
        )
    resp, call_s = _with_retries(_call)
    return "".join(block.text for block in resp.content if block.type == "text"), call_s


def run_judge(client: Anthropic, model: str, scenario: dict, agent_output: str) -> dict:
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=1024,
            system=JUDGE_SYSTEM,
            messages=[{"role": "user", "content": build_judge_prompt(scenario, agent_output)}],
        )
    resp, _ = _with_retries(_call)
    raw = "".join(block.text for block in resp.content if block.type == "text").strip()
    if raw.startswith("```"):
        raw = raw.split("```", 2)[1]
        if raw.startswith("json"):
            raw = raw[4:]
        raw = raw.rsplit("```", 1)[0]
    return json.loads(raw.strip())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--trials", type=int, default=3, help="Trials per fixture per condition")
    parser.add_argument("--fixtures", default="", help="Comma-separated fixture IDs (prefix match); empty = all")
    parser.add_argument("--agent-model", default=DEFAULT_AGENT_MODEL)
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--output", default="eval_results.json", help="Where to write the raw results")
    parser.add_argument("--fresh", action="store_true", help="Ignore an existing results file and start clean (default: resume/fill gaps)")
    # Default treatment-only: control cells are already on disk from the original screening run
    # and reused. Pass --conditions control,treatment to (re)run both arms in one pass.
    parser.add_argument("--conditions", default="treatment",
                        help="Comma-separated arms to run: control, treatment, or both (default: treatment)")
    args = parser.parse_args()

    conditions = [c.strip() for c in args.conditions.split(",") if c.strip()]
    bad = [c for c in conditions if c not in ("control", "treatment")]
    if bad:
        print(f"ERROR: unknown condition(s) {bad}; valid: control, treatment", file=sys.stderr)
        return 2

    if "ANTHROPIC_API_KEY" not in os.environ:
        print("ERROR: ANTHROPIC_API_KEY not set", file=sys.stderr)
        return 1

    skill_md_text = ""
    if "treatment" in conditions:
        if not SKILL_MD.exists():
            print(f"ERROR: treatment arm needs a SKILL.md at {SKILL_MD}", file=sys.stderr)
            return 1
        skill_md_text = SKILL_MD.read_text()

    client = Anthropic()

    to_run = SCENARIOS
    if args.fixtures:
        filters = [f.strip() for f in args.fixtures.split(",")]
        to_run = [s for s in SCENARIOS if any(s["id"].startswith(f) for f in filters)]

    n_cells = len(to_run) * len(conditions) * args.trials
    print(f"LIFT eval [{', '.join(conditions)}]: {len(to_run)} fixtures x {len(conditions)} conditions x {args.trials} trials = {n_cells} agent calls")
    print(f"Agent model: {args.agent_model}, Judge model: {args.judge_model}\n")

    # Resume: reload any completed trials from a prior run so a re-run fills ONLY the gaps
    # (e.g. trials dropped to a transient overload), never redoing finished work. Pass
    # --fresh to ignore an existing results file and start clean.
    results: list[dict] = []
    completed: set[tuple[str, str, int]] = set()
    out_path = Path(args.output)
    if out_path.exists() and not args.fresh:
        try:
            results = json.loads(out_path.read_text())
            completed = {(r["fixture"], r["condition"], r["trial"]) for r in results}
            print(f"Resuming from {args.output}: {len(completed)} trials already complete; filling gaps only.\n")
        except (json.JSONDecodeError, KeyError, OSError):
            results, completed = [], set()

    for scenario in to_run:
        for condition in conditions:
            for trial in range(args.trials):
                if (scenario["id"], condition, trial) in completed:
                    continue  # already have this cell from a prior run
                t_start = time.time()
                prompt = (build_treatment_prompt(scenario, skill_md_text) if condition == "treatment"
                          else build_control_prompt(scenario))
                try:
                    agent_output, agent_s = run_agent(client, args.agent_model, prompt)  # agent_s excludes retry backoff
                    judge_result = run_judge(client, args.judge_model, scenario, agent_output)
                    score = sum(judge_result["scores"])
                except Exception as e:
                    print(f"  ERROR on {scenario['id']} {condition} trial {trial}: {e}", file=sys.stderr)
                    continue
                elapsed = time.time() - t_start  # agent + judge, for cost/wall-clock accounting
                results.append({
                    "fixture": scenario["id"],
                    "condition": condition,
                    "trial": trial,
                    "score": score,
                    "scores_by_item": judge_result["scores"],
                    "notes": judge_result.get("notes", []),
                    "agent_output": agent_output,
                    "agent_chars": len(agent_output),
                    "agent_s": round(agent_s, 1),
                    "elapsed_s": round(elapsed, 1),
                })
                # Crash-safe: persist after every trial via atomic temp+rename so an
                # overload-induced death never throws away completed work.
                tmp = Path(str(args.output) + ".tmp")
                tmp.write_text(json.dumps(results, indent=2))
                tmp.replace(args.output)
                print(f"  {scenario['id']:<30} | {condition:9s} | trial {trial} | score {score}/7 | review {agent_s:.0f}s", flush=True)

    Path(args.output).write_text(json.dumps(results, indent=2))
    print(f"\nRaw results: {args.output}\n")
    print_summary(results, to_run)
    return 0


def print_summary(results: list[dict], to_run: list[dict]) -> None:
    ctrl: dict[str, list[int]] = {}
    treat: dict[str, list[int]] = {}
    item2: dict[str, dict[str, list[int]]] = {"control": {}, "treatment": {}}
    item3: dict[str, dict[str, list[int]]] = {"control": {}, "treatment": {}}
    for r in results:
        cond = r.get("condition")
        bucket = ctrl if cond == "control" else treat
        bucket.setdefault(r["fixture"], []).append(r["score"])
        sbi = r.get("scores_by_item") or []
        if cond in ("control", "treatment"):
            if len(sbi) >= 2:
                item2[cond].setdefault(r["fixture"], []).append(sbi[1])  # surfaces-the-needle
            if len(sbi) >= 3:
                item3[cond].setdefault(r["fixture"], []).append(sbi[2])  # no-over-flag

    print(f"{'Fixture':<30} {'Control':>8} {'Treat':>8} {'Lift':>8} {'Nc':>4} {'Nt':>4}")
    print("-" * 70)
    c_means: list[float] = []
    t_means: list[float] = []
    lifts: list[float] = []
    paired_ids: list[str] = []
    for scenario in to_run:
        cs = ctrl.get(scenario["id"], [])
        ts = treat.get(scenario["id"], [])
        if not cs and not ts:
            continue
        c = statistics.mean(cs) if cs else float("nan")
        t = statistics.mean(ts) if ts else float("nan")
        c_str = f"{c:>8.2f}" if cs else f"{'n/a':>8}"
        t_str = f"{t:>8.2f}" if ts else f"{'n/a':>8}"
        if cs and ts:
            lift = t - c
            lifts.append(lift); c_means.append(c); t_means.append(t); paired_ids.append(scenario["id"])
            l_str = f"{lift:>+8.2f}"
            flag = "  <- treat still <5" if t < 5.0 else ("  <- no lift" if lift <= 0 else "")
        else:
            l_str = f"{'-':>8}"
            flag = ""
        print(f"{scenario['id']:<30} {c_str} {t_str} {l_str} {len(cs):>4} {len(ts):>4}{flag}")
    print("-" * 70)

    # Load-bearing item rates, per arm, so a lift on items 2 and 3 is visible directly.
    for cond in ("control", "treatment"):
        all_i2 = [v for vs in item2[cond].values() for v in vs]
        all_i3 = [v for vs in item3[cond].values() for v in vs]
        if all_i2:
            print(f"  [{cond}] Item 2 (surfaces the buried live needle) pass rate: {statistics.mean(all_i2):.2f}")
        if all_i3:
            print(f"  [{cond}] Item 3 (does not over-flag the neutralised/scoped baits) pass rate: {statistics.mean(all_i3):.2f}")

    if not lifts:
        print("\nNo paired control/treatment fixtures to summarize "
              f"(control: {sum(len(v) for v in ctrl.values())} cells, "
              f"treatment: {sum(len(v) for v in treat.values())} cells).")
        return

    c_agg = statistics.mean(c_means)
    t_agg = statistics.mean(t_means)
    print(f"\nAggregate: control {c_agg:.2f}/7  ->  treatment {t_agg:.2f}/7   (lift {t_agg - c_agg:+.2f})")
    print(f"  Fixtures improved: {sum(1 for l in lifts if l > 0)} / {len(lifts)};  "
          f"treatment >= 6/7: {sum(1 for t in t_means if t >= 6.0)} / {len(t_means)};  "
          f"treatment >= 5/7: {sum(1 for t in t_means if t >= 5.0)} / {len(t_means)}")
    weakest = min(zip(t_means, paired_ids))
    print(f"  Weakest treatment fixture: {weakest[1]} at {weakest[0]:.2f}/7 "
          "(the next one to close with a SKILL.md edit)")


if __name__ == "__main__":
    sys.exit(main())

"""
Per-fixture estate contexts and expected answers for the s3-estate-calibration-auditor
screening eval.

The "expected_*" fields are the deterministic answers from the reused engine (_estate.py,
which delegates to the validated per-bucket _resolve.py) run against each estate fixture.
They are the source of truth the LLM judge compares the agent's output against, so the
findings are computed here by importing the engine rather than hand-copied (which would drift).

load_fixture_text renders the FULL volume the agent sees: EVERY bucket in the estate and
EVERY config layer it has (public-access-block.json / bucket-policy.json / bucket-acl.json /
access-points.json) -- an 8-12 bucket haystack per fixture, no pre-filtering. The control
prompt (in run_eval.py) is deliberately GENERIC and does NOT name public exposure,
cross-account access, BPA neutralisation, or the calibration vector; these expected fields
exist only for the judge, never for the agent.

Stdlib only. No external dependencies. `python scenarios.py` prints ground truth, no key.
"""

from __future__ import annotations

import json
import sys
from pathlib import Path

TESTS_DIR = Path(__file__).resolve().parent.parent
FIXTURES_DIR = TESTS_DIR.parent / "fixtures"

sys.path.insert(0, str(TESTS_DIR))
from _estate import run_estate  # noqa: E402

# Each entry pairs an estate fixture with the GENERIC context the eval feeds the agent
# (an estate description, no vector hint), plus the headline / fix / boundary the
# deterministic engine grounds (the judge's anchor only). Keep aligned with replay_*.py.
SCENARIOS = [
    {
        "id": "01-media-platform-clean",
        "estate": "media-platform",
        "context": "A 10-bucket media platform estate (origin, thumbnails, uploads, transcode temp, CDN logs, office exports, an access-point-fronted delivery bucket, archive, staging, shared config). Asked to review the estate's public-exposure posture.",
        "expected_headline": "NO live exposure anywhere in the estate. Several buckets READ as exposed -- an AllUsers READ ACL, a Principal '*' GetObject policy, Principal '*' narrowed by org id / SourceIp, an access-point delegation -- but every one is neutralised by Block Public Access (IgnorePublicAcls / RestrictPublicBuckets) or scoped by a Condition. The correct verdict is clean. Do NOT report any of these neutralised/scoped buckets as a live public bucket.",
        "expected_top_fix": "None for live exposure. Optionally, as defence in depth, remove the latent public ACL grant and the public-looking policy statements so the estate does not depend on BPA staying on as its only guardrail, but nothing is live today.",
        "expected_boundary_join": "per-object ACLs (an object can be public even when the bucket is private), whether a CloudFront distribution serves any bucket publicly, the account-level BPA settings (turning BPA off would expose the latent statements), and data sensitivity; the bucket configs alone cannot complete the effective-exposure join.",
    },
    {
        "id": "02-data-lake-clean",
        "estate": "data-lake",
        "context": "An 11-bucket data lake estate (raw, curated, glue scripts, athena results, EMR logs, an export bucket, quarantine, checkpoints, models, feature store, temp). Asked to review the estate's public-exposure posture.",
        "expected_headline": "NO live exposure anywhere. Multiple buckets carry public-looking policies and a public ACL grant, but all are neutralised by BPA (RestrictPublicBuckets / IgnorePublicAcls) or scoped by a Condition (aws:PrincipalOrgPaths, sts:ExternalId). The correct verdict is clean. Do NOT over-flag the neutralised lake buckets as live.",
        "expected_top_fix": "None for live exposure. Defence-in-depth: drop the latent public statements and the ignored public ACL so the lake is not one BPA toggle away from exposure, but nothing is live today.",
        "expected_boundary_join": "per-object ACLs, CloudFront/CDN fronting, the account-level BPA settings the bucket policies depend on, and the identity policies of any org/external-id principals the scoped grants trust; not derivable from the bucket configs alone.",
    },
    {
        "id": "03-saas-tenancy-clean",
        "estate": "saas-tenancy",
        "context": "A 9-bucket multi-tenant SaaS estate (tenant assets, uploads, billing, a partner feed, exports, config, backups, logs, search index). Asked to review the estate's public-exposure posture.",
        "expected_headline": "NO live exposure anywhere. Several tenant-shared buckets use Principal '*' narrowed by aws:PrincipalOrgID or sts:ExternalId (scoped sharing, not public), and one AllUsers ACL grant is rendered inert by IgnorePublicAcls. The correct verdict is clean. Do NOT read Principal '*' and call these buckets public.",
        "expected_top_fix": "None for live exposure. Confirm the org id and external id on the scoped buckets are the intended values; defence-in-depth, remove the ignored public ACL. Nothing is live today.",
        "expected_boundary_join": "the identity policies of the org accounts / external-id partner that the scoped grants trust (org membership scopes WHO, not WHAT they can do), per-object ACLs, and the account BPA settings; not in the bucket configs.",
    },
    {
        "id": "04-backup-estate-clean",
        "estate": "backup-estate",
        "context": "A 10-bucket backup/DR estate (daily, weekly, snapshots, db dumps, a restore staging bucket, an office share, config, vault, glacier archive, test restore). Asked to review the estate's public-exposure posture.",
        "expected_headline": "NO live exposure anywhere. A few buckets carry public-looking policies and a public ACL grant, all neutralised by BPA (RestrictPublicBuckets / IgnorePublicAcls); one Principal '*' is narrowed by an aws:SourceIp office allowlist (IP-scoped, not public). The correct verdict is clean. Do NOT over-flag the neutralised backup buckets.",
        "expected_top_fix": "None for live exposure. Defence-in-depth: remove the latent public statements and the ignored public ACL; confirm the SourceIp CIDRs are current office ranges. Nothing is live today.",
        "expected_boundary_join": "per-object ACLs, the account BPA settings the neutralisation depends on, whether the SourceIp ranges are still the intended office network, and data sensitivity; not in the bucket configs alone.",
    },
    {
        "id": "05-logging-estate-needle",
        "estate": "logging-estate",
        "context": "An 11-bucket logging estate (app, access, cloudtrail, vpc flow, elb, a log-shipping bucket, waf, cloudfront, archive, metrics, audit). Asked to review the estate's public-exposure posture.",
        "expected_headline": "Exactly one LIVE finding, buried among clean and neutralised buckets: acme-log-shipping has all four Block Public Access switches on (which reads as locked down) but its bucket policy grants a named other account (905638217741) read/list. A cross-account grant is NOT public, so BPA does not block it: it is fully live. The other buckets (a BPA-neutralised public policy, an ignored public ACL, an org-scoped policy) are NOT live and must not be flagged.",
        "expected_top_fix": "On acme-log-shipping, confirm account 905638217741 is a deliberate, current trust and the actions are minimal; scope to specific prefixes and prefer an aws:PrincipalOrgID / sts:ExternalId condition over a bare account root. BPA-all-on does not make this bucket safe.",
        "expected_boundary_join": "the identity policies of the trusted account 905638217741 (what it can actually do with the grant and whether it re-shares onward), which live in that account; plus per-object ACLs. Not in this bucket config.",
    },
    {
        "id": "06-analytics-estate-needle",
        "estate": "analytics-estate",
        "context": "A 10-bucket analytics estate (an org-shared events bucket, dashboards, a partner extract, reports, clickstream, ML features, staging, warehouse, ingest, temp). Asked to review the estate's public-exposure posture.",
        "expected_headline": "Exactly one LIVE finding, hidden among conditional lookalikes: acme-analytics-clickstream grants Principal '*' s3:GetObject with NO Condition and BPA is not restricting (RestrictPublicBuckets and BlockPublicPolicy both off), so it is live public. It sits next to sibling buckets that look identical but carry an aws:PrincipalOrgID or sts:ExternalId condition (scoped, fine) or have RestrictPublicBuckets on (neutralised, fine). The needle is the one missing its Condition block; the scoped/neutralised siblings must not be flagged.",
        "expected_top_fix": "On acme-analytics-clickstream, remove the public statement or scope the Principal to named accounts; if public read is genuinely intended, front it with CloudFront + Origin Access Control and turn RestrictPublicBuckets on. The conditional sibling buckets are intentional scoped sharing and should be left alone.",
        "expected_boundary_join": "per-object ACLs and the data sensitivity of the clickstream objects (the config cannot tell you what is exposed), and the account BPA setting the verdict depends on; not in the bucket config.",
    },
    {
        "id": "07-partner-share-needle",
        "estate": "partner-share",
        "context": "A 9-bucket partner-sharing estate (a public mirror, a press kit, a partner drop, downloads, uploads, config, an org feed, archive, staging). Asked to review the estate's public-exposure posture.",
        "expected_headline": "Exactly one LIVE finding, disguised as a routine ACL: acme-share-partner-drop grants READ to a different account's CANONICAL USER via its bucket ACL. IgnorePublicAcls (on for this estate) only neutralises the public GROUPS (AllUsers / AuthenticatedUsers); a cross-account canonical-user grant is untouched and stays live even with BPA tightened, and the bucket's TLS-only Deny does not address it. The lookalike buckets carry AllUsers public ACL grants that ARE ignored by IgnorePublicAcls (not live) -- the needle reads like them but is a named other identity, not a public group.",
        "expected_top_fix": "On acme-share-partner-drop, remove the cross-account ACL grant unless it is a deliberate, current sharing relationship; express any intended sharing as a scoped bucket policy with a named principal and disable ACLs with Bucket Owner Enforced. The TLS Deny does not cover the partner's read access.",
        "expected_boundary_join": "the identity policies of the trusted account behind the canonical user (what it can do, whether it re-shares), and the per-object ACLs; the cross-account ACL grant's blast radius is not in this bucket config.",
    },
]


def fixture_dir(scenario: dict) -> Path:
    return FIXTURES_DIR / scenario["id"]


def load_fixture_text(scenario: dict) -> str:
    """The raw config JSON the agent is given for this estate.

    Renders the FULL volume: EVERY bucket sub-directory in the estate and EVERY config
    layer it has (BPA, policy, ACL, access points), so the agent genuinely sees the
    8-12 bucket haystack, not a pre-filtered slice. NOTE: the agent prompt itself
    (run_eval.py) is generic and does not name public exposure / cross-account / BPA.
    """
    d = fixture_dir(scenario)
    layers = [
        ("public-access-block.json", "BLOCK PUBLIC ACCESS (public-access-block.json)"),
        ("bucket-policy.json", "BUCKET POLICY (bucket-policy.json)"),
        ("bucket-acl.json", "BUCKET ACL (bucket-acl.json)"),
        ("access-points.json", "ACCESS POINTS (access-points.json)"),
    ]
    bucket_dirs = sorted(
        child for child in d.iterdir()
        if child.is_dir() and any((child / f).exists() for f, _ in layers)
    )
    parts: list[str] = [f"S3 estate '{scenario['estate']}' -- {len(bucket_dirs)} buckets, "
                        "raw config exactly as returned by the S3 API:"]
    for bd in bucket_dirs:
        meta_path = bd / "meta.json"
        bucket_name = bd.name
        if meta_path.exists():
            bucket_name = json.loads(meta_path.read_text()).get("bucket", bd.name)
        parts.append("\n" + "=" * 78)
        parts.append(f"BUCKET: {bucket_name}")
        parts.append("=" * 78)
        for filename, label in layers:
            path = bd / filename
            if path.exists():
                parts.append(f"{label}:\n" + json.dumps(json.loads(path.read_text()), indent=2))
    return "\n".join(parts)


def expected_estate(scenario: dict) -> dict:
    """Run the reused deterministic engine to get the ground-truth verdict for the judge.

    The engine runs the verbatim per-bucket resolution on every bucket in the estate, then
    aggregates: the estate is clean iff no bucket carries a live finding, and the needle is
    whichever bucket(s) carry a live finding among the neutralised/scoped lookalikes.
    """
    e = run_estate(fixture_dir(scenario))
    return {
        "codes": sorted(e.codes()),                 # LIVE codes only
        "all_codes": sorted(e.all_codes()),         # live + neutralised/scoped baits
        "top_severity": e.top_severity,
        "clean": e.clean,
        "needle_buckets": e.needle_buckets,
        "bucket_count": e.bucket_count,
        "live_bucket_count": len(e.live_buckets),
        "boundary_count": len(e.boundary),
    }


# Alias kept for parity with the sibling harnesses' naming.
expected_resolution = expected_estate


if __name__ == "__main__":
    # `python tests/eval/scenarios.py` prints the ground-truth answers, no API key needed.
    for s in SCENARIOS:
        exp = expected_estate(s)
        needle = ", ".join(exp["needle_buckets"]) if exp["needle_buckets"] else "(none -- clean)"
        print(f"{s['id']:<30} clean={exp['clean']!s:<5} top={str(exp['top_severity']):<9} "
              f"live={str(exp['codes']):<20} buckets={exp['bucket_count']}")
        print(f"{'':<30} live_needle={needle}")
        print(f"{'':<30} baits(non-live)={sorted(set(exp['all_codes']) - set(exp['codes']))}")