{
  "principal": "role/build-fleet-runner",
  "note": "Looks like the classic PassRole + RunInstances escalation: both actions are present. But an explicit Deny on iam:PassRole (Resource '*') overrides the scoped Allow, and the role it would have passed is a retired instance profile. The PassRole half is dead; there is no real escalation path."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RunBuildFleet",
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:DescribeInstances",
        "ec2:DescribeImages"
      ],
      "Resource": "*"
    },
    {
      "Sid": "PassFleetInstanceProfile",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::488213749302:role/legacy-fleet-instance-profile"
    },
    {
      "Sid": "DenyAllPassRole",
      "Effect": "Deny",
      "Action": "iam:PassRole",
      "Resource": "*"
    }
  ]
}

{
  "principal": "role/sandbox-experimenter",
  "note": "Action '*' reads like AdministratorAccess, but it is pinned to a single sandbox S3 bucket (never Resource '*'), and a Deny on every escalation-bearing service (iam, sts, kms, lambda, ec2:RunInstances, ssm, secretsmanager) on Resource '*' overrides the star for anything dangerous. The wildcard expands to nothing useful outside one scratch bucket."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FullControlOfSandboxBucketOnly",
      "Effect": "Allow",
      "Action": "*",
      "Resource": [
        "arn:aws:s3:::team-sandbox-scratch",
        "arn:aws:s3:::team-sandbox-scratch/*"
      ]
    },
    {
      "Sid": "DenyEscalationServicesEverywhere",
      "Effect": "Deny",
      "Action": [
        "iam:*",
        "sts:*",
        "organizations:*",
        "kms:*",
        "lambda:*",
        "ec2:RunInstances",
        "ecs:RunTask",
        "ssm:*",
        "secretsmanager:*"
      ],
      "Resource": "*"
    }
  ]
}

{
  "principal": "role/deploy-orchestrator",
  "note": "The permissions policy grants sts:AssumeRole on role/org-admin-break-glass, which sounds like a lateral move into an admin role. But the target role's trust policy only trusts security-break-glass-operator and incident-commander, behind an ExternalId: it does not trust this principal back. The AssumeRole grant is inert, there is no actual path, and this principal has no iam:UpdateAssumeRolePolicy to fix the trust. No escalation."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeDeployTargetRole",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::488213749302:role/org-admin-break-glass"
    },
    {
      "Sid": "ReadDeployConfig",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::deploy-config-eu",
        "arn:aws:s3:::deploy-config-eu/*"
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OnlyBreakGlassAdminsCanAssume",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::488213749302:role/security-break-glass-operator",
          "arn:aws:iam::488213749302:role/incident-commander"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "breakglass-2026"
        }
      }
    }
  ]
}

{
  "principal": "role/identity-platform-operator",
  "note": "policy-1 reads like a full identity-takeover kit: PutRolePolicy, AttachRolePolicy, CreatePolicyVersion, SetDefaultPolicyVersion, UpdateAssumeRolePolicy, PassRole and CreateAccessKey, every E2/E4/E5/E6 primitive in one place. But all of them are scoped to a single break-glass role/policy ARN (never Resource '*', so no W4), and policy-2 carries a permission-boundary-style explicit Deny on every one of those mutation/credential/assume actions across Resource '*'. An explicit Deny wins over any Allow, so the effective permission set collapses to read-only IAM inventory. The escalation kit is fully capped: no E-combo can fire."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageBreakGlassRoleInlinePolicies",
      "Effect": "Allow",
      "Action": [
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy",
        "iam:CreatePolicyVersion",
        "iam:SetDefaultPolicyVersion",
        "iam:UpdateAssumeRolePolicy",
        "iam:PassRole",
        "iam:CreateAccessKey"
      ],
      "Resource": [
        "arn:aws:iam::488213749302:role/break-glass-admin",
        "arn:aws:iam::488213749302:policy/break-glass-admin-policy"
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BoundaryDenyAllIdentityMutationEverywhere",
      "Effect": "Deny",
      "Action": [
        "iam:PutRolePolicy",
        "iam:PutUserPolicy",
        "iam:PutGroupPolicy",
        "iam:AttachRolePolicy",
        "iam:AttachUserPolicy",
        "iam:AttachGroupPolicy",
        "iam:CreatePolicyVersion",
        "iam:SetDefaultPolicyVersion",
        "iam:UpdateAssumeRolePolicy",
        "iam:PassRole",
        "iam:CreateAccessKey",
        "iam:CreateLoginProfile",
        "iam:UpdateLoginProfile",
        "iam:AddUserToGroup",
        "sts:AssumeRole"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ReadOnlyIamInventory",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:GetPolicy",
        "iam:GetPolicyVersion",
        "iam:ListRolePolicies",
        "iam:ListAttachedRolePolicies"
      ],
      "Resource": "*"
    }
  ]
}

{
  "principal": "role/cost-reporting-collector",
  "note": "sts:AssumeRole on a role in a DIFFERENT account (905512347781) reads like a cross-account pivot into a foreign account. But the AssumeRole Allow is gated by an sts:ExternalId + aws:PrincipalOrgID Condition this principal cannot satisfy (it is not in org o-9f3kxample and holds no ExternalId), and the target role's own trust policy (supplied) only trusts callers that present BOTH the org id and the ExternalId, so the wildcard Principal is fully narrowed and X1 does not fire. The principal has no iam:UpdateAssumeRolePolicy to relax either side. The pivot looks open but is condition-sealed at both ends: no real path, no escalation."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AssumeVendorAuditRoleCrossAccount",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::905512347781:role/external-cost-auditor",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "vendor-cost-auditor-prod"
        },
        "StringEquals_PrincipalOrgID": {
          "aws:PrincipalOrgID": "o-9f3kxample"
        }
      }
    },
    {
      "Sid": "ReadCostAndUsageData",
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "ce:GetCostForecast",
        "cur:DescribeReportDefinitions"
      ],
      "Resource": "*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadBillingArtifacts",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::cost-reports-export",
        "arn:aws:s3:::cost-reports-export/*"
      ]
    },
    {
      "Sid": "ReadBudgets",
      "Effect": "Allow",
      "Action": [
        "budgets:ViewBudget",
        "budgets:DescribeBudgetActionsForBudget"
      ],
      "Resource": "*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AnyPrincipalButOnlyFromOurOrgWithExternalId",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-9f3kxample",
          "sts:ExternalId": "vendor-cost-auditor-prod"
        }
      }
    }
  ]
}

{
  "principal": "role/sandbox-compute-operator",
  "note": "policy-1 grants iam:PassRole and policy-2 grants a fistful of compute verbs (ec2:StartInstances, ecs:StartTask, lambda:InvokeFunction), which together read like the textbook PassRole + launch-compute escalation. Two reasons it is inert. First, none of the compute verbs are creation/launch primitives that bind a passed role: StartInstances/StartTask/InvokeFunction operate on EXISTING compute and do not accept an iam:PassRole argument, so there is no RunInstances/CreateFunction/RunTask to pair PassRole with (E1 needs a role-binding launch action). Second, PassRole is scoped to one role, sandbox-readonly-compute, whose OWN policy is attached here (policy-3) and is strictly read-only: GetObject/Query/GetLogEvents on a handful of sandbox resources. Passing a role no more privileged than the caller, to compute that cannot bind it anyway, yields zero privilege gain. An orphaned escalation: the shape is there, the gain is not."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PassSandboxComputeRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::488213749302:role/sandbox-readonly-compute"
    },
    {
      "Sid": "ValidatePassableRole",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:ListRolePolicies",
        "iam:ListAttachedRolePolicies"
      ],
      "Resource": "arn:aws:iam::488213749302:role/sandbox-readonly-compute"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OperateExistingSandboxCompute",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:RebootInstances",
        "ec2:DescribeInstances",
        "ecs:StartTask",
        "ecs:StopTask",
        "lambda:InvokeFunction"
      ],
      "Resource": [
        "arn:aws:ec2:eu-west-1:488213749302:instance/*",
        "arn:aws:ecs:eu-west-1:488213749302:task/*",
        "arn:aws:lambda:eu-west-1:488213749302:function:sandbox-*"
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SandboxReadonlyComputeRoleOwnPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "logs:GetLogEvents",
        "logs:DescribeLogStreams",
        "cloudwatch:GetMetricData",
        "dynamodb:GetItem",
        "dynamodb:Query"
      ],
      "Resource": [
        "arn:aws:s3:::sandbox-fixtures-eu",
        "arn:aws:s3:::sandbox-fixtures-eu/*",
        "arn:aws:logs:eu-west-1:488213749302:log-group:/sandbox/*",
        "arn:aws:dynamodb:eu-west-1:488213749302:table/sandbox-readonly-cache"
      ]
    }
  ]
}

{
  "principal": "role/ml-training-platform",
  "note": "Six attached policies for an ML training platform, ~16 statements, every one plausible for a SageMaker pipeline: read feature stores, describe training jobs, pull ECR images, manage the training queue, track experiments, write job outputs. The escalation is split three ways and only composes from the union. policy-4 grants iam:PassRole on Resource '*' (framed as passing the training execution role, with iam:GetRole/ListRoles right next to it as innocent validation). policy-6 grants sagemaker:CreateTrainingJob, a real role-binding launch action. Composed, that is the E1 primitive: launch a training job with ANY role in the account attached (PassRole is unscoped), then use that job's credentials. Neither half is alarming alone, PassRole reads like routine execution-role plumbing and CreateTrainingJob reads like the platform's core job, and they sit four policies apart behind heavy benign bait. A per-statement read clears every statement; only the union is critical."
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadFeatureStoreBuckets",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::ml-feature-store-eu",
        "arn:aws:s3:::ml-feature-store-eu/*",
        "arn:aws:s3:::ml-training-artifacts-eu",
        "arn:aws:s3:::ml-training-artifacts-eu/*"
      ]
    },
    {
      "Sid": "ReadTrainingMetadata",
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query",
        "dynamodb:BatchGetItem"
      ],
      "Resource": "arn:aws:dynamodb:eu-west-1:488213749302:table/training-runs"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DescribeTrainingInfra",
      "Effect": "Allow",
      "Action": [
        "sagemaker:DescribeTrainingJob",
        "sagemaker:ListTrainingJobs",
        "sagemaker:DescribeModel",
        "sagemaker:ListModels",
        "sagemaker:DescribeEndpoint",
        "sagemaker:ListEndpoints"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ReadTrainingLogs",
      "Effect": "Allow",
      "Action": [
        "logs:GetLogEvents",
        "logs:FilterLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:eu-west-1:488213749302:log-group:/aws/sagemaker/*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadModelRegistryEcr",
      "Effect": "Allow",
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:GetAuthorizationToken",
        "ecr:DescribeImages"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ReadTrainingParameters",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParametersByPath"
      ],
      "Resource": "arn:aws:ssm:eu-west-1:488213749302:parameter/ml/training/*"
    },
    {
      "Sid": "ReadTrainingMetrics",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricData",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageTrainingQueue",
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:eu-west-1:488213749302:training-jobs"
    },
    {
      "Sid": "PassTrainingExecutionRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "*"
    },
    {
      "Sid": "ReadRolesForValidation",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:ListRoles"
      ],
      "Resource": "*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageNotebookLifecycleConfigsReadOnly",
      "Effect": "Allow",
      "Action": [
        "sagemaker:DescribeNotebookInstance",
        "sagemaker:ListNotebookInstances",
        "sagemaker:DescribeNotebookInstanceLifecycleConfig",
        "sagemaker:ListNotebookInstanceLifecycleConfigs"
      ],
      "Resource": "*"
    },
    {
      "Sid": "WriteExperimentTracking",
      "Effect": "Allow",
      "Action": [
        "sagemaker:CreateExperiment",
        "sagemaker:CreateTrial",
        "sagemaker:CreateTrialComponent",
        "sagemaker:AddTags"
      ],
      "Resource": "arn:aws:sagemaker:eu-west-1:488213749302:experiment/*"
    },
    {
      "Sid": "ReadKmsForArtifactDecryption",
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:ListAliases"
      ],
      "Resource": "arn:aws:kms:eu-west-1:488213749302:key/ml-artifacts-key"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RunTrainingAndProcessingJobs",
      "Effect": "Allow",
      "Action": [
        "sagemaker:CreateTrainingJob",
        "sagemaker:CreateProcessingJob",
        "sagemaker:StopTrainingJob"
      ],
      "Resource": "arn:aws:sagemaker:eu-west-1:488213749302:training-job/*"
    },
    {
      "Sid": "WriteJobOutputs",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::ml-training-artifacts-eu/*"
    }
  ]
}

[
  {
    "fixture": "01-orphaned-passrole-deny",
    "condition": "control",
    "trial": 0,
    "score": 2,
    "scores_by_item": [
      1,
      0,
      0,
      0,
      0,
      1,
      0
    ],
    "notes": [
      "Agent correctly reads all statements including the explicit Deny on iam:PassRole with Resource:* and the Allow on RunInstances.",
      "Ground truth is zero findings (clean fixture); agent incorrectly reports multiple high-severity findings, missing that the Deny neutralises the combo and the policy is effectively clean from an escalation perspective.",
      "Agent falsely asserts a live privilege escalation path from ec2:RunInstances + PassRole despite the Deny making PassRole dead; this is a material false positive.",
      "Agent recognises the Deny overrides the Allow (correct cross-statement reasoning) but then contradicts itself by treating the post-Deny-removal scenario as a current finding rather than understanding the current state is neutralised.",
      "Agent headlines 'High (Privilege Escalation)' for a neutralised grant, which is exactly what the rubric penalises; the correct answer is no escalation exists currently.",
      "Agent explicitly notes it cannot determine the privileges of the passed role and mentions other attached policies / permissions boundaries as out-of-scope, matching the boundary requirement.",
      "Ground truth top fix is 'None needed on the escalation' (optionally clean up inert statements); agent instead recommends removing the Deny, which is the opposite of the correct guidance and could reintroduce risk."
    ],
    "agent_output": "## IAM Policy Audit: `role/build-fleet-runner`\n\n---\n\n### Finding 1: Explicit Deny on `iam:PassRole` Completely Nullifies the Allow\n\n**Severity: High (Functional Breakage / Security Logic Error)**\n\n#### What's happening\n\nAWS IAM policy evaluation follows a strict precedence order:\n\n1. Explicit Deny\n2. Explicit Allow\n3. Implicit Deny (default)\n\nThe `DenyAllPassRole` statement uses `\"Resource\": \"*\"`, which matches `arn:aws:iam::488213749302:role/legacy-fleet-instance-profile`. Because an explicit Deny **always wins** over an explicit Allow \u2014 regardless of statement order \u2014 the `PassFleetInstanceProfile` Allow is completely overridden.\n\n**Net result:** `iam:PassRole` is denied on all resources, including the intended `legacy-fleet-instance-profile`. The fleet cannot launch instances with an instance profile attached. This policy is broken as written.\n\n#### Fix\n\nRemove the `DenyAllPassRole` statement entirely. The Allow on `iam:PassRole` is already scoped to a single ARN, which is the correct security control. A blanket Deny on `*` adds no additional protection \u2014 the implicit deny already covers everything not explicitly allowed.\n\n```json\n// DELETE this statement entirely:\n{\n  \"Sid\": \"DenyAllPassRole\",\n  \"Effect\": \"Deny\",\n  \"Action\": \"iam:PassRole\",\n  \"Resource\": \"*\"\n}\n```\n\n---\n\n### Finding 2: `ec2:RunInstances` on `\"Resource\": \"*\"` Without `iam:PassRole` Scope Guard Creates a Latent Privilege Escalation Path\n\n**Severity: High (Privilege Escalation)**\n\n#### What's happening\n\nOnce Finding 1 is fixed (the Deny removed), the principal can call `iam:PassRole` scoped to `legacy-fleet-instance-profile` and launch EC2 instances via `ec2:RunInstances`. This is the intended behavior.\n\nHowever, `ec2:RunInstances` is granted on `\"Resource\": \"*\"` with **no condition restricting which IAM role can be passed**. The `iam:PassRole` resource constraint is the only guardrail. If that role is ever misconfigured (e.g., a wildcard is accidentally introduced on the PassRole resource), or if a second Allow statement is later added, this principal could pass **any role in the account** \u2014 including administrator roles \u2014 to a newly launched EC2 instance and then retrieve credentials from the instance metadata service (IMDS).\n\nMore immediately: the `ec2:RunInstances` permission with `Resource: *` allows launching instances **without any instance profile at all**, or with any profile the principal can pass. There is no `aws:RequestedRegion`, instance type, AMI, or VPC constraint.\n\n#### Concrete escalation path (post-fix)\n\n```\n1. Attacker compromises build-fleet-runner credentials\n2. Calls ec2:RunInstances targeting any AMI, any instance type, any region\n3. If iam:PassRole scope is ever widened (or a second policy is attached),\n   passes a role with AdministratorAccess\n4. SSHes/SSMs into instance \u2192 curl http://169.254.169.254/latest/meta-data/\n   iam/security-credentials/<role> \u2192 retrieves admin credentials\n```\n\n#### Fix\n\nApply least-privilege constraints on `ec2:RunInstances` using IAM condition keys:\n\n```json\n{\n  \"Sid\": \"RunBuildFleet\",\n  \"Effect\": \"Allow\",\n  \"Action\": [\n    \"ec2:RunInstances\",\n    \"ec2:DescribeInstances\",\n    \"ec2:DescribeImages\"\n  ],\n  \"Resource\": \"*\",\n  \"Condition\": {\n    \"StringEquals\": {\n      \"aws:RequestedRegion\": \"us-east-1\",\n      \"ec2:InstanceType\": [\"m5.large\", \"m5.xlarge\"]\n    },\n    \"ArnEquals\": {\n      \"ec2:AssociatePublicIpAddress\": \"false\"\n    }\n  }\n}\n```\n\nAdditionally, enforce that only the intended instance profile can be passed by adding a condition to the PassRole statement:\n\n```json\n{\n  \"Sid\": \"PassFleetInstanceProfile\",\n  \"Effect\": \"Allow\",\n  \"Action\": \"iam:PassRole\",\n  \"Resource\": \"arn:aws:iam::488213749302:role/legacy-fleet-instance-profile\",\n  \"Condition\": {\n    \"StringEquals\": {\n      \"iam:PassedToService\": \"ec2.amazonaws.com\"\n    }\n  }\n}\n```\n\nThe `iam:PassedToService` condition ensures this role can only be passed to EC2, not to Lambda, ECS, or other services the principal might be able to invoke.\n\n---\n\n### Finding 3: `legacy-fleet-instance-profile` Role Name Suggests Unreviewed Permissions\n\n**Severity: Medium (Requires Out-of-Band Verification)**\n\n#### What's happening\n\nThe role name `legacy-fleet-instance-profile` contains the word **\"legacy\"**, which is a strong signal that this role's permissions have not been recently audited. Instance profile roles are a common vector for privilege escalation because they inherit whatever permissions were granted at the time of creation, which may include overly broad policies that have since been forgotten.\n\nIf `legacy-fleet-instance-profile` has policies like `ec2:*`, `s3:*`, or any `iam:*` actions, the build fleet runner can indirectly exercise those permissions by launching an instance and querying IMDS.\n\n#### Fix\n\n1. Run `aws iam list-attached-role-policies` and `aws iam list-role-policies` on `legacy-fleet-instance-profile`.\n2. Audit all attached policies for over-broad grants.\n3. Replace \"legacy\" role with a purpose-built role scoped to only what the build fleet instance needs (e.g., S3 read for build artifacts, ECR pull).\n4. Enable IAM Access Analyzer to identify unused permissions on the role.\n\n---\n\n### Summary Table\n\n| # | Finding | Severity | Immediate Action |\n|---|---------|----------|-----------------|\n| 1 | Explicit Deny on `iam:PassRole *` negates the Allow \u2014 policy is broken | High | Remove `DenyAllPassRole` statement |\n| 2 | `ec2:RunInstances` on `*` with no condition constraints; latent escalation via instance profile | High | Add region/instance-type conditions; add `iam:PassedToService` condition |\n| 3 | `legacy-fleet-instance-profile` likely has unreviewed, over-broad permissions | Medium | Audit and replace with scoped role |",
    "agent_chars": 5706,
    "agent_s": 30.3,
    "elapsed_s": 40.2
  },
  {
    "fixture": "01-orphaned-passrole-deny",
    "condition": "control",
    "trial": 1,
    "score": 3,
    "scores_by_item": [
      1,
      0,
      0,
      1,
      0,
      1,
      0
    ],
    "notes": [
      "Agent correctly parses all statements including PassRole, RunInstances, and the explicit Deny, and understands the Deny overrides the Allow.",
      "Ground truth finding is empty (clean policy); agent reports three findings including a 'High' latent escalation, so it does not correctly report 'no real defect'.",
      "Agent asserts a material 'High' severity latent privilege escalation finding that the ground truth explicitly says does not exist, constituting a false positive.",
      "Agent correctly reasons across statements and recognises that the Deny on '*' neutralises the scoped PassRole Allow, demonstrating proper cross-statement evaluation.",
      "Agent headlines a neutralised/latent RunInstances+PassRole combo as 'High' severity, which contradicts the rubric requirement not to headline a neutralised grant as critical.",
      "Agent explicitly states it cannot determine the permissions of legacy-fleet-instance-profile from the policy alone, matching the ground truth boundary about target role privileges.",
      "Ground truth top fix is 'None needed on the escalation; it is already neutralised'; agent's top fix is to remove the Deny and re-enable PassRole with a condition, which contradicts the correct recommendation."
    ],
    "agent_output": "## IAM Policy Audit: `role/build-fleet-runner`\n\n---\n\n### Finding 1: Explicit Deny on `iam:PassRole` Renders the Allow Statement Dead Code\n\n**Severity: Medium (Operational / Security Hygiene)**\n\n#### What's happening\n\nAWS IAM evaluation logic applies a strict precedence rule: **an explicit Deny always overrides an Allow**, regardless of statement order or resource scope.\n\nThe policy contains:\n- `PassFleetInstanceProfile`: Allow `iam:PassRole` on `arn:aws:iam::488213749302:role/legacy-fleet-instance-profile`\n- `DenyAllPassRole`: Deny `iam:PassRole` on `Resource: *`\n\nThe Deny on `*` subsumes the specific ARN. When the principal attempts to call `iam:PassRole` on `legacy-fleet-instance-profile`, **both** statements match. The Deny wins. The net effective permission is:\n\n```\niam:PassRole \u2192 DENIED on all resources, including legacy-fleet-instance-profile\n```\n\nThe Allow statement is completely neutralized. This means:\n- `ec2:RunInstances` calls that require passing an instance profile **will fail at runtime**\n- The policy does not function as the author intended\n- The intended scoping of PassRole to a single role ARN provides **zero actual restriction**\n\n#### Why this matters beyond \"it's broken\"\n\nThe person who wrote this policy likely believed the Allow on the specific ARN was active, providing a scoped, least-privilege PassRole grant. **It is not.** This is a logic error that creates a false sense of security in either direction:\n- Operators may believe PassRole is safely scoped and permitted \u2192 it is actually fully blocked\n- A future reviewer may remove the Deny thinking it's unnecessary \u2192 it would suddenly expose PassRole with the broad Allow still in place\n\n---\n\n### Finding 2: `ec2:RunInstances` on `Resource: *` Without `iam:PassRole` Creates a Latent Privilege Escalation Path\n\n**Severity: High (if Deny is ever corrected without tightening PassRole)**\n\n#### What's happening\n\n`ec2:RunInstances` on `Resource: *` allows the principal to launch any EC2 instance type, in any subnet, with any AMI. When combined with `iam:PassRole` (even scoped), this is the classic **EC2 privilege escalation path**:\n\n1. Principal calls `ec2:RunInstances`, passing an instance profile attached to a high-privilege role\n2. Code running on that instance inherits the role's permissions via the instance metadata service (IMDS)\n3. Principal SSHs into, SSMs into, or plants user-data on the instance to retrieve credentials\n\nRight now this path is blocked because `iam:PassRole` is fully denied. **But the Deny is almost certainly a bug.** If a developer \"fixes\" the policy by removing `DenyAllPassRole`, the Allow on `legacy-fleet-instance-profile` becomes active \u2014 and the question becomes: what permissions does that role have?\n\n#### The role name is a red flag\n\n`legacy-fleet-instance-profile` \u2014 the word **legacy** suggests:\n- This role may have accumulated permissions over time\n- It is likely not reviewed or rotated regularly\n- It may have overly broad permissions granted in an earlier, less security-conscious era\n\nIf `legacy-fleet-instance-profile` has significant permissions (e.g., `s3:*`, `ec2:*`, broad SSM, or worse), then correcting the Deny without auditing that role immediately creates a privilege escalation path.\n\n---\n\n### Finding 3: `ec2:DescribeInstances` and `ec2:DescribeImages` on `Resource: *`\n\n**Severity: Low (Information Disclosure)**\n\nDescribe permissions cannot be resource-scoped in IAM (AWS does not support resource-level restrictions on most Describe calls). However, it is worth noting these allow the principal to enumerate:\n- All EC2 instances across the account (IDs, IPs, tags, security groups, subnet placement)\n- All AMIs visible to the account, including private ones\n\nThis is a low-severity finding but relevant in multi-tenant or sensitive environments where instance metadata (tags containing environment names, IP ranges, etc.) should not be broadly visible.\n\n---\n\n## Summary Table\n\n| # | Finding | Severity | Current State |\n|---|---------|----------|---------------|\n| 1 | Deny overrides Allow \u2014 `iam:PassRole` is fully blocked | Medium | Broken/Dead code |\n| 2 | `ec2:RunInstances *` + latent PassRole escalation path | High | Latent (blocked by bug) |\n| 3 | Describe permissions expose account-wide inventory | Low | Active |\n\n---\n\n## Recommended Fix\n\nRemove both `iam:PassRole` statements and replace them with a single, correctly scoped Allow. Add a `Condition` to bind PassRole to the EC2 service, which is best practice and prevents the role from being passed to unintended services:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"RunBuildFleet\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ec2:RunInstances\",\n        \"ec2:DescribeInstances\",\n        \"ec2:DescribeImages\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Sid\": \"PassFleetInstanceProfile\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"iam:PassRole\",\n      \"Resource\": \"arn:aws:iam::488213749302:role/legacy-fleet-instance-profile\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"iam:PassedToService\": \"ec2.amazonaws.com\"\n        }\n      }\n    }\n  ]\n}\n```\n\n**Additionally, before deploying this fix:**\n\n1. **Audit `legacy-fleet-instance-profile` immediately.** Pull its attached policies and determine the blast radius if an instance running with this profile were compromised. If it has permissions beyond what the build fleet needs, scope it down.\n2. **Consider replacing it entirely.** A role named \"legacy\" should be replaced with a purpose-built, minimally scoped instance profile for the build fleet.\n3. **Enable IMDSv2 enforcement** on launched instances (`HttpTokens: required` in `ec2:RunInstances` via a condition or a Service Control Policy) to make credential theft from IMDS harder.\n4. **Add a resource condition on `ec2:RunInstances`** if possible \u2014 restricting to specific subnets, instance types, or requiring specific tags \u2014 to reduce the surface area of what the build fleet can launch.",
    "agent_chars": 5979,
    "agent_s": 34.2,
    "elapsed_s": 41.5
  },
  {
    "fixture": "01-orphaned-passrole-deny",
    "condition": "control",
    "trial": 2,
    "score": 3,
    "scores_by_item": [
      1,
      0,
      0,
      1,
      0,
      1,
      0
    ],
    "notes": [
      "Agent correctly reads all statements including the PassRole Allow scoped to a specific ARN and the Deny on Resource '*', and recognizes ec2:RunInstances.",
      "Ground truth says findings are empty (clean policy); agent reports multiple findings including a Medium severity misconfiguration and a conditional High escalation path, so it fails to correctly identify the clean state.",
      "Agent asserts a material defect (privilege escalation path via PassRole+RunInstances as Finding 3) that the ground truth explicitly says is neutralised, constituting a false positive escalation claim.",
      "Agent correctly reasons that the explicit Deny on '*' overrides the scoped Allow, demonstrating cross-statement reasoning about the Deny neutralising the PassRole combo.",
      "Agent headlines a neutralised PassRole+RunInstances combo as a High severity escalation (Finding 3), which the ground truth says is a dead path; criticality ranking is incorrect.",
      "Agent explicitly notes it cannot determine the permissions of legacy-fleet-instance-profile without further investigation, matching the ground-truth boundary about the target role's privileges.",
      "Ground truth says the correct top fix is 'None on the escalation: it is already neutralised'; agent recommends removing the Deny and adding PassedToService condition, which contradicts the ground-truth finding that no escalation fix is needed."
    ],
    "agent_output": "## IAM Policy Audit: `role/build-fleet-runner`\n\n---\n\n### Finding 1: Explicit Deny on `iam:PassRole` Completely Nullifies the Allow \u2014 Misconfiguration, Not Overgrant\n\n**Severity: Medium (Operational / Security Misconfiguration)**\n\n#### What's Happening\n\nAWS evaluates IAM policy statements using a **strict precedence rule: an explicit Deny always wins over any Allow**, regardless of statement order.\n\nThe policy contains:\n\n| Statement | Effect | Action | Resource |\n|---|---|---|---|\n| `PassFleetInstanceProfile` | Allow | `iam:PassRole` | `arn:aws:iam::488213749302:role/legacy-fleet-instance-profile` |\n| `DenyAllPassRole` | Deny | `iam:PassRole` | `*` (wildcard \u2014 matches everything) |\n\nThe Deny on `Resource: \"*\"` matches `arn:aws:iam::488213749302:role/legacy-fleet-instance-profile` because `*` is a superset of that ARN. The Allow is therefore **completely and silently overridden**. The principal **cannot pass any role at all**, including the intended one.\n\nThis is likely the result of a developer adding the Deny as a \"safety net\" without understanding that it cancels the Allow they wrote above it.\n\n#### Why This Matters\n\n1. **Operational breakage**: `ec2:RunInstances` with an instance profile requires `iam:PassRole`. Because PassRole is effectively denied, any attempt to launch an instance with a profile attached will fail with `AccessDenied`. The build fleet will be broken.\n\n2. **False sense of security**: An operator reading this policy might believe the Allow grants PassRole for a specific role and the Deny restricts everything else. That is not how AWS policy evaluation works. The intended \"allow one, deny rest\" pattern is not achievable this way.\n\n3. **The attempted restriction doesn't work the way intended**: If the goal was \"allow *only* this one role to be passed, block all others,\" that goal is completely defeated \u2014 the result is \"block *all* roles,\" including the intended one.\n\n---\n\n### Finding 2: `ec2:RunInstances` on `Resource: \"*\"` \u2014 Overly Broad\n\n**Severity: Low\u2013Medium**\n\n`ec2:RunInstances` with `Resource: \"*\"` allows launching instances of any type, in any VPC/subnet, with any AMI, with no constraints on instance size, region, or networking placement. For a build fleet runner this is broader than necessary and creates cost and abuse risk (e.g., launching GPU instances, launching in unintended regions).\n\n---\n\n### Finding 3: Privilege Escalation Path via `iam:PassRole` + `ec2:RunInstances` (if Deny is removed)\n\n**Severity: High \u2014 if the Deny is corrected without scope-limiting the PassRole target**\n\nThis deserves explicit attention because the natural fix for Finding 1 is to remove the `DenyAllPassRole` statement. **If you do that naively**, the Allow on `iam:PassRole` scoped to `role/legacy-fleet-instance-profile` is restored \u2014 but you need to verify what permissions that role has.\n\n#### The Escalation Path\n\n```\nbuild-fleet-runner\n  \u2514\u2500\u25ba ec2:RunInstances  (launch any instance)\n  \u2514\u2500\u25ba iam:PassRole      (attach legacy-fleet-instance-profile to that instance)\n        \u2514\u2500\u25ba EC2 instance runs as legacy-fleet-instance-profile\n              \u2514\u2500\u25ba if that role has broad permissions (e.g., AdministratorAccess,\n                  s3:*, iam:*, etc.), the operator of that instance inherits them\n```\n\nThe `build-fleet-runner` principal itself may have limited permissions, but by launching an EC2 instance and attaching a powerful role, anyone who can execute code on that instance (or anyone who controls what code runs on the build fleet) **effectively operates as that role**.\n\n**This is a classic IAM privilege escalation pattern** documented by Rhino Security Labs and in AWS's own security guidance.\n\n#### What to check immediately\n\n```bash\naws iam get-role --role-name legacy-fleet-instance-profile\naws iam list-attached-role-policies --role-name legacy-fleet-instance-profile\naws iam list-role-policies --role-name legacy-fleet-instance-profile\n```\n\nIf `legacy-fleet-instance-profile` has any of the following, this is a **Critical** finding:\n- `AdministratorAccess`\n- `iam:*` or `iam:CreateRole`, `iam:AttachRolePolicy`\n- `sts:AssumeRole` without conditions\n- Broad `s3:*`, `ec2:*`, or data-plane access beyond what build jobs need\n\n---\n\n## Summary Table\n\n| # | Finding | Severity | Impact |\n|---|---|---|---|\n| 1 | Explicit Deny overrides Allow on `iam:PassRole` \u2014 PassRole is fully blocked | Medium | Build fleet broken; policy intent not achieved |\n| 2 | `ec2:RunInstances` on `Resource: \"*\"` | Low\u2013Medium | Unconstrained instance launches |\n| 3 | PassRole + RunInstances = privilege escalation via `legacy-fleet-instance-profile` (if Deny removed) | High (pending role audit) | Potential escalation to any permission the passed role holds |\n\n---\n\n## Recommended Fixes\n\n### Fix 1: Remove the conflicting Deny; use a Condition to scope PassRole correctly\n\nThe **only correct way** to express \"allow passing exactly one role, implicitly deny all others\" is to **scope the Allow tightly and omit the Deny entirely**. An explicit Deny is not needed because anything not explicitly allowed is already denied by default in AWS IAM.\n\n```json\n{\n  \"Sid\": \"PassFleetInstanceProfileOnly\",\n  \"Effect\": \"Allow\",\n  \"Action\": \"iam:PassRole\",\n  \"Resource\": \"arn:aws:iam::488213749302:role/legacy-fleet-instance-profile\",\n  \"Condition\": {\n    \"StringEquals\": {\n      \"iam:PassedToService\": \"ec2.amazonaws.com\"\n    }\n  }\n}\n```\n\nThe `iam:PassedToService` condition ensures this PassRole can only be used when passing the role to EC2, not to Lambda, ECS, or other services \u2014 limiting lateral movement.\n\n**Remove `DenyAllPassRole` entirely.**\n\n### Fix 2: Constrain `ec2:RunInstances`\n\n```json\n{\n  \"Sid\": \"RunBuildFleet\",\n  \"Effect\": \"Allow\",\n  \"Action\": [\n    \"ec2:RunInstances\",\n    \"ec2:DescribeInstances\",\n    \"ec2:DescribeImages\"\n  ],\n  \"Resource\": \"*\",\n  \"Condition\": {\n    \"StringEquals\": {\n      \"ec2:Region\": \"us-east-1\"\n    },\n    \"StringLike\": {\n      \"ec2:InstanceType\": [\"c5.*\", \"m5.*\"]\n    }\n  }\n}\n```\n\nAdjust region and instance type prefixes to match your actual build fleet requirements.\n\n### Fix 3: Audit and right-size `legacy-fleet-instance-profile`\n\nAudit the role immediately. It should hold **only** the permissions build jobs legitimately need (e.g., read from a specific S3 artifact bucket, write to a specific CloudWatch log group). If it currently holds broad permissions, scope it down before removing the Deny in Fix 1.\n\n---\n\n## Key Takeaway\n\n> **Explicit Deny + Allow on the same action/resource = the Allow is dead.** The policy as written both breaks the build fleet and creates a false impression of controlled PassRole behavior. The fix is not to \"balance\" Allows and Denys \u2014 it is to write a tight Allow and rely on default-deny for everything else.",
    "agent_chars": 6724,
    "agent_s": 36.8,
    "elapsed_s": 45.5
  },
  {
    "fixture": "02-action-star-blanket-deny",
    "condition": "control",
    "trial": 0,
    "score": 2,
    "scores_by_item": [
      1,
      0,
      0,
      0,
      0,
      1,
      0
    ],
    "notes": [
      "The agent correctly parses Action:'*' as expanding to all S3 actions including dangerous control-plane operations, demonstrating proper wildcard expansion.",
      "The ground truth says there are NO findings (clean policy); the agent raises 10 findings including CRITICAL ones, missing that the Deny caps the star and the star is resource-scoped to one sandbox bucket making it clean.",
      "The agent asserts multiple material defects (CRITICAL cross-account exfiltration, HIGH escalation paths) that the ground truth explicitly says do not exist because the Deny neutralises the star and the resource scope contains it.",
      "The agent fails cross-statement reasoning: it does not recognise that the broad Deny statement neutralises the dangerous S3 control-plane actions that Action:'*' would otherwise grant, and that Resource scoping prevents non-S3 action expansion.",
      "The agent headlines a Deny-neutralised, Resource-scoped Action:'*' as CRITICAL privilege escalation, which the ground truth explicitly says is not a real escalation \u2014 this is exactly the false positive the rubric warns against.",
      "The agent does mention boundaries it cannot determine, specifically referencing SCPs and other attached policies that could re-grant denied services, matching the ground-truth boundary acknowledgment.",
      "The ground truth says 'no fix required for security' and the agent recommends extensive remediation including replacing Action:'*' with explicit allowlist and adding multiple deny statements, which contradicts the correct answer of no fix needed."
    ],
    "agent_output": "## IAM Policy Audit: `role/sandbox-experimenter`\n\n---\n\n### Overall Design Intent vs. Reality\n\nThe policy attempts a \"allow everything on one bucket, deny dangerous services everywhere\" pattern. This pattern is **fundamentally flawed** because the Allow statement is `Action: \"*\"` scoped to S3 ARNs \u2014 but AWS evaluates service-to-resource bindings loosely in some cases, and more importantly, `Action: \"*\"` on S3 resources still means every S3 action, many of which are dangerous. The Deny list then has meaningful **gaps**.\n\n---\n\n## Finding 1 \u2014 CRITICAL: `Action: \"*\"` on S3 Enables Cross-Account Exfiltration via Bucket Policy Manipulation\n\n**Severity: CRITICAL**\n\n**What's wrong:**\n\n`Action: \"*\"` scoped to the bucket ARN includes:\n- `s3:PutBucketPolicy`\n- `s3:PutBucketAcl`\n- `s3:PutBucketCORS`\n- `s3:PutBucketWebsite`\n- `s3:PutBucketReplication` (requires IAM role ARN \u2014 but the role can point to an attacker-controlled account)\n- `s3:PutBucketNotification` (can trigger Lambda **in another account**)\n- `s3:PutBucketLogging` (redirect logs to attacker bucket)\n\n**Escalation path:**\n\n```\n1. Attacker calls s3:PutBucketPolicy on team-sandbox-scratch\n2. Sets policy granting s3:GetObject / s3:PutObject to Principal: attacker-account-root\n3. All sandbox data is now readable/writable cross-account\n4. Alternatively: s3:PutBucketReplication pointing to attacker-controlled bucket\n   \u2192 all future writes silently replicated externally\n```\n\nThis bypasses all the Deny statements because none of them cover `s3:*`.\n\n**Fix:**\n\nReplace `Action: \"*\"` with an explicit allowlist of S3 actions actually needed:\n\n```json\n\"Action\": [\n  \"s3:GetObject\",\n  \"s3:PutObject\",\n  \"s3:DeleteObject\",\n  \"s3:ListBucket\",\n  \"s3:GetBucketLocation\"\n],\n```\n\nExplicitly deny the dangerous S3 control-plane actions:\n\n```json\n{\n  \"Sid\": \"DenyS3DataPlaneEscalation\",\n  \"Effect\": \"Deny\",\n  \"Action\": [\n    \"s3:PutBucketPolicy\",\n    \"s3:DeleteBucketPolicy\",\n    \"s3:PutBucketAcl\",\n    \"s3:PutBucketCORS\",\n    \"s3:PutBucketWebsite\",\n    \"s3:PutBucketReplication\",\n    \"s3:PutBucketNotification\",\n    \"s3:PutBucketLogging\",\n    \"s3:PutEncryptionConfiguration\",\n    \"s3:PutLifecycleConfiguration\",\n    \"s3:PutBucketVersioning\",\n    \"s3:PutObjectAcl\",\n    \"s3:PutBucketOwnershipControls\",\n    \"s3:PutPublicAccessBlock\"\n  ],\n  \"Resource\": \"*\"\n}\n```\n\n---\n\n## Finding 2 \u2014 HIGH: `s3:PutBucketPublicAccessBlock` / Public ACL Grants Not Denied\n\n**Severity: HIGH**\n\n**What's wrong:**\n\nEven without full bucket policy control, `Action: \"*\"` includes `s3:PutPublicAccessBlock` (to *disable* the block) and `s3:PutObjectAcl` (to make individual objects public). Sequence:\n\n```\n1. s3:PutPublicAccessBlock \u2192 set BlockPublicAcls: false, IgnorePublicAcls: false\n2. s3:PutObjectAcl on any object \u2192 grant public-read\n3. Sensitive data in sandbox bucket is now publicly readable on the internet\n```\n\nThis is a data exfiltration path that requires no cross-account access.\n\n**Fix:** Covered by the Deny list in Finding 1. Additionally, enforce at the **account level** via S3 Account Public Access Block (this is a defense-in-depth control outside the policy itself).\n\n---\n\n## Finding 3 \u2014 HIGH: Missing `glue:*`, `athena:*`, `s3:GetObject` combinations Enable Data Lake Access\n\n**Severity: HIGH**\n\n**What's wrong:**\n\nThe Deny list only blocks compute/identity escalation services. It does **not** block:\n- `glue:*`\n- `athena:*`\n- `redshift-data:*`\n- `datasync:*`\n- `macie2:*`\n\n`Action: \"*\"` on S3 ARNs does not grant these services directly, **but** the `Action: \"*\"` wording means if AWS ever evaluates a non-S3 action against an S3 ARN and matches, it would be allowed. More concretely: this role can call `s3:GetObject` to read **any object in the bucket**, then use `athena` or `glue` (not denied) to query data across the account if Glue/Athena resource policies permit cross-resource access.\n\nMore critically: **the Deny statement does not include `glue:*` or `athena:*`**, so if this role somehow gets invoked in a Glue job context, there is no deny backstop.\n\n**Fix:** Add to the Deny statement:\n\n```json\n\"glue:*\",\n\"athena:*\",\n\"datasync:*\",\n\"redshift-data:*\",\n\"lakeformation:*\"\n```\n\n---\n\n## Finding 4 \u2014 HIGH: `sts:*` Deny is Incomplete \u2014 `aws:PrincipalArn` Confusion / `s3:PutBucketPolicy` Re-enables STS indirectly\n\n**Severity: HIGH**\n\n**What's wrong:**\n\n`sts:AssumeRole` is denied. Good. However:\n\n1. **`sts:GetFederationToken` and `sts:GetSessionToken`** are under `sts:*` so covered \u2014 but only for **this principal calling STS**. \n2. The deeper issue: if Finding 1 is not fixed, this role can write a bucket policy that allows a **different, less-restricted principal** to assume a role via a trust policy that references the bucket condition. This is an **indirect STS bypass** \u2014 the role doesn't call STS itself, it manipulates resources so another principal does.\n\n**Fix:** Fix Finding 1. The `sts:*` Deny is correctly written but only effective after bucket policy manipulation is prevented.\n\n---\n\n## Finding 5 \u2014 HIGH: `ec2:RunInstances` Denied but EC2 Privilege Escalation Remains\n\n**Severity: HIGH**\n\n**What's wrong:**\n\nOnly `ec2:RunInstances` is denied. The following EC2 actions are **not denied** and could be used for escalation:\n\n```\nec2:CreateLaunchTemplate        \u2192 encode userdata with credential-stealing script\nec2:ModifyLaunchTemplate        \u2192 backdoor existing templates\nec2:RequestSpotInstances        \u2192 alternate compute launch path (not RunInstances)\nec2:RequestSpotFleet\nec2:CreateFleet                 \u2192 another compute launch path bypassing RunInstances deny\nec2:ImportInstance              \u2192 import a VM\nec2:CreateImage / ec2:RegisterImage  \u2192 snapshot manipulation\n```\n\n**Specific bypass:**\n\n```\n1. ec2:CreateFleet with LaunchTemplateConfig\n   \u2192 Launches EC2 instances without calling ec2:RunInstances\n   \u2192 AWS IAM evaluates ec2:CreateFleet, not ec2:RunInstances\n   \u2192 Deny on ec2:RunInstances does NOT catch this\n```\n\nThis is a well-documented IAM gotcha: `ec2:RunInstances` is not the only way to launch compute.\n\n**Fix:**\n\n```json\n\"ec2:*\"  // deny all EC2, not just RunInstances\n```\n\nOr at minimum:\n\n```json\n\"ec2:RunInstances\",\n\"ec2:CreateFleet\",\n\"ec2:RequestSpotInstances\",\n\"ec2:RequestSpotFleet\",\n\"ec2:CreateLaunchTemplate\",\n\"ec2:ModifyLaunchTemplate\",\n\"ec2:CreateLaunchTemplateVersion\",\n\"ec2:ImportInstance\"\n```\n\n---\n\n## Finding 6 \u2014 HIGH: `ecs:RunTask` Denied but ECS Escalation Remains\n\n**Severity: HIGH**\n\n**What's wrong:**\n\nSame pattern as Finding 5. `ecs:RunTask` is denied, but:\n\n```\necs:CreateService          \u2192 launches tasks as a service (not RunTask)\necs:UpdateService          \u2192 modify existing service to run attacker task definition\necs:RegisterTaskDefinition \u2192 create malicious task definition\necs:StartTask              \u2192 another launch path (places task on specific container instance)\n```\n\n**Specific bypass:**\n\n```\n1. ecs:RegisterTaskDefinition \u2192 define task with privileged container + IMDS access\n2. ecs:CreateService \u2192 launch it (bypasses ecs:RunTask deny)\n3. Container calls IMDS \u2192 gets role credentials \u2192 privilege escalation\n```\n\n**Fix:**\n\n```json\n\"ecs:*\"  // deny all ECS\n```\n\n---\n\n## Finding 7 \u2014 MEDIUM: `kms:*` Denied but S3 Bucket Key Rotation/Deletion in Scope\n\n**Severity: MEDIUM**\n\n**What's wrong:**\n\n`kms:*` is denied, which is correct. However, `s3:PutEncryptionConfiguration` is **not** denied (covered under `Action: \"*\"` Allow). This allows the principal to:\n\n```\n1. s3:PutEncryptionConfiguration \u2192 change bucket encryption from SSE-KMS to SSE-S3 or disable\n2. This downgrades encryption without touching KMS directly\n3. Future objects stored with weaker encryption\n```\n\nThis is a data integrity/compliance issue rather than a direct escalation, but depending on the compliance posture it may be critical.\n\n**Fix:** Add `s3:PutEncryptionConfiguration` to the S3 Deny list from Finding 1 (already included there).\n\n---\n\n## Finding 8 \u2014 MEDIUM: CloudTrail Manipulation Not Denied\n\n**Severity: MEDIUM**\n\n**What's wrong:**\n\nThe Deny list does not include:\n- `cloudtrail:StopLogging`\n- `cloudtrail:DeleteTrail`\n- `cloudtrail:UpdateTrail`\n- `cloudtrail:PutEventSelectors` (exclude S3 data events from logging)\n\nIf this role is compromised, an attacker can blind the audit trail before exfiltrating data:\n\n```\n1. cloudtrail:PutEventSelectors \u2192 exclude s3.amazonaws.com data events\n2. All subsequent s3:GetObject calls are unlogged\n3. Exfiltrate bucket contents silently\n```\n\n**Fix:** Add to Deny statement:\n\n```json\n\"cloudtrail:StopLogging\",\n\"cloudtrail:DeleteTrail\",\n\"cloudtrail:UpdateTrail\",\n\"cloudtrail:PutEventSelectors\",\n\"cloudtrail:PutInsightSelectors\"\n```\n\n---\n\n## Finding 9 \u2014 MEDIUM: `s3:GetObject` on `Resource: *` Scope Issue\n\n**Severity: MEDIUM**\n\n**What's wrong:**\n\nThe Allow statement correctly scopes to the specific bucket ARNs. However, certain S3 actions operate at the **account level** and are not ARN-scoped, notably:\n\n- `s3:ListAllMyBuckets` \u2192 returns names of **all buckets in the account**\n- `s3:GetAccountPublicAccessBlock`\n- `s3:PutAccountPublicAccessBlock` \u2190 disables account-level public access block for **entire account**\n\n`s3:PutAccountPublicAccessBlock` requires `Resource: arn:aws:s3:::*` which **does match** the wildcard object ARN `arn:aws:s3:::team-sandbox-scratch/*` pattern evaluation. In practice AWS requires the resource be `*` for account-level S3 actions, but `s3:ListAllMyBuckets` **does** work and leaks bucket enumeration data.\n\n**Fix:** Explicitly deny:\n\n```json\n\"s3:ListAllMyBuckets\",\n\"s3:PutAccountPublicAccessBlock\",\n\"s3:GetAccountPublicAccessBlock\"\n```\n\n---\n\n## Finding 10 \u2014 LOW: No Explicit Deny on `s3:GetObject` for Other Buckets\n\n**Severity: LOW / Defense-in-Depth**\n\n**What's wrong:**\n\nThe policy relies on the implicit deny for other buckets. This is correct IAM behavior \u2014 no Allow exists for other bucket ARNs. However, there is no **explicit Deny**. If an SCP or another policy attached to this role (e.g., a permissive managed policy added later) inadvertently grants S3 read elsewhere, there is no backstop.\n\n**Fix:** Add an explicit Deny for S3 actions on resources outside the sandbox bucket:\n\n```json\n{\n  \"Sid\": \"DenyS3OutsideSandbox\",\n  \"Effect\": \"Deny\",\n  \"Action\": \"s3:*\",\n  \"NotResource\": [\n    \"arn:aws:s3:::team-sandbox-scratch\",\n    \"arn:aws:s3:::team-sandbox-scratch/*\"\n  ]\n}\n```\n\n---\n\n## Corrected Policy\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"AllowSandboxBucketDataPlaneOnly\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:GetObject\",\n        \"s3:PutObject\",\n        \"s3:DeleteObject\",\n        \"s3:ListBucket\",\n        \"s3:GetBucketLocation\",\n        \"s3:GetObjectVersion\",\n        \"s3:DeleteObjectVersion\",\n        \"s3:GetObjectTagging\",\n        \"s3:PutObjectTagging\"\n      ],\n      \"Resource\": [\n        \"arn:aws:s3:::team-sandbox-scratch\",\n        \"arn:aws:s3:::team-sandbox-scratch/*\"\n      ]\n    },\n    {\n      \"Sid\": \"DenyAllS3OutsideSandbox\",\n      \"Effect\": \"Deny\",\n      \"Action\": \"s3:*\",\n      \"NotResource\": [\n        \"arn:aws:s3:::team-sandbox-scratch\",\n        \"arn:aws:s3:::team-sandbox-scratch/*\"\n      ]\n    },\n    {\n      \"Sid\": \"DenyEscalationServicesEverywhere\",\n      \"Effect\": \"Deny\",\n      \"Action\": [\n        \"iam:*\",\n        \"sts:*\",\n        \"organizations:*\",\n        \"kms:*\",\n        \"lambda:*\",\n        \"ec2:*\",\n        \"ecs:*\",\n        \"ssm:*\",\n        \"secretsmanager:*\",\n        \"glue:*\",\n        \"athena:*\",\n        \"lakeformation:*\",\n        \"datasync:*\",\n        \"cloudtrail:StopLogging\",\n        \"cloudtrail:DeleteTrail\",\n        \"cloudtrail:UpdateTrail\",\n        \"cloudtrail:PutEventSelectors\",\n        \"cloudtrail:PutInsightSelectors\",\n        \"s3:PutBucketPolicy\",\n        \"s3:DeleteBucketPolicy\",\n        \"s3:PutBucketAcl\",\n        \"s3:PutBucketCORS\",\n        \"s3:PutBucketWebsite\",\n        \"s3:PutBucketReplication\",\n        \"s3:PutBucketNotification\",\n        \"s3:PutBucketLogging\",\n        \"s3:PutEncryptionConfiguration\",\n        \"s3:PutLifecycleConfiguration\",\n        \"s3:PutBucketVersioning\",\n        \"s3:PutObjectAcl\",\n        \"s3:PutBucketOwnershipControls\",\n        \"s3:PutPublicAccessBlock\",\n        \"s3:PutAccountPublicAccessBlock\",\n        \"s3:ListAllMyBuckets\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}\n```\n\n---\n\n## Summary Table\n\n| # | Finding | Severity | Root Cause |\n|---|---------|----------|------------|\n| 1 | `Action:*` allows bucket policy takeover / cross-account exfil | **CRITICAL** |",
    "agent_chars": 12386,
    "agent_s": 68.1,
    "elapsed_s": 77.7
  },
  {
    "fixture": "02-action-star-blanket-deny",
    "condition": "control",
    "trial": 1,
    "score": 2,
    "scores_by_item": [
      1,
      0,
      0,
      0,
      0,
      1,
      0
    ],
    "notes": [
      "The agent correctly parses Action '*' and recognizes it expands to concrete sensitive S3 permissions like s3:PutBucketPolicy.",
      "The ground truth finding is 'no real defect' (clean: True, findings: []), but the agent reports multiple CRITICAL and HIGH findings \u2014 failing to identify that the policy is actually clean.",
      "The agent asserts multiple material defects (CRITICAL s3:PutBucketPolicy escalation, HIGH glue/athena gaps, HIGH cross-bucket S3 access) that the ground truth explicitly says do not exist after considering the Deny scope.",
      "The agent fails cross-statement reasoning: it does not correctly conclude that the broad Deny statement neutralizes the Action '*' allow, which is the key analytical step the ground truth identifies.",
      "The agent headlines a 'CRITICAL' escalation via s3:PutBucketPolicy, but the ground truth explicitly states the star expands to nothing useful outside one scratch bucket and the Deny caps it \u2014 this neutralized grant should not be called critical.",
      "The agent correctly names things it cannot determine from the policy alone, including org SCPs, trust policy details, and other attached policies.",
      "The ground truth top fix is 'None required for security' (optionally replace Action * with concrete S3 actions for clarity), but the agent recommends a major policy rewrite treating the policy as actively dangerous \u2014 the substance does not match."
    ],
    "agent_output": "## IAM Policy Audit: `role/sandbox-experimenter`\n\n---\n\n### Overall Design Assessment\n\nThe policy follows a reasonable \"allow-narrow, deny-broad\" pattern, but has **several significant gaps** that undermine the intent. The core problem is that `Action: \"*\"` on an S3 resource is more powerful than it appears, and the deny list has meaningful omissions.\n\n---\n\n## Finding 1 \u2014 `Action: \"*\"` on S3 Resources is Not Just S3\n\n**Severity: CRITICAL**\n\n**The Problem:**\n\nThis is the most counterintuitive issue. When IAM evaluates `Action: \"*\"` against a resource, it does not scope to \"S3 actions only\" just because the resource is an S3 ARN. However, the real danger is subtler: **`Action: \"*\"` means this role is granted every action in AWS**, subject to resource-level filtering. For actions that support `Resource: *` only (i.e., they don't accept specific ARNs), AWS silently ignores those actions when the resource doesn't match \u2014 but for services where the ARN *happens to match*, the grant applies.\n\nMore concretely: **you are granting every S3 action including ones that enable escalation**:\n\n```\ns3:PutBucketPolicy\ns3:PutBucketAcl\ns3:PutBucketWebsite\ns3:PutBucketNotification\ns3:PutBucketCORS\n```\n\n**Escalation Path via `s3:PutBucketPolicy`:**\n\n```\n1. role/sandbox-experimenter calls s3:PutBucketPolicy on team-sandbox-scratch\n2. Injects a bucket policy granting s3:GetObject / s3:PutObject \n   to an external AWS account (attacker-controlled)\n3. Exfiltrates data or uses the bucket as a relay \u2014 no IAM, STS, \n   or KMS required, so the Deny statement never fires\n```\n\n**Escalation Path via `s3:PutBucketNotification`:**\n\n```\n1. Attacker configures bucket event notifications pointing to a \n   Lambda function ARN in ANOTHER account (not blocked by the \n   Lambda deny, which only blocks lambda:* actions, not S3 \n   triggering Lambda)\n2. Any object upload triggers cross-account Lambda execution\n3. That Lambda runs under its own role \u2014 potentially privileged\n```\n\n**Escalation Path via `s3:PutBucketWebsite` + `s3:PutBucketPolicy`:**\n```\n1. Enable static website hosting\n2. Set bucket policy to allow public access\n3. Host malicious content / exfil endpoint publicly\n```\n\n**Fix:**\n\nReplace `Action: \"*\"` with an explicit, minimal S3 action list:\n\n```json\n{\n  \"Sid\": \"FullControlOfSandboxBucketOnly\",\n  \"Effect\": \"Allow\",\n  \"Action\": [\n    \"s3:GetObject\",\n    \"s3:PutObject\",\n    \"s3:DeleteObject\",\n    \"s3:ListBucket\",\n    \"s3:GetBucketLocation\",\n    \"s3:AbortMultipartUpload\",\n    \"s3:ListMultipartUploadParts\"\n  ],\n  \"Resource\": [\n    \"arn:aws:s3:::team-sandbox-scratch\",\n    \"arn:aws:s3:::team-sandbox-scratch/*\"\n  ]\n}\n```\n\nThen add an **explicit Deny** for the dangerous bucket-level actions:\n\n```json\n{\n  \"Sid\": \"DenyDangerousS3BucketManagement\",\n  \"Effect\": \"Deny\",\n  \"Action\": [\n    \"s3:PutBucketPolicy\",\n    \"s3:DeleteBucketPolicy\",\n    \"s3:PutBucketAcl\",\n    \"s3:PutBucketNotification\",\n    \"s3:PutBucketWebsite\",\n    \"s3:PutBucketCORS\",\n    \"s3:PutBucketLogging\",\n    \"s3:PutEncryptionConfiguration\",\n    \"s3:PutLifecycleConfiguration\",\n    \"s3:PutReplicationConfiguration\",\n    \"s3:PutBucketVersioning\",\n    \"s3:PutBucketPublicAccessBlock\"\n  ],\n  \"Resource\": \"*\"\n}\n```\n\n---\n\n## Finding 2 \u2014 `glue:*`, `athena:*`, `redshift-data:*` Not in Deny List\n\n**Severity: HIGH**\n\n**The Problem:**\n\nThe deny list blocks compute escalation vectors but **completely omits analytics services** that can read arbitrary S3 data across the account:\n\n```\nglue:CreateCrawler       \u2192 crawl any S3 bucket the Glue role can access\nathena:StartQueryExecution \u2192 query S3 data via Glue catalog\nglue:CreateJob           \u2192 run arbitrary code (PySpark/Python) under the Glue IAM role\nglue:CreateConnection    \u2192 connect to VPC resources\n```\n\n**Escalation Path via `glue:CreateJob`:**\n\n```\n1. role/sandbox-experimenter calls glue:CreateJob\n2. Specifies an existing high-privilege Glue service role \n   (AWSGlueServiceRole or similar) \u2014 attacker doesn't create \n   the role, just references it\n3. Job runs arbitrary Python/Spark code under that role\n4. Effectively assumes a privileged role without touching iam:* or sts:*\n```\n\nThis bypasses the `sts:*` deny entirely because **Glue does the AssumeRole internally** \u2014 the principal never calls `sts:AssumeRole` directly.\n\n**Fix:**\n\nAdd to the Deny statement:\n\n```json\n\"glue:*\",\n\"athena:*\",\n\"redshift-data:*\",\n\"redshift:*\",\n\"lakeformation:*\",\n\"databrew:*\"\n```\n\n---\n\n## Finding 3 \u2014 `cloudformation:*` Not in Deny List\n\n**Severity: HIGH**\n\n**The Problem:**\n\nCloudFormation is not blocked. This is a well-known escalation primitive:\n\n```\n1. Create a CloudFormation stack\n2. Stack uses a CloudFormation service role (passed via --role-arn) \n   OR uses the default CFN execution role\n3. Template provisions IAM roles, Lambda functions, EC2 instances\n4. CFN calls iam:CreateRole, lambda:CreateFunction internally \u2014 \n   the *principal* never calls those directly\n```\n\nEven without specifying a role, if the account has a default CFN role with broad permissions, the attacker inherits it.\n\nNote: `cloudformation:CreateStack` requires resources other than S3 to be useful, and the allow statement only covers S3 ARNs \u2014 **but CloudFormation's resource ARN is `arn:aws:cloudformation:*`**, so it does not match the S3 allow. However, the policy only *denies* what's listed \u2014 anything not in the Allow is an implicit deny already. Let me re-examine this.\n\n**Correction and Clarification:**\n\nActually, because the Allow is scoped to S3 ARNs only, `cloudformation:CreateStack` would be an implicit deny (the allow doesn't match `arn:aws:cloudformation:*` resources). **This finding is lower risk than initially assessed**, but still worth noting because:\n\n- Some CFN actions accept `Resource: *` and could theoretically match in edge cases\n- This is a gap in the *deny* list that could matter if the allow were ever broadened\n- Best practice is explicit deny on escalation services regardless\n\n**Revised Severity: MEDIUM** (implicit deny provides protection, but defense-in-depth is absent)\n\n**Fix:**\n\nAdd to Deny: `\"cloudformation:*\"`\n\n---\n\n## Finding 4 \u2014 `s3:GetObject` on Non-Sandbox Buckets is Not Denied\n\n**Severity: HIGH**\n\n**The Problem:**\n\nThe Allow only grants access to `team-sandbox-scratch`. But the **Deny does not block S3 actions on other buckets**. This means:\n\n- There is no explicit allow for other S3 buckets \u2192 **implicit deny applies**\n- This is currently safe\n\n**However**, this creates a fragile posture. If any **Service Control Policy (SCP)** or **Resource-based policy** at the account level grants broader S3 access, this role would silently inherit it. More importantly:\n\n```\nBucket policies can grant cross-principal access.\nIf another bucket in this account has a permissive bucket policy \n(e.g., Allow Principal: * for the same account), this role could \nread/write it \u2014 and the permissions policy would not block it \nbecause there's no explicit Deny.\n```\n\n**Fix:**\n\nAdd an explicit S3 deny for non-sandbox resources:\n\n```json\n{\n  \"Sid\": \"DenyAllS3OutsideSandbox\",\n  \"Effect\": \"Deny\",\n  \"Action\": \"s3:*\",\n  \"NotResource\": [\n    \"arn:aws:s3:::team-sandbox-scratch\",\n    \"arn:aws:s3:::team-sandbox-scratch/*\"\n  ]\n}\n```\n\n---\n\n## Finding 5 \u2014 `sts:AssumeRole` Blocked, But `sts:AssumeRoleWithWebIdentity` / `sts:AssumeRoleWithSAML` Also Blocked \u2014 Verify OIDC Posture\n\n**Severity: LOW / Informational**\n\n**The Problem:**\n\n`sts:*` is denied, which is correct. However, auditors should verify:\n\n- Does this role have a **trust policy** that allows other principals to assume *it*?\n- If `role/sandbox-experimenter` can be assumed by a broader principal (e.g., `Principal: {\"AWS\": \"*\"}` with a weak condition), the deny on sts:AssumeRole *from this role* is moot \u2014 an attacker assumes the role from outside\n\n**Check:**\n\n```bash\naws iam get-role --role-name sandbox-experimenter \\\n  --query 'Role.AssumeRolePolicyDocument'\n```\n\nEnsure the trust policy is scoped to specific, known principals with strong conditions (e.g., `aws:PrincipalArn`, MFA conditions).\n\n---\n\n## Finding 6 \u2014 `ses:*`, `sns:*`, `sqs:*` Not in Deny List\n\n**Severity: MEDIUM**\n\n**The Problem:**\n\nThese services are not blocked and, while they require matching resource ARNs for the Allow to apply (same implicit-deny logic as CloudFormation), they represent exfiltration and abuse vectors if the policy is ever loosened, and SNS/SQS specifically can be used for:\n\n```\nsqs:SendMessage \u2192 exfiltrate data to an attacker-controlled queue\nses:SendEmail   \u2192 exfiltrate data via email\nsns:Publish     \u2192 trigger downstream systems\n```\n\nAgain, currently protected by implicit deny on the Allow side, but not defense-in-depth.\n\n**Fix:** Add to Deny statement: `\"ses:*\"`, `\"sns:*\"`, `\"sqs:SendMessage\"`\n\n---\n\n## Summary Table\n\n| # | Finding | Severity | Currently Exploitable? | Fix |\n|---|---------|----------|----------------------|-----|\n| 1 | `Action: \"*\"` allows `s3:PutBucketPolicy`, `s3:PutBucketNotification`, etc. | **CRITICAL** | **YES** | Replace with explicit S3 action allowlist; deny dangerous bucket-management actions |\n| 2 | `glue:*`, `athena:*` not denied \u2014 code execution under Glue role | **HIGH** | **YES** | Add `glue:*`, `athena:*`, `lakeformation:*` to Deny |\n| 3 | `cloudformation:*` not in Deny | **MEDIUM** | Partially (implicit deny helps) | Add `cloudformation:*` to Deny |\n| 4 | No explicit S3 Deny outside sandbox; permissive bucket policies could grant access | **HIGH** | Conditionally | Add `Deny s3:* NotResource sandbox` |\n| 5 | Trust policy not reviewed \u2014 role assumability unknown | **LOW** | Unknown | Audit trust policy |\n| 6 | `ses:*`, `sns:*`, `sqs:*` not denied | **MEDIUM** | Partially (implicit deny helps) | Add to Deny for defense-in-depth |\n\n---\n\n## Recommended Revised Policy\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"AllowSandboxS3DataOperationsOnly\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:GetObject\",\n        \"s3:PutObject\",\n        \"s3:DeleteObject\",\n        \"s3:ListBucket\",\n        \"s3:GetBucketLocation\",\n        \"s3:AbortMultipartUpload\",\n        \"s3:ListMultipartUploadParts\",\n        \"s3:GetObjectVersion\",\n        \"s3:DeleteObjectVersion\"\n      ],\n      \"Resource\": [\n        \"arn:aws:s3:::team-sandbox-scratch\",\n        \"arn:aws:s3:::team-sandbox-scratch/*\"\n      ]\n    },\n    {\n      \"Sid\": \"DenyAllS3OutsideSandbox\",\n      \"Effect\": \"Deny\",\n      \"Action\": \"s3:*\",\n      \"NotResource\": [\n        \"arn:aws:s3:::team-sandbox-scratch\",\n        \"arn:aws:s3:::team-sandbox-scratch/*\"\n      ]\n    },\n    {\n      \"Sid\": \"DenyEscalationAndExfiltrationServices\",\n      \"Effect\": \"Deny\",\n      \"Action\": [\n        \"iam:*\",\n        \"sts:*\",\n        \"organizations:*\",\n        \"kms:*\",\n        \"lambda:*\",\n        \"ec2:RunInstances\",\n        \"ecs:RunTask\",\n        \"ssm:*\",\n        \"secretsmanager:*\",\n        \"glue:*\",\n        \"athena:*\",\n        \"lakeformation:*\",\n        \"databrew:*\",\n        \"cloudformation:*\",\n        \"codestar:*\",\n        \"codebuild:*\",\n        \"codepipeline:*\",\n        \"sagemaker:CreateTrainingJob\",\n        \"sagemaker:CreateProcessingJob\",\n        \"ses:SendEmail\",\n        \"ses:SendRawEmail\",\n        \"sns:Publish\",\n        \"sqs:SendMessage\",\n        \"s3:PutBucketPolicy\",\n        \"s3:DeleteBucketPolicy\",\n        \"s3:PutBucketAcl\",\n        \"s3:PutBucketNotification\",\n        \"s3:PutBucketWebsite\",\n        \"s3:PutBucketCORS\",\n        \"s3:PutReplicationConfiguration\",\n        \"s3:PutBucketPublicAccessBlock\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}\n```\n\n---\n\n**Bottom line:** The original policy's most dangerous flaw is `Action: \"*\"` \u2014 it is not neutralized by scoping the resource to an S3 bucket. The bucket becomes a control plane, not just a data plane. Fix that first.",
    "agent_chars": 11699,
    "agent_s": 65.4,
    "elapsed_s": 74.7
  },
  {
    "fixture": "02-action-star-blanket-deny",
    "condition": "control",
    "trial": 2,
    "score": 2,
    "scores_by_item": [
      1,
      0,
      0,
      0,
      0,
      1,
      0
    ],
    "notes": [
      "The agent correctly reads Action '*' as expanding to all permissions and recognizes it as scoped to an S3 resource ARN.",
      "Ground truth says there are zero real findings (clean fixture); the agent invents multiple HIGH-severity escalation paths, failing to identify the correct ground truth of no defects.",
      "The agent asserts multiple material defects (Glue escalation, CloudFormation bypass, CodeBuild, bucket policy takeover, EC2 snapshot exfiltration) that the ground truth explicitly says do not exist because the Deny neutralizes them and Action '*' on S3 ARNs doesn't grant other services.",
      "The agent fails cross-statement reasoning: it does not correctly conclude that the Deny on dangerous services caps the Action '*', instead treating the Allow as if the Deny weren't present for most findings.",
      "The agent headlines multiple findings as HIGH severity when the ground truth explicitly states the policy is clean and no escalation paths exist; this is the opposite of correct criticality assessment.",
      "The agent correctly acknowledges it cannot determine the effect of other attached policies, org SCPs, or permission boundaries \u2014 matching the ground-truth boundary.",
      "The ground truth top fix is 'None required'; the agent recommends extensive Deny additions and policy rewrites, directly contradicting the correct answer that no security fix is needed."
    ],
    "agent_output": "## IAM Policy Audit: `role/sandbox-experimenter`\n\n---\n\n### Executive Summary\n\nThe policy has a **structurally sound intent** but contains **multiple meaningful gaps** in the Deny list that leave real escalation and exfiltration paths open. The Allow statement (`Action: \"*\"` scoped to one S3 bucket) is itself the root design smell, but the Deny blocklist is the layer that actually fails.\n\n---\n\n## Finding 1 \u2014 `Action: \"*\"` on S3 Resource Scope Is Broader Than S3\n\n**Severity: HIGH**\n\n**The problem:**\n\n```json\n\"Action\": \"*\",\n\"Resource\": [\n  \"arn:aws:s3:::team-sandbox-scratch\",\n  \"arn:aws:s3:::team-sandbox-scratch/*\"\n]\n```\n\n`Action: \"*\"` does not mean \"all S3 actions.\" It means **every action in every AWS service**. IAM evaluates whether the action *applies* to the resource ARN, but this is more subtle than it appears.\n\nFor S3 ARNs, most non-S3 actions will simply not match and be harmless \u2014 **however**, several cross-service actions legitimately accept S3 ARNs as a resource, including:\n\n- `s3-object-lambda:*` \u2014 Object Lambda access points, not covered by the Deny\n- `s3-outposts:*` \u2014 S3 on Outposts, not covered by the Deny\n- Any future AWS service that accepts `arn:aws:s3:::*` as a resource type\n\nMore critically, the use of `Action: \"*\"` is a **code smell that signals the author is relying entirely on the Deny list for safety**. Any gap in the Deny list directly becomes an exploitable permission. This is defense-in-depth failure.\n\n**Fix:**\n\nReplace the wildcard with an explicit allowlist:\n\n```json\n\"Action\": [\n  \"s3:GetObject\",\n  \"s3:PutObject\",\n  \"s3:DeleteObject\",\n  \"s3:ListBucket\",\n  \"s3:GetBucketLocation\",\n  \"s3:AbortMultipartUpload\",\n  \"s3:ListMultipartUploadParts\"\n],\n\"Resource\": [\n  \"arn:aws:s3:::team-sandbox-scratch\",\n  \"arn:aws:s3:::team-sandbox-scratch/*\"\n]\n```\n\nGrant only what experimentation actually requires. This eliminates the entire class of \"what did we forget to Deny?\" risk.\n\n---\n\n## Finding 2 \u2014 `s3:PutBucketPolicy` / `s3:PutBucketAcl` Allows Bucket Takeover\n\n**Severity: HIGH**\n\n**The problem:**\n\n`Action: \"*\"` on the sandbox bucket includes:\n\n- `s3:PutBucketPolicy`\n- `s3:PutBucketAcl`\n- `s3:PutBucketCORS`\n- `s3:PutBucketWebsite`\n- `s3:DeleteBucketPolicy`\n\nThe principal can **rewrite the bucket policy to grant themselves or any external principal full access**, effectively making the bucket public or sharing it with an attacker-controlled AWS account. This is a **data exfiltration path** that bypasses the Deny list entirely because bucket policy manipulation is a pure S3 action.\n\nExample attack:\n\n```json\n// Attacker calls s3:PutBucketPolicy on team-sandbox-scratch:\n{\n  \"Effect\": \"Allow\",\n  \"Principal\": {\"AWS\": \"arn:aws:iam::ATTACKER_ACCOUNT:root\"},\n  \"Action\": \"s3:*\",\n  \"Resource\": [\n    \"arn:aws:s3:::team-sandbox-scratch\",\n    \"arn:aws:s3:::team-sandbox-scratch/*\"\n  ]\n}\n```\n\nThe bucket is now exfiltrable from an external account, completely outside this AWS account's IAM controls.\n\n**Fix:**\n\nExplicitly deny bucket-level administrative actions, or (preferred) use the explicit allowlist from Finding 1 which simply never grants these actions:\n\n```json\n{\n  \"Sid\": \"DenyBucketAdminActions\",\n  \"Effect\": \"Deny\",\n  \"Action\": [\n    \"s3:PutBucketPolicy\",\n    \"s3:DeleteBucketPolicy\",\n    \"s3:PutBucketAcl\",\n    \"s3:PutBucketCORS\",\n    \"s3:PutBucketWebsite\",\n    \"s3:PutBucketLogging\",\n    \"s3:DeleteBucket\"\n  ],\n  \"Resource\": \"arn:aws:s3:::team-sandbox-scratch\"\n}\n```\n\n---\n\n## Finding 3 \u2014 `glue:*` Not Denied \u2014 Privilege Escalation via Glue Jobs\n\n**Severity: HIGH**\n\n**The problem:**\n\nAWS Glue is entirely absent from the Deny list. Glue jobs execute arbitrary code under a **caller-supplied IAM role**. The attack path:\n\n1. Principal calls `glue:CreateJob` specifying `Role: arn:aws:iam::ACCOUNT:role/high-privilege-role`\n2. Principal calls `glue:StartJobRun`\n3. Glue executes the job **as the high-privilege role**, with full access to exfiltrate credentials, read secrets, call any AWS API\n\nThis is a **full privilege escalation to any role the Glue service can assume**, which in many accounts includes roles far more powerful than sandbox-experimenter.\n\nGlue does not appear in the Deny list. `Action: \"*\"` on an S3 ARN does not grant Glue permissions, **but** this finding highlights the broader problem: the policy relies on a Deny blocklist, and Glue is a well-known escalation vector that was simply missed.\n\n**Fix:**\n\nAdd

# Judge prompt template (control-only screen)

The judge is an LLM given the deterministic reference audit (the copied `_audit.py`, via `scenarios.py`) as ground truth, then asked to score one agent output against the 7-item [`rubric.md`](./rubric.md). It returns JSON only.

The live template is built in `run_eval.py` (`JUDGE_SYSTEM` + `build_judge_prompt`). It is reproduced here so a contributor can swap in a different judge model, or grade by hand, without reading the runner.

## System prompt

```
You are an expert AWS / cloud-security evaluator grading an IAM policy audit against a
7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial
credit.

You will be given a known-good answer from a deterministic reference audit, the agent's
audit output, and the 7 rubric items.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}
```

## User prompt

```
FIXTURE: <id>
PRINCIPAL: <principal>

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): <codes>  (severity: <top_severity>; clean: <clean>)
- What the defect is: <expected_headline>
- Correct top fix: <expected_top_fix>
- Boundary the audit cannot cross: <expected_boundary_join>

AGENT AUDIT OUTPUT:
<agent_output>

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: ... (wildcard Actions expanded; Deny and resource scope read as constraints)
2. Findings: ... (the ground-truth defect by substance, or NO real escalation for a deceptive-clean fixture)
3. No false positives: ... (no invented defect; a Deny-neutralised / scoped / broken-trust / read-only grant is NOT a critical escalation)
4. Cross-statement reasoning: ... (unions statements to catch a buried combo; recognises a Deny/scope that neutralises an apparent combo)
5. Criticality: ... (real escalation is the headline; a neutralised/read-only grant is not critical)
6. Boundary: ... (names a join it cannot make from the policy alone)
7. Recommendation: ... (top fix matches the ground-truth fix, or "no fix needed" for a neutralised policy)

Return JSON only.
```

The placeholders in angle brackets are filled per fixture from `scenarios.py`. The full rubric text is in [`rubric.md`](./rubric.md); the runner inlines a one-line version of each item.

## Why anchor the judge to the reference audit

Without a ground-truth anchor, an LLM judge grades against its own opinion of what an IAM audit should say, which is exactly the thing under test. Feeding it the deterministic `_audit.py` findings (the same ones the replay tests assert) makes the judge score *agreement with a known-good answer* rather than *its own re-derivation*. This matters most on the deceptive-clean fixtures: the anchor tells the judge the correct answer is "no real escalation", so an agent that over-flags is scored against the truth, not against the judge's own (possibly equally over-eager) instinct. The judge can still be wrong; spot-check graded outputs to calibrate trust.

# Control-only screening eval

Is a dedicated `iam-deceptive-escalation-auditor` skill worth building? Only if a COLD agent (no skill, generic "review this policy for problems" prompt) already fails on this domain. This harness measures exactly that. There is **no SKILL.md and no treatment arm**: it runs the control condition only and reports whether the cold agent scores LOW.

The replay tests under `tests/replay_*.py` prove the reference engine produces the intended verdict on every fixture (six deceptive-clean, one buried-hard needle). This eval then runs a live agent against the same fixtures and scores it against a 7-item rubric ([`rubric.md`](./rubric.md)), anchored to that engine as ground truth.

**Screening rule:** build the skill only if the aggregate control mean is **< 4 / 7**. A high control score means the base model already handles deceptive IAM escalation and a skill adds little.

## Quickstart

```bash
# Install the only non-stdlib dependency
pip install anthropic

# Set your API key
export ANTHROPIC_API_KEY=sk-ant-...

# Smoke test: 1 trial, 3 fixtures (two clean, one needle)
python tests/eval/run_eval.py --trials 1 --fixtures 02,05,08

# Full screen: 5 trials, all 7 fixtures (~70 LLM calls)
python tests/eval/run_eval.py --trials 5
```

The script writes raw per-trial results to `eval_results.json` and prints a per-fixture summary table with the aggregate control mean and a screening verdict. `python tests/eval/scenarios.py` prints the deterministic ground-truth findings for every fixture with **no API key required**.

## Resume / crash-safety

The eval is **resumable, and each trial is independent**. Every completed `(fixture, condition, trial)` cell is written to `eval_results.json` immediately via an atomic temp-file rename, and on startup the runner reloads what is already on disk and runs **only the missing cells**. So:

- An interrupt (`Ctrl-C`), a crash, or an API overload mid-run never throws away completed work.
- To finish a partial run, **re-run the exact same command** — it fills the gaps and stops.
- `run_eval.py` also wraps every API call in exponential backoff (`_with_retries`) over 429 / 5xx / "overloaded" errors.
- Pass `--fresh` to ignore an existing results file and start clean.

## The fixtures (why they are all in the cold agent's weak region)

Every fixture is engineered so a single-statement read gives the wrong answer. None is an obvious admin-star the base model trivially flags.

| Fixture | Engine verdict | The trap |
|---|---|---|
| 01 orphaned-passrole-deny | clean | PassRole + RunInstances both present, but an explicit `Deny` on `iam:PassRole` kills the combo. |
| 02 action-star-blanket-deny | clean | `Action '*'` pinned to one sandbox bucket (never Resource `*`), plus a `Deny` on every escalation service. |
| 03 assumerole-broken-trust | clean | `sts:AssumeRole` on an admin-sounding role whose trust does not point back; no `UpdateAssumeRolePolicy` to fix it. |
| 05 iam-mutation-boundary-capped | clean | A full mutation kit (PutRolePolicy/Attach/CreatePolicyVersion/UpdateAssumeRolePolicy/PassRole/CreateAccessKey) scoped to one break-glass ARN, fully capped by a permission-boundary-style `Deny` across Resource `*`. |
| 06 cross-account-assume-condition-gated | clean | A cross-account `sts:AssumeRole` that looks like a pivot but is sealed by an `aws:PrincipalOrgID` + `sts:ExternalId` Condition the principal cannot satisfy; the target trust's wildcard Principal is narrowed by the same condition (no X1). |
| 07 passrole-sandboxed-role-orphaned | clean | `iam:PassRole` + compute verbs (Start/Invoke) read like the launch combo, but the verbs bind no role and the one passable role is read-only: an orphaned escalation with no privilege gain. |
| 08 ml-platform-passrole-launch-needle | E1 (critical) | `iam:PassRole` (`*`) and `sagemaker:CreateTrainingJob` split four policies apart across six attached policies of benign ML-platform bait. |

The cold agent is expected to over-flag the six clean fixtures as critical privilege escalation (item 3), the proven capability gap, and to miss the one buried combo by reading each statement in isolation (item 4).

## What the eval does

For each fixture, the script runs N trials in the control condition: the agent is given the raw IAM policy JSON (permissions policy, plus trust where present) and a generic "audit this policy for misconfigurations" prompt. It uses whatever it brings from training. Each output is graded by an LLM judge against the 7-item rubric, anchored to the deterministic reference audit (`_audit.py`, via `scenarios.py`) as ground truth, so grading is against a known-good answer rather than the judge's own opinion.

## Files

| File | Purpose |
|---|---|
| `run_eval.py` | The control-only runner. Calls the API once per trial, calls the judge, aggregates. Resumable. Needs `ANTHROPIC_API_KEY`. |
| `scenarios.py` | Per-fixture contexts and ground-truth findings (computed by importing the copied `_audit.py`, so they never drift). Runs offline. |
| `rubric.md` | The 7-item rubric, one sentence per item. |
| `judge_prompt.md` | The judge prompt template, for swapping in a different judge. |

## What the eval does NOT measure

- **Lift.** There is no treatment arm; this screen measures only whether the cold baseline is weak enough to justify building the skill.
- **Narrative quality / speed / cost.** Not part of the score.
- **Real-world generalization.** The seven fixtures are constructed, not pulled from production. A low control score here is a *signal* that the domain is hard for a cold agent, not proof of operational value.

# Eval rubric: `iam-deceptive-escalation-auditor` (control-only screen)

Seven binary items (1 = pass, 0 = fail). No partial credit. The judge is given the deterministic reference audit (from the copied `_audit.py`, via `scenarios.py`) as ground truth.

This is a **control-only screening** rubric: there is no SKILL.md and no treatment arm. The point is to measure whether a COLD agent fails on this deliberately deceptive corpus. A skill is worth building only if the cold agent's mean score is LOW (< 4/7).

1. **Parse.** The agent reads the statements correctly and expands wildcard Actions: a `*` or `svc:*` is recognised as the concrete sensitive permissions it grants, not skimmed as "broad". It must also read `Effect: Deny` statements (and resource scopes) as constraints, not ignore them.

2. **Findings.** The agent identifies the defect(s) the ground truth lists for this fixture by substance (the same escalation path; it need not say "E1"). For a **deceptive-clean** fixture, the agent must report *no real escalation* (the apparent danger is neutralised).

3. **No false positives.** The dominant item for this corpus. On the four deceptive-clean fixtures the agent must NOT call a defect that the ground truth lacks: a `iam:PassRole` killed by an explicit `Deny`, an `Action '*'` pinned to one bucket, an `sts:AssumeRole` whose target does not trust the principal back, or a read-only `iam:Get*/List*` wildcard. Manufacturing a "critical privilege escalation" on any of these fails this item.

4. **Cross-statement reasoning.** Two directions. On the **buried-hard needles**, where the escalation is a combination split across 5 attached policies (`iam:PassRole` in one + `lambda:CreateFunction` in another; `iam:UpdateAssumeRolePolicy` + `sts:AssumeRole`; `lambda:UpdateFunctionCode` + `iam:PassRole`), the agent must evaluate the *union* of all statements and name the combo, not clear each statement in isolation. On the **deceptive-clean** fixtures, it must recognise when a `Deny` or a resource scope neutralises an apparent combo. This is the item a cold agent most often gets wrong in BOTH directions.

5. **Criticality.** The agent ranks a real escalation-to-admin (PassRole+compute, trust-rewrite+assume, function-code hijack) as the headline on the needle fixtures, and does NOT headline a neutralised or read-only grant as critical on the clean fixtures.

6. **Boundary.** The agent names at least one thing it cannot determine from the policy alone, matching the ground-truth join (the principal's other attached policies, its permissions boundary, the org SCPs, or the privileges of the role an escalation targets). An agent that presents a single policy read as a complete access verdict fails.

7. **Recommendation.** The agent's top recommended fix matches the ground-truth fix in substance (scope the action/resource, remove the escalation grant, add a narrowing condition) on the needles, or correctly states that no fix is needed because the policy is already neutralised on the clean fixtures.

A perfect audit scores 7. The cold agent is expected to lose points heavily on items 3 (over-flagging the deceptive-clean fixtures), 4 (missing the buried cross-policy combos and missing the neutralising Deny/scope), and 5 (mis-ranking severity), which is exactly why this domain is a candidate for a dedicated skill.

"""
Control-only screening eval for the candidate iam-deceptive-escalation-auditor skill.

There is NO SKILL.md and NO treatment arm here. This harness measures one thing: does a
COLD agent (no skill, generic "review this policy for problems" prompt) fail on this
domain? A skill is worth building only if the cold agent scores LOW (mean < 4/7) on these
deliberately deceptive fixtures.

For each fixture, runs N trials in the control condition only:
- Control: the agent is given the raw IAM policy JSON and a generic "audit this policy"
  prompt. It uses whatever it brings from training.

Each agent output is graded against the 7-item rubric (rubric.md) by an LLM judge, anchored
to the deterministic reference audit (_audit.py, via scenarios.py) as ground truth.

Final report: per-fixture mean control score, stdev across trials, the aggregate mean, and
a screening verdict (build the skill only if the cold agent is weak here).

Requirements:
- ANTHROPIC_API_KEY environment variable.
- `pip install anthropic` (the only non-stdlib dependency in the repo; isolated to tests/eval/).

Usage:
    python tests/eval/run_eval.py --trials 5
    python tests/eval/run_eval.py --trials 1 --fixtures 02,05,07   # smoke test
    python tests/eval/run_eval.py --trials 5 --fresh               # ignore prior results

Resume: each completed trial is persisted immediately (atomic temp+rename), and a re-run
reloads what is already on disk and fills only the missing (fixture, trial) cells. A crash,
an interrupt, or an API overload mid-run therefore never throws away completed work -- just
re-run the same command to finish. Pass --fresh to start clean.

Cost note: 7 fixtures x 1 condition x 5 trials, plus a judge call per output, is
~35 agent calls + ~35 judge calls = ~70 LLM calls. Expect a few dollars depending on model.
"""

from __future__ import annotations

import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path

try:
    from anthropic import Anthropic
except ImportError:
    print("ERROR: anthropic SDK not installed. Run: pip install anthropic", file=sys.stderr)
    sys.exit(1)

sys.path.insert(0, str(Path(__file__).parent))
from scenarios import SCENARIOS, load_fixture_text, expected_audit  # noqa: E402

DEFAULT_AGENT_MODEL = os.environ.get("EVAL_AGENT_MODEL", "claude-sonnet-4-6")
DEFAULT_JUDGE_MODEL = os.environ.get("EVAL_JUDGE_MODEL", "claude-sonnet-4-6")
MAX_TOKENS = 4096

# SKILL.md lives at the skill root (three levels up from tests/eval/).
SKILL_MD = Path(__file__).resolve().parents[2] / "SKILL.md"


def build_control_prompt(scenario: dict) -> str:
    return f"""You are a cloud security engineer reviewing an AWS IAM policy for misconfigurations.

Principal: {scenario['principal']}
Context: {scenario['role']}

Here is the policy, exactly as returned by the IAM API:

{load_fixture_text(scenario)}

Audit this policy. Identify any over-broad grant or privilege-escalation path: anything that
lets this principal do more than it should, or escalate to administrator. For each, give the
severity and the fix. Be specific."""


def build_treatment_prompt(scenario: dict, skill_md_text: str) -> str:
    """Treatment arm: the SAME audit ask and the SAME policy JSON as control, plus SKILL.md
    prepended as the methodology to apply. The only variable between arms is the skill, so
    the lift is attributable to it."""
    return f"""You are a cloud security engineer reviewing an AWS IAM policy for misconfigurations. Apply the methodology below to the policy that follows it.

==== METHODOLOGY (SKILL.md) ====
{skill_md_text}
==== END METHODOLOGY ====

Principal: {scenario['principal']}
Context: {scenario['role']}

Here is the policy, exactly as returned by the IAM API:

{load_fixture_text(scenario)}

Apply the methodology above. Identify any over-broad grant or privilege-escalation path, or
state that an apparent escalation is neutralised. For each finding give the severity and the
fix, and name what you cannot determine from the policy alone. Be specific."""


JUDGE_SYSTEM = """You are an expert AWS / cloud-security evaluator grading an IAM policy audit against a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial credit.

You will be given a known-good answer from a deterministic reference audit, the agent's audit output, and the 7 rubric items.

Return JSON only (no prose), with this exact schema:

{
  "scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
  "notes": ["<one sentence>", ...]
}"""


def build_judge_prompt(scenario: dict, agent_output: str) -> str:
    exp = expected_audit(scenario)
    return f"""FIXTURE: {scenario['id']}
PRINCIPAL: {scenario['principal']}

KNOWN-GOOD ANSWER (from the deterministic reference audit):
- Findings (ground truth): {exp['codes']}  (severity: {exp['top_severity']}; clean: {exp['clean']})
- What the defect is: {scenario['expected_headline']}
- Correct top fix: {scenario['expected_top_fix']}
- Boundary the audit cannot cross: {scenario['expected_boundary_join']}

AGENT AUDIT OUTPUT:
{agent_output}

RUBRIC (score each 1 = pass, 0 = fail):
1. Parse: reads the statements correctly, including expanding wildcard Actions (a `*` or `svc:*` is recognised as the concrete sensitive permissions it grants, not skipped).
2. Findings: identifies the ground-truth defect(s) by substance (the same escalation path or over-broad grant; it need not use the rule code), or correctly reports none for a deceptive-clean fixture.
3. No false positives: does not assert a material defect the ground truth lacks (esp. not calling a Deny-neutralised PassRole, a Resource-scoped Action '*', a broken-back AssumeRole, or a read-only wildcard a live privilege escalation).
4. Cross-statement reasoning: where the defect is a combination (PassRole + a launch action split across attached policies, etc.), the agent evaluates the union of statements rather than clearing each statement in isolation; and conversely recognises when a Deny or a scope neutralises an apparent combo.
5. Criticality: ranks a real escalation-to-admin as most serious, and does NOT headline a neutralised or read-only grant as critical.
6. Boundary: names at least one thing it cannot determine from the policy alone, matching the ground-truth join (other attached policies, permissions boundary, org SCPs, the target role's privileges).
7. Recommendation: top fix matches the ground-truth fix in substance (scope the action/resource, remove the escalation grant, add a condition; or correctly state no fix is needed for a neutralised policy).

Return JSON only."""


RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 529}
MAX_RETRIES = 6


def _with_retries(fn, *args, **kwargs):
    """Call fn with exponential backoff on transient API errors (429/5xx/529/overloaded).

    The Anthropic SDK already retries a couple of times; this widens the window so a
    multi-minute overload spell drops far fewer trials. Re-raises on non-retryable
    errors or once retries are exhausted.

    Returns (result, call_seconds) where call_seconds is the wall-time of the SUCCESSFUL
    attempt only -- backoff sleeps and failed attempts are excluded, so duration metrics
    reflect real audit latency, not how overloaded the API happened to be.
    """
    delay = 2.0
    last_exc = None
    for attempt in range(MAX_RETRIES):
        try:
            t_call = time.time()
            return fn(*args, **kwargs), time.time() - t_call
        except Exception as e:  # noqa: BLE001 - inspect, then decide retryable
            status = getattr(e, "status_code", None)
            msg = str(e).lower()
            retryable = status in RETRYABLE_STATUS or "overloaded" in msg or "rate" in msg or "timeout" in msg
            if not retryable:
                raise
            last_exc = e
            if attempt < MAX_RETRIES - 1:
                time.sleep(delay)
                delay = min(delay * 2, 60.0)
    raise last_exc


def run_agent(client: Anthropic, model: str, prompt: str) -> tuple[str, float]:
    """Returns (agent_output_text, audit_seconds). Seconds excludes retry backoff."""
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=MAX_TOKENS,
            messages=[{"role": "user", "content": prompt}],
        )
    resp, call_s = _with_retries(_call)
    return "".join(block.text for block in resp.content if block.type == "text"), call_s


def run_judge(client: Anthropic, model: str, scenario: dict, agent_output: str) -> dict:
    def _call():
        return client.messages.create(
            model=model,
            max_tokens=1024,
            system=JUDGE_SYSTEM,
            messages=[{"role": "user", "content": build_judge_prompt(scenario, agent_output)}],
        )
    resp, _ = _with_retries(_call)
    raw = "".join(block.text for block in resp.content if block.type == "text").strip()
    if raw.startswith("```"):
        raw = raw.split("```", 2)[1]
        if raw.startswith("json"):
            raw = raw[4:]
        raw = raw.rsplit("```", 1)[0]
    return json.loads(raw.strip())


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--trials", type=int, default=5, help="Trials per fixture (control only)")
    parser.add_argument("--fixtures", default="", help="Comma-separated fixture IDs (prefix match); empty = all")
    parser.add_argument("--agent-model", default=DEFAULT_AGENT_MODEL)
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--output", default="eval_results.json", help="Where to write the raw results")
    parser.add_argument("--fresh", action="store_true", help="Ignore an existing results file and start clean (default: resume/fill gaps)")
    # Default treatment-only: control cells are already on disk from screening and reused.
    parser.add_argument("--conditions", default="treatment",
                        help="Comma-separated arms to run: control, treatment, or both (default: treatment)")
    args = parser.parse_args()

    conditions = [c.strip() for c in args.conditions.split(",") if c.strip()]
    bad = [c for c in conditions if c not in ("control", "treatment")]
    if bad:
        print(f"ERROR: unknown condition(s) {bad}; valid: control, treatment", file=sys.stderr)
        return 2

    if "ANTHROPIC_API_KEY" not in os.environ:
        print("ERROR: ANTHROPIC_API_KEY not set", file=sys.stderr)
        return 1

    skill_md_text = ""
    if "treatment" in conditions:
        if not SKILL_MD.exists():
            print(f"ERROR: treatment arm needs a SKILL.md at {SKILL_MD}", file=sys.stderr)
            return 1
        skill_md_text = SKILL_MD.read_text()

    client = Anthropic()

    to_run = SCENARIOS
    if args.fixtures:
        filters = [f.strip() for f in args.fixtures.split(",")]
        to_run = [s for s in SCENARIOS if any(s["id"].startswith(f) for f in filters)]

    n_cells = len(to_run) * len(conditions) * args.trials
    print(f"LIFT eval [{', '.join(conditions)}]: {len(to_run)} fixtures x {len(conditions)} conditions x {args.trials} trials = {n_cells} agent calls")
    print(f"Agent model: {args.agent_model}, Judge model: {args.judge_model}\n")

    # Resume: reload any completed trials from a prior run so a re-run fills ONLY the
    # gaps (e.g. trials dropped to a transient overload), never redoing finished work.
    # Pass --fresh to ignore an existing results file and start clean.
    results: list[dict] = []
    completed: set[tuple[str, str, int]] = set()
    out_path = Path(args.output)
    if out_path.exists() and not args.fresh:
        try:
            results = json.loads(out_path.read_text())
            completed = {(r["fixture"], r["condition"], r["trial"]) for r in results}
            print(f"Resuming from {args.output}: {len(completed)} trials already complete; filling gaps only.\n")
        except (json.JSONDecodeError, KeyError, OSError):
            results, completed = [], set()

    for scenario in to_run:
        for condition in conditions:
            for trial in range(args.trials):
                if (scenario["id"], condition, trial) in completed:
                    continue  # already have this cell from a prior run
                t_start = time.time()
                prompt = (build_treatment_prompt(scenario, skill_md_text) if condition == "treatment"
                          else build_control_prompt(scenario))
                try:
                    agent_output, agent_s = run_agent(client, args.agent_model, prompt)  # agent_s excludes retry backoff
                    judge_result = run_judge(client, args.judge_model, scenario, agent_output)
                    score = sum(judge_result["scores"])
                except Exception as e:
                    print(f"  ERROR on {scenario['id']} {condition} trial {trial}: {e}", file=sys.stderr)
                    continue
                elapsed = time.time() - t_start  # agent + judge, for cost/wall-clock accounting
                results.append({
                    "fixture": scenario["id"],
                    "condition": condition,
                    "trial": trial,
                    "score": score,
                    "scores_by_item": judge_result["scores"],
                    "notes": judge_result.get("notes", []),
                    "agent_output": agent_output,
                    "agent_chars": len(agent_output),
                    "agent_s": round(agent_s, 1),
                    "elapsed_s": round(elapsed, 1),
                })
                # Crash-safe: persist after every trial via atomic temp+rename so an
                # overload-induced death never throws away completed work.
                tmp = Path(str(args.output) + ".tmp")
                tmp.write_text(json.dumps(results, indent=2))
                tmp.replace(args.output)
                print(f"  {scenario['id']:<40} | {condition:9s} | trial {trial} | score {score}/7 | audit {agent_s:.0f}s", flush=True)

    Path(args.output).write_text(json.dumps(results, indent=2))
    print(f"\nRaw results: {args.output}\n")
    print_summary(results, to_run)
    return 0


def print_summary(results: list[dict], to_run: list[dict]) -> None:
    ctrl: dict[str, list[int]] = {}
    treat: dict[str, list[int]] = {}
    for r in results:
        bucket = ctrl if r.get("condition") == "control" else treat
        bucket.setdefault(r["fixture"], []).append(r["score"])

    print(f"{'Fixture':<40} {'Control':>8} {'Treat':>8} {'Lift':>8} {'Nc':>4} {'Nt':>4}")
    print("-" * 80)
    c_means: list[float] = []
    t_means: list[float] = []
    lifts: list[float] = []
    paired_ids: list[str] = []
    for scenario in to_run:
        cs = ctrl.get(scenario["id"], [])
        ts = treat.get(scenario["id"], [])
        if not cs and not ts:
            continue
        c = statistics.mean(cs) if cs else float("nan")
        t = statistics.mean(ts) if ts else float("nan")
        c_str = f"{c:>8.2f}" if cs else f"{'n/a':>8}"
        t_str = f"{t:>8.2f}" if ts else f"{'n/a':>8}"
        if cs and ts:
            lift = t - c
            lifts.append(lift); c_means.append(c); t_means.append(t); paired_ids.append(scenario["id"])
            l_str = f"{lift:>+8.2f}"
            flag = "  <- treat still <5" if t < 5.0 else ("  <- no lift" if lift <= 0 else "")
        else:
            l_str = f"{'-':>8}"
            flag = ""
        print(f"{scenario['id']:<40} {c_str} {t_str} {l_str} {len(cs):>4} {len(ts):>4}{flag}")
    print("-" * 80)

    if not lifts:
        print("\nNo paired control/treatment fixtures to summarize "
              f"(control: {sum(len(v) for v in ctrl.values())} cells, "
              f"treatment: {sum(len(v) for v in treat.values())} cells).")
        return

    c_agg = statistics.mean(c_means)
    t_agg = statistics.mean(t_means)
    print(f"\nAggregate: control {c_agg:.2f}/7  ->  treatment {t_agg:.2f}/7   (lift {t_agg - c_agg:+.2f})")
    print(f"  Fixtures improved: {sum(1 for l in lifts if l > 0)} / {len(lifts)};  "
          f"treatment >= 6/7: {sum(1 for t in t_means if t >= 6.0)} / {len(t_means)};  "
          f"treatment >= 5/7: {sum(1 for t in t_means if t >= 5.0)} / {len(t_means)}")
    weakest = min(zip(t_means, paired_ids))
    print(f"  Weakest treatment fixture: {weakest[1]} at {weakest[0]:.2f}/7 "
          "(the next one to close with a SKILL.md edit)")


if __name__ == "__main__":
    sys.exit(main())

"""
Per-fixture principal contexts and expected answers, used by run_eval.py.

The ground-truth findings are NOT hand-written. The "expected_audit" function runs the
copied reference engine (_audit.py) against each fixture, so the codes / severity / clean
flag the judge is anchored to are exactly what the engine computes (see tests/replay_*.py).
The "expected_headline / fix / boundary" strings are human-readable framing for the judge
prompt; they describe the SAME verdict the engine grounds, never a different one.

This screening harness is CONTROL-ONLY: there is no SKILL.md and no treatment arm. The set
targets the cold agent's proven weak region, OVER-FLAGGING neutralised policies: six
deceptive-clean fixtures that look like critical escalation but are capped (the engine finds
nothing), each with a distinct neutralisation mechanism (explicit Deny on PassRole; Action '*'
scoped + service Deny; broken trust; a permission-boundary Deny over a full mutation kit; a
cross-account AssumeRole sealed by an unsatisfiable Condition; a PassRole whose only passable
role is read-only and whose compute verbs bind no role). One buried-hard needle is kept where a
real escalation (E1) only emerges from the union of six attached policies / ~16 statements.

Stdlib only. No external dependencies.
"""

from __future__ import annotations

import json
import sys
from pathlib import Path

TESTS_DIR = Path(__file__).resolve().parent.parent
FIXTURES_DIR = TESTS_DIR.parent / "fixtures"

sys.path.insert(0, str(TESTS_DIR))
from _audit import run_audit  # noqa: E402

# Each entry pairs a fixture with the human-readable context the eval feeds the agent,
# plus the headline / fix / boundary that match the deterministic audit's verdict (the
# judge's anchor). Keep this list aligned with the replay_*.py files under tests/.
SCENARIOS = [
    {
        "id": "01-orphaned-passrole-deny",
        "principal": "role/build-fleet-runner",
        "role": "Runs an EC2 build fleet. Granted ec2:RunInstances and, in a separate statement, iam:PassRole; a third statement is an explicit Deny on iam:PassRole.",
        "expected_headline": "No real escalation. PassRole + RunInstances looks like the classic E1 combo, but an explicit Deny on iam:PassRole (Resource '*') overrides the scoped Allow, so the PassRole half is dead and the combo cannot complete.",
        "expected_top_fix": "None on the escalation: it is already neutralised by the Deny. Optionally remove the now-inert PassRole Allow and the retired instance-profile reference to reduce confusion.",
        "expected_boundary_join": "the privileges of the roles PassRole could pass IF it were allowed (moot here), and the principal's other attached policies / permissions boundary / org SCPs.",
    },
    {
        "id": "02-action-star-blanket-deny",
        "principal": "role/sandbox-experimenter",
        "role": "An experimentation role. Granted Action '*' on a single sandbox S3 bucket, with a Deny on every escalation-bearing service (iam, sts, kms, lambda, ec2:RunInstances, ssm, secretsmanager) on Resource '*'.",
        "expected_headline": "No real escalation. Action '*' reads like AdministratorAccess, but it is pinned to one sandbox bucket (never Resource '*'), and the Deny removes every dangerous service. The star expands to nothing useful outside one scratch bucket.",
        "expected_top_fix": "None required for security: the star is already scoped and the Deny caps it. Optionally replace Action '*' with the concrete S3 actions in use for clarity.",
        "expected_boundary_join": "whether anything outside this document re-grants the denied services (other attached policies could not, since an explicit Deny wins), and the org SCPs.",
    },
    {
        "id": "03-assumerole-broken-trust",
        "principal": "role/deploy-orchestrator",
        "role": "Granted sts:AssumeRole on role/org-admin-break-glass (an admin-sounding target) and read access to a deploy-config bucket. The target role's trust policy is supplied.",
        "expected_headline": "No real escalation. sts:AssumeRole on an admin role sounds like a lateral move, but the target's trust policy only trusts two specific operator roles behind an ExternalId, not this principal, and this principal has no iam:UpdateAssumeRolePolicy to rewrite it. The path is broken.",
        "expected_top_fix": "None: the AssumeRole grant is inert because the trust does not point back. Optionally remove the unused AssumeRole grant.",
        "expected_boundary_join": "the actual contents of the target role's trust policy and whether any OTHER principal this role can reach closes the loop (confirmed broken here from the supplied trust).",
    },
    {
        "id": "05-iam-mutation-boundary-capped",
        "principal": "role/identity-platform-operator",
        "role": "An identity-platform role across two attached policies. policy-1 grants a full mutation kit (iam:PutRolePolicy, AttachRolePolicy, CreatePolicyVersion, SetDefaultPolicyVersion, UpdateAssumeRolePolicy, PassRole, CreateAccessKey) scoped to one break-glass role/policy ARN; policy-2 is a permission-boundary-style explicit Deny on every one of those actions (plus sts:AssumeRole and the credential-minting set) across Resource '*'.",
        "expected_headline": "No real escalation. policy-1 looks like a full identity-takeover kit (every E2/E4/E5/E6 primitive), but policy-2's explicit Deny on all of them across Resource '*' wins over the Allow, so the effective permission set collapses to read-only IAM inventory. The mutation kit is fully capped.",
        "expected_top_fix": "None for security: the Deny boundary already neutralises the kit. Optionally remove the now-inert mutation Allow so the policy reads honestly, and keep the Deny as the boundary.",
        "expected_boundary_join": "whether any other attached policy or the org SCPs re-grant the denied actions (they cannot, since an explicit Deny wins), and what the break-glass role/policy ARN would have permitted if the Deny were removed.",
    },
    {
        "id": "06-cross-account-assume-condition-gated",
        "principal": "role/cost-reporting-collector",
        "role": "A cost-reporting role. policy-1 grants sts:AssumeRole on a role in a DIFFERENT account (905512347781) behind an sts:ExternalId + aws:PrincipalOrgID Condition; the target role's trust policy is supplied and uses a wildcard Principal narrowed by the same org-id + ExternalId condition. The rest is read-only billing/cost access.",
        "expected_headline": "No real escalation. The cross-account sts:AssumeRole reads like a pivot into a foreign account, but it is gated by an sts:ExternalId + aws:PrincipalOrgID Condition this principal cannot satisfy, and the target's trust wildcard Principal is fully narrowed by the same condition (so no open trust). The principal has no iam:UpdateAssumeRolePolicy to relax either side. The path is condition-sealed at both ends.",
        "expected_top_fix": "None: the AssumeRole grant is inert because the condition cannot be met and the trust does not open. Optionally remove the unused cross-account AssumeRole grant.",
        "expected_boundary_join": "whether the principal can ever present the required org-id / ExternalId (it cannot from this identity), and what the foreign-account target role can do if the path were ever opened.",
    },
    {
        "id": "07-passrole-sandboxed-role-orphaned",
        "principal": "role/sandbox-compute-operator",
        "role": "A sandbox compute-operator role across three attached policies. policy-1 grants iam:PassRole scoped to one role, sandbox-readonly-compute; policy-2 grants compute verbs (ec2:StartInstances, ecs:StartTask, lambda:InvokeFunction) on existing compute; policy-3 is the passed role's OWN policy, which is strictly read-only.",
        "expected_headline": "No real escalation. iam:PassRole plus compute verbs reads like the E1 launch combo, but Start/Invoke operate on EXISTING compute and accept no PassRole argument, so there is no role-binding launch action (RunInstances/CreateFunction/RunTask) to pair PassRole with. And the one passable role is read-only, no more privileged than the caller. An orphaned escalation: the shape is there, the gain is not.",
        "expected_top_fix": "None for security: the combo cannot complete and the passed role grants nothing extra. Optionally remove the unused PassRole grant if no role-binding launch action will be added later.",
        "expected_boundary_join": "the actual privileges of sandbox-readonly-compute (confirmed read-only here from the attached policy), and whether any future policy adds a role-binding launch action that would re-arm the combo.",
    },
    {
        "id": "08-ml-platform-passrole-launch-needle",
        "principal": "role/ml-training-platform",
        "role": "A SageMaker training platform across six attached policies (~16 statements). iam:PassRole on Resource '*' is granted in one policy (framed as passing the training execution role); sagemaker:CreateTrainingJob is granted in another; the rest are routine read/queue/experiment/output permissions.",
        "expected_headline": "Real privilege escalation (critical). iam:PassRole on Resource '*' plus sagemaker:CreateTrainingJob is the E1 combo: launch a training job with ANY role in the account attached, then use that job's credentials. The two halves sit four policies apart behind heavy benign bait, so a per-statement read clears every statement; only the union is critical.",
        "expected_top_fix": "Scope iam:PassRole to the exact execution-role ARNs the training platform must pass (never '*'), with an iam:PassedToService condition pinning it to sagemaker, or remove CreateTrainingJob from this role.",
        "expected_boundary_join": "the privileges of the roles iam:PassRole can pass (not in this policy): the escalation's blast radius is whatever the most-privileged passable role can do.",
    },
]


def fixture_dir(scenario: dict) -> Path:
    return FIXTURES_DIR / scenario["id"]


def load_fixture_text(scenario: dict) -> str:
    """The raw policy JSON the agent is given: the permissions policy, plus trust / boundary if present."""
    d = fixture_dir(scenario)
    parts = []
    for path in sorted(d.glob("policy*.json")):
        doc = json.loads(path.read_text())
        parts.append(f"PERMISSIONS POLICY ({path.name}):\n" + json.dumps(doc, indent=2))
    trust = d / "trust-policy.json"
    if trust.exists():
        parts.append("TRUST POLICY (AssumeRolePolicyDocument):\n" + json.dumps(json.loads(trust.read_text()), indent=2))
    boundary = d / "boundary.json"
    if boundary.exists():
        parts.append("PERMISSIONS BOUNDARY (boundary.json):\n" + json.dumps(json.loads(boundary.read_text()), indent=2))
    return "\n\n".join(parts)


def expected_audit(scenario: dict) -> dict:
    """Run the deterministic reference audit to get the ground-truth findings for the judge."""
    audit = run_audit(fixture_dir(scenario))
    return {
        "codes": sorted(audit.codes()),
        "top_severity": audit.top_severity,
        "clean": audit.clean,
        "boundary_count": len(audit.boundary),
    }


if __name__ == "__main__":
    # `python tests/eval/scenarios.py` prints the ground-truth answers, no API needed.
    for s in SCENARIOS:
        exp = expected_audit(s)
        print(f"{s['id']:<40} codes={str(exp['codes']):<10} top={str(exp['top_severity']):<9} clean={exp['clean']}")