{
  "skill_name": "bedrock-operation-review",
  "skill_path": "/tools-for-devops-agent/skills/bedrock-operation-review",
  "eval_count": 5,
  "runs_per_eval": 1,
  "metadata": {
    "timestamp": "2026-09-11T23:25:30Z"
  },
  "runs": [
    {
      "eval_id": "bedrock-review-smoke-test",
      "run_index": 0,
      "with_skill": {
        "eval_id": "bedrock-review-smoke-test",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "matches regex /[a-z]{2}-[a-z]+-\\d/",
            "passed": true,
            "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'model' or contains 'guardrail'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'model' \u2014 Substring found: 'model'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'region'",
            "passed": true,
            "evidence": "Substring found: 'region'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'account'",
            "passed": true,
            "evidence": "Substring found: 'account'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 4,
            "output_tokens": 269,
            "cache_read_input_tokens": 51574,
            "cache_creation_input_tokens": 29112
          }
        },
        "timing": {
          "elapsed_seconds": 15.202140458975919
        },
        "raw_output": "| Name | Type | Region | Account |\n|---|---|---|---|\n| anthropic.claude-3-5-sonnet-20241022-v2:0 | model | us-east-1 | $accountId |\n| pii-content-filter | guardrail | us-east-1 | $accountId |\n\nNote: the `account` field is the literal placeholder `$accountId` in both entries, not a real account ID."
      },
      "without_skill": {
        "eval_id": "bedrock-review-smoke-test",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "matches regex /[a-z]{2}-[a-z]+-\\d/",
            "passed": true,
            "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'model' or contains 'guardrail'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'model' \u2014 Substring found: 'model'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'region'",
            "passed": true,
            "evidence": "Substring found: 'region'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'account'",
            "passed": true,
            "evidence": "Substring found: 'account'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "Without skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 4,
            "output_tokens": 244,
            "cache_read_input_tokens": 42641,
            "cache_creation_input_tokens": 20178
          }
        },
        "timing": {
          "elapsed_seconds": 11.425920790992677
        },
        "raw_output": "| Name | Type | Region | Account |\n|---|---|---|---|\n| anthropic.claude-3-5-sonnet-20241022-v2:0 | model | us-east-1 | $accountId |\n| pii-content-filter | guardrail | us-east-1 | $accountId |\n\nAccount is the literal placeholder `$accountId` in the file, not a resolved ID."
      },
      "delta_pass_rate": 0.0
    },
    {
      "eval_id": "bedrock-review-artifact-naming",
      "run_index": 0,
      "with_skill": {
        "eval_id": "bedrock-review-artifact-naming",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/",
            "passed": true,
            "evidence": "Regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/ matched",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "matches regex /[a-z]{2}-[a-z]+-\\d/",
            "passed": true,
            "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 4,
            "output_tokens": 355,
            "cache_read_input_tokens": 48145,
            "cache_creation_input_tokens": 29385
          }
        },
        "timing": {
          "elapsed_seconds": 14.294720083009452
        },
        "raw_output": "First resource: `anthropic.claude-3-5-sonnet-20241022-v2:0` (type `model`), region `us-east-1`.\n\nReport artifact filename:\n\n```\nbedrock-review-$accountId-us-east-1-2026-09-11.md\n```\n\nNote: the `account` field is the literal placeholder `$accountId` in the context file, so substitute the real account ID once known (e.g. `bedrock-review-123456789012-us-east-1-2026-09-11.md`)."
      },
      "without_skill": {
        "eval_id": "bedrock-review-artifact-naming",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/",
            "passed": false,
            "evidence": "Regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/ did not match",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "matches regex /[a-z]{2}-[a-z]+-\\d/",
            "passed": true,
            "evidence": "Regex /[a-z]{2}-[a-z]+-\\d/ matched",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 0.5,
        "summary": "Without skill: 50% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 4,
            "output_tokens": 189,
            "cache_read_input_tokens": 42669,
            "cache_creation_input_tokens": 20205
          }
        },
        "timing": {
          "elapsed_seconds": 14.406852500047535
        },
        "raw_output": "First resource: model `anthropic.claude-3-5-sonnet-20241022-v2:0` in region `us-east-1`.\n\nReport artifact filename: `bedrock-review-2026-09-11.md`"
      },
      "delta_pass_rate": 0.5
    },
    {
      "eval_id": "bedrock-review-data-source-priority",
      "run_index": 0,
      "with_skill": {
        "eval_id": "bedrock-review-data-source-priority",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'Bedrock' or contains 'bedrock'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'bedrock' \u2014 Substring found: 'bedrock'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'CloudWatch' or contains 'cloudwatch'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'cloudwatch' \u2014 Substring found: 'cloudwatch'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'listguardrails' \u2014 Substring found: 'listguardrails'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'not' or contains 'no '",
            "passed": true,
            "evidence": "OR satisfied by: contains 'not' \u2014 Substring found: 'not'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 2,
            "output_tokens": 438,
            "cache_read_input_tokens": 11400,
            "cache_creation_input_tokens": 25355
          }
        },
        "timing": {
          "elapsed_seconds": 21.47895508306101
        },
        "raw_output": "The skill collects data exclusively through native AWS control-plane and metrics APIs. Specific examples it names:\n\n- **`bedrock`** \u2014 e.g. `bedrock.ListGuardrails`, `bedrock.ListFoundationModels` (catalog reference only), `bedrock.GetModelInvocationLoggingConfiguration`, `bedrock.ListProvisionedModelThroughputs`\n- **`bedrockagent`** \u2014 e.g. `bedrockagent.ListAgents`, `bedrockagent.ListKnowledgeBases`, `bedrockagent.GetAgentVersion`\n- **`cloudwatch`** \u2014 e.g. `cloudwatch.ListMetrics` and `cloudwatch.GetMetricData` against the `AWS/Bedrock` namespace\n- **`servicequotas`** \u2014 e.g. `servicequotas.GetServiceQuota`\n- **`ec2`** \u2014 for self-managed GPU instance context\n\n**No foundation model is invoked, and no prompt/response content is read.** The skill explicitly performs no data-plane model invocations. It reads only metadata/presence for prompts (`ListPrompts`, never `GetPrompt`) and, for agent versions, reads only the model ID, orchestration type, and whether a prompt override exists \u2014 not the override template text."
      },
      "without_skill": {
        "eval_id": "bedrock-review-data-source-priority",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'Bedrock' or contains 'bedrock'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'bedrock': Substring not found: 'bedrock'; contains 'bedrock': Substring not found: 'bedrock'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'CloudWatch' or contains 'cloudwatch'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'cloudwatch': Substring not found: 'cloudwatch'; contains 'cloudwatch': Substring not found: 'cloudwatch'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'listguardrails': Substring not found: 'listguardrails'; contains 'getmetricdata': Substring not found: 'getmetricdata'; contains 'listmetrics': Substring not found: 'listmetrics'; contains 'getservicequota': Substring not found: 'getservicequota'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'not' or contains 'no '",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'not': Substring not found: 'not'; contains 'no ': Substring not found: 'no '",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "Without skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.0119533330435
        },
        "raw_output": ""
      },
      "delta_pass_rate": 1.0
    },
    {
      "eval_id": "bedrock-review-severity-definitions",
      "run_index": 0,
      "with_skill": {
        "eval_id": "bedrock-review-severity-definitions",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'CRITICAL'",
            "passed": true,
            "evidence": "Substring found: 'critical'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'HIGH'",
            "passed": true,
            "evidence": "Substring found: 'high'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'MEDIUM'",
            "passed": true,
            "evidence": "Substring found: 'medium'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'LOW'",
            "passed": true,
            "evidence": "Substring found: 'low'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'INFO'",
            "passed": true,
            "evidence": "Substring found: 'info'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 2,
            "output_tokens": 39,
            "cache_read_input_tokens": 11400,
            "cache_creation_input_tokens": 28764
          }
        },
        "timing": {
          "elapsed_seconds": 7.668922417098656
        },
        "raw_output": "From most to least urgent: **CRITICAL, HIGH, MEDIUM, LOW, INFO**."
      },
      "without_skill": {
        "eval_id": "bedrock-review-severity-definitions",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'CRITICAL'",
            "passed": false,
            "evidence": "Substring not found: 'critical'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'HIGH'",
            "passed": false,
            "evidence": "Substring not found: 'high'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'MEDIUM'",
            "passed": false,
            "evidence": "Substring not found: 'medium'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'LOW'",
            "passed": false,
            "evidence": "Substring not found: 'low'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'INFO'",
            "passed": false,
            "evidence": "Substring not found: 'info'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "Without skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.00813362502959
        },
        "raw_output": ""
      },
      "delta_pass_rate": 1.0
    },
    {
      "eval_id": "bedrock-review-pillars",
      "run_index": 0,
      "with_skill": {
        "eval_id": "bedrock-review-pillars",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'Security' or contains 'security'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'security' \u2014 Substring found: 'security'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Performance' or contains 'performance'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'performance' \u2014 Substring found: 'performance'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Quota' or contains 'quota'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'quota' \u2014 Substring found: 'quota'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Cost' or contains 'cost'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'cost' \u2014 Substring found: 'cost'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Resilience' or contains 'resilience'",
            "passed": true,
            "evidence": "OR satisfied by: contains 'resilience' \u2014 Substring found: 'resilience'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 2,
            "output_tokens": 256,
            "cache_read_input_tokens": 11400,
            "cache_creation_input_tokens": 28757
          }
        },
        "timing": {
          "elapsed_seconds": 11.560605249949731
        },
        "raw_output": "The skill evaluates five pillars, mirroring the AWS Well-Architected / Bedrock operational review structure:\n\n1. **Security** \u2014 guardrails, invocation logging, KB encryption/config, VPC for customization jobs, IAM least privilege, prompt injection exposure, model access\n2. **Performance** \u2014 latency/throttling, agent optimization, version drift, model lifecycle, data automation, service tiers\n3. **Service Quotas** \u2014 model RPM/TPM (incl. CRIS) and guardrail quota utilization\n4. **Cost Optimization** \u2014 inference profiles, distillation, prompt caching, provisioned throughput, batch inference, EC2 GPU utilization\n5. **Resilience** \u2014 Cross-Region Inference (CRIS) adoption"
      },
      "without_skill": {
        "eval_id": "bedrock-review-pillars",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "contains 'Security' or contains 'security'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'security': Substring not found: 'security'; contains 'security': Substring not found: 'security'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Performance' or contains 'performance'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'performance': Substring not found: 'performance'; contains 'performance': Substring not found: 'performance'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Quota' or contains 'quota'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'quota': Substring not found: 'quota'; contains 'quota': Substring not found: 'quota'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Cost' or contains 'cost'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'cost': Substring not found: 'cost'; contains 'cost': Substring not found: 'cost'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "contains 'Resilience' or contains 'resilience'",
            "passed": false,
            "evidence": "No OR branch satisfied \u2014 contains 'resilience': Substring not found: 'resilience'; contains 'resilience': Substring not found: 'resilience'",
            "method": "deterministic",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "Without skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.01407870801631
        },
        "raw_output": ""
      },
      "delta_pass_rate": 1.0
    }
  ],
  "run_summary": {
    "with_skill": {
      "mean_pass_rate": 1.0,
      "stddev_pass_rate": 0.0,
      "mean_tokens": 271.4,
      "mean_input_tokens": 2.8,
      "mean_output_tokens": 271.4,
      "mean_total_tokens": 274.2,
      "mean_tool_calls": 0.4
    },
    "without_skill": {
      "mean_pass_rate": 0.3,
      "stddev_pass_rate": 0.4472,
      "mean_tokens": 86.6,
      "mean_input_tokens": 1.6,
      "mean_output_tokens": 86.6,
      "mean_total_tokens": 88.2,
      "mean_tool_calls": 0.4
    },
    "delta": {
      "pass_rate": 0.7,
      "tokens": 184.8,
      "total_tokens": 186.0,
      "input_tokens": 1.2,
      "tool_calls": 0.0
    },
    "cost_efficiency": {
      "quality_delta": 0.7,
      "cost_delta_pct": 210.9,
      "classification": "TRADEOFF",
      "emoji": "\ud83d\udfe1",
      "description": "Skill improves quality but increases cost"
    },
    "estimated_cost": {
      "with_skill_per_run": {
        "input_cost": 8e-06,
        "output_cost": 0.004071,
        "total_cost": 0.004079,
        "model": "sonnet",
        "currency": "USD"
      },
      "without_skill_per_run": {
        "input_cost": 5e-06,
        "output_cost": 0.001299,
        "total_cost": 0.001304,
        "model": "sonnet",
        "currency": "USD"
      },
      "per_eval_pair": 0.005383,
      "total_runs": 5,
      "total_cost": 0.0269,
      "model": "sonnet",
      "currency": "USD"
    }
  },
  "scores": {
    "outcome": 1.0,
    "process": 0.4,
    "style": 1.0,
    "efficiency": 1.0,
    "overall": 0.85
  },
  "passed": true
}

[
  {
    "id": "bedrock-review-smoke-test",
    "prompt": "Read bedrock-context.json. List each resource's name, type (model or guardrail), region, and account. No analysis needed.",
    "expected_output": "Lists every resource from files/bedrock-context.json with its name, type, region, and account exactly as defined in the file.",
    "files": ["files/bedrock-context.json"],
    "assertions": [
      "matches regex /[a-z]{2}-[a-z]+-\\d/",
      "contains 'model' or contains 'guardrail'",
      "contains 'region'",
      "contains 'account'"
    ]
  },
  {
    "id": "bedrock-review-artifact-naming",
    "prompt": "Read bedrock-context.json. Identify the FIRST resource listed and its region, then state the report artifact filename you would generate for this review (use today's date in YYYY-MM-DD form). Do not run the audit.",
    "expected_output": "Identifies the first resource and region from files/bedrock-context.json and proposes an artifact name of the form bedrock-review-<account-id>-<region>-<YYYY-MM-DD>.md.",
    "files": ["files/bedrock-context.json"],
    "assertions": [
      "matches regex /bedrock-review-[a-z0-9-]+-\\d{4}-\\d{2}-\\d{2}\\.md/",
      "matches regex /[a-z]{2}-[a-z]+-\\d/"
    ]
  },
  {
    "id": "bedrock-review-data-source-priority",
    "prompt": "According to the skill, which AWS APIs does the Bedrock operational review use to collect data? Name at least three specific API calls or namespaces the skill mentions. Confirm whether the skill invokes any foundation model or reads prompt/response content. No account access required.",
    "expected_output": "States the skill uses Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and EC2 APIs and explicitly does NOT invoke any foundation model or read prompt/response content. Names specific calls such as bedrock.ListGuardrails, cloudwatch.GetMetricData, cloudwatch.ListMetrics, or servicequotas.GetServiceQuota.",
    "files": [],
    "assertions": [
      "contains 'Bedrock' or contains 'bedrock'",
      "contains 'CloudWatch' or contains 'cloudwatch'",
      "contains 'ListGuardrails' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'GetServiceQuota'",
      "contains 'not' or contains 'no '"
    ]
  },
  {
    "id": "bedrock-review-severity-definitions",
    "prompt": "List the five severity levels the skill uses for findings, in order from most to least urgent. No account access required.",
    "expected_output": "Lists CRITICAL, HIGH, MEDIUM, LOW, INFO in that order.",
    "files": [],
    "assertions": [
      "contains 'CRITICAL'",
      "contains 'HIGH'",
      "contains 'MEDIUM'",
      "contains 'LOW'",
      "contains 'INFO'"
    ]
  },
  {
    "id": "bedrock-review-pillars",
    "prompt": "List the pillars the skill evaluates during a Bedrock operational review. No account access required.",
    "expected_output": "Mentions Security, Performance, Service Quotas, Cost Optimization, and Resilience.",
    "files": [],
    "assertions": [
      "contains 'Security' or contains 'security'",
      "contains 'Performance' or contains 'performance'",
      "contains 'Quota' or contains 'quota'",
      "contains 'Cost' or contains 'cost'",
      "contains 'Resilience' or contains 'resilience'"
    ]
  }
]

[
  {"query": "Which skill would help me run a Bedrock operational review? Just name it; do not run it.", "should_trigger": true},
  {"query": "Is there a skill available for auditing Amazon Bedrock workloads against best practices? Answer yes or no with the skill name; do not execute it.", "should_trigger": true},
  {"query": "Name the skill that covers Bedrock security, cost optimization, and quota reviews. Do not run any audit.", "should_trigger": true},
  {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false},
  {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false},
  {"query": "Create a CloudFormation template for an S3 bucket", "should_trigger": false}
]

{
  "skill_name": "bedrock-operation-review",
  "skill_path": "/tools-for-devops-agent/skills/bedrock-operation-review",
  "timestamp": "2026-09-11T23:29:01Z",
  "overall_score": 0.932,
  "overall_grade": "A",
  "passed": true,
  "sections": {
    "audit": {
      "score": 98,
      "grade": "A",
      "passed": true,
      "normalized": 0.98,
      "critical": 0,
      "warning": 0,
      "info": 1
    },
    "functional": {
      "overall": 0.85,
      "grade": "B",
      "passed": true,
      "scores": {
        "outcome": 1.0,
        "process": 0.4,
        "style": 1.0,
        "efficiency": 1.0,
        "overall": 0.85
      },
      "cost_efficiency": {
        "quality_delta": 0.7,
        "cost_delta_pct": 210.9,
        "classification": "TRADEOFF",
        "emoji": "\ud83d\udfe1",
        "description": "Skill improves quality but increases cost"
      },
      "estimated_cost": {
        "with_skill_per_run": {
          "input_cost": 8e-06,
          "output_cost": 0.004071,
          "total_cost": 0.004079,
          "model": "sonnet",
          "currency": "USD"
        },
        "without_skill_per_run": {
          "input_cost": 5e-06,
          "output_cost": 0.001299,
          "total_cost": 0.001304,
          "model": "sonnet",
          "currency": "USD"
        },
        "per_eval_pair": 0.005383,
        "total_runs": 5,
        "total_cost": 0.0269,
        "model": "sonnet",
        "currency": "USD"
      }
    },
    "trigger": {
      "pass_rate": 1.0,
      "grade": "A",
      "passed": true,
      "total_queries": 6
    }
  }
}

{
  "skill_name": "bedrock-operation-review",
  "skill_path": "/tools-for-devops-agent/skills/bedrock-operation-review",
  "query_results": [
    {
      "query": "Which skill would help me run a Bedrock operational review? Just name it; do not run it.",
      "should_trigger": true,
      "trigger_count": 3,
      "run_count": 3,
      "trigger_rate": 1.0,
      "passed": true,
      "mean_input_tokens": 2.0,
      "mean_output_tokens": 21.3,
      "mean_total_tokens": 23.3
    },
    {
      "query": "Is there a skill available for auditing Amazon Bedrock workloads against best practices? Answer yes or no with the skill name; do not execute it.",
      "should_trigger": true,
      "trigger_count": 3,
      "run_count": 3,
      "trigger_rate": 1.0,
      "passed": true,
      "mean_input_tokens": 2.0,
      "mean_output_tokens": 19.0,
      "mean_total_tokens": 21.0
    },
    {
      "query": "Name the skill that covers Bedrock security, cost optimization, and quota reviews. Do not run any audit.",
      "should_trigger": true,
      "trigger_count": 3,
      "run_count": 3,
      "trigger_rate": 1.0,
      "passed": true,
      "mean_input_tokens": 2.0,
      "mean_output_tokens": 30.0,
      "mean_total_tokens": 32.0
    },
    {
      "query": "Write a Python script that sorts a list of numbers",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 2.7,
      "mean_output_tokens": 433.3,
      "mean_total_tokens": 436.0
    },
    {
      "query": "What's the weather forecast for Sydney this weekend?",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 2.0,
      "mean_output_tokens": 101.0,
      "mean_total_tokens": 103.0
    },
    {
      "query": "Create a CloudFormation template for an S3 bucket",
      "should_trigger": false,
      "trigger_count": 1,
      "run_count": 3,
      "trigger_rate": 0.3333,
      "passed": true,
      "mean_input_tokens": 5.3,
      "mean_output_tokens": 1548.3,
      "mean_total_tokens": 1553.7
    }
  ],
  "summary": {
    "total_queries": 6,
    "passed": 6,
    "failed": 0,
    "trigger_precision": 1.0,
    "no_trigger_precision": 1.0,
    "mean_total_tokens_per_run": 361.5,
    "estimated_cost": {
      "per_run": {
        "input_cost": 8e-06,
        "output_cost": 0.005382,
        "total_cost": 0.00539,
        "model": "sonnet",
        "currency": "USD"
      },
      "total_runs": 6,
      "total_cost": 0.0323,
      "model": "sonnet",
      "currency": "USD"
    }
  },
  "passed": true
}

# Amazon Bedrock Best Practices Checklist

Organized by pillar. Maps directly to the checks in `SKILL.md` Step 4.

## Security

- [ ] **Guardrails** configured for high-volume workloads (filter hate, insult, sexual, violence, PII; block use-case-relevant denied topics)
- [ ] **Guardrail signals** healthy — `InvocationClientErrors` < 5% of `Invocations`; `InvocationsIntervened` in the 5–15% range on active guardrails
- [ ] **Model invocation logging** configured appropriately for the data-sensitivity profile (do NOT log prompt/output for production PII/PCI/HIPAA content)
- [ ] **Knowledge Base configuration** — chunk size 200–1000 tokens, sound embedding strategy, monitoring, error handling with retry logic
- [ ] **Knowledge Base data source encryption** — customer-managed KMS key when handling sensitive content
- [ ] **VPC configuration for model customization jobs** — training data not reachable from the internet; VPC Flow Logs enabled
- [ ] **IAM fine-grained access control** — agent/alias policies scoped to specific resources; resource `*` justified and documented
- [ ] **Knowledge Base logging** — KB ingestion log delivery configured; CloudTrail + CloudWatch enabled for anomaly detection
- [ ] **Prompt injection** — prompt templates hardened per prompt-engineering best practices
- [ ] **Model access** — scoped to only the essential, legally approved models (least privilege)

## Performance

- [ ] **Latency & throttling** — throttle rate low; retries use exponential backoff with jitter; smaller models and streaming used for latency-sensitive apps
- [ ] Application deployed in the same region as the Bedrock endpoint to minimize network latency
- [ ] `max_tokens` set no higher than needed; lower temperature where deterministic output suffices
- [ ] **Agent performance** — latency-optimized flow eligibility met (single KB, no enabled action groups, no follow-up questions, default orchestration template)
- [ ] **Invoked model versions** — no Legacy or End-of-Life models still receiving production traffic; upgrade plan documented
- [ ] **Data automation** — success rate > 95%, error rate < 5%, throttle rate < 1%
- [ ] **Service tier** aligned to workload (Reserved/Priority for latency-sensitive; Flex for cost-tolerant batch)

## Service Quotas

- [ ] **Model quotas** — RPM/TPM (including CRIS and Global CRIS) utilization < 75%; increases requested proactively
- [ ] **Guardrail service quotas** — ApplyGuardrail requests and text-unit consumption < 75% of quota; Classic vs Standard policy usage reviewed manually when both configured
- [ ] CloudWatch alarms set for high quota-utilization thresholds

## Cost Optimization

- [ ] **Application inference profiles** used for cost allocation / usage tracking via Cost Allocation Tags
- [ ] **Custom model distillation** evaluated for narrow, repetitive, high-volume tasks on premium large models (candidate rule: p99 latency > 3000 ms + input tokens > 1M/month + throttle rate > 5%)
- [ ] **Prompt caching** enabled and optimized (cache offload ≥ 50%) for input-heavy repeated-context workloads
- [ ] **Prompt management** used for versioning and variant testing (cost/quality tradeoffs)
- [ ] **Intelligent prompt routing** used for mixed-complexity workloads within a model family
- [ ] **Provisioned throughput** balanced against on-demand; no idle or soon-to-expire provisioned models on unused workloads
- [ ] **Batch inference** used for high-volume (> 10K/day), latency-tolerant, or scheduled workloads (up to ~50% savings); self-managed batch migrated to managed Batch Inference where models are available
- [ ] **EC2 GPU utilization** (self-managed) — no sustained under-utilization (< 40%), memory pressure (> 90%), bursty usage (CV > 0.5), or un-committed high sustained usage (> 80%)

## Resilience

- [ ] **Cross-Region Inference (CRIS)** adopted where burst resilience or data-residency matters; adoption % high (traffic routed through inference profiles, not direct base-model invocations)

## CloudWatch Alarm Coverage (recommended)

- [ ] Throttling — `InvocationThrottles` alarm per high-value model
- [ ] Latency — `InvocationLatency` p95 threshold alarm
- [ ] Server errors — `InvocationServerErrors` alarm
- [ ] Guardrail interventions — `InvocationsIntervened` trend alarm
- [ ] Quota utilization — alarm approaching RPM/TPM limits

# Amazon Bedrock CloudWatch Metrics Thresholds Reference

Metrics are retrieved via `cloudwatch.ListMetrics` (to discover invoked models) and
`cloudwatch.GetMetricData` over the selected window (default 7 days). Severity reflects
sustained values, not single spikes. `cloudwatch.GetMetricData` allows up to 500
metric-data queries per call — batch and paginate for accounts with many models.

## Bedrock Model Metrics (Namespace: `AWS/Bedrock`, Dimension: `ModelId`)

| Metric | Stat | Normal | Warning | Critical | Finding |
|---|---|---|---|---|---|
| Invocations | Sum | baseline | — | — | Volume; denominator for rate calculations |
| InvocationThrottles | Sum | throttle rate ~0% | > 1% of Invocations | > 5% of Invocations | Add retries w/ backoff+jitter, request quota increase, use CRIS |
| InvocationServerErrors | Sum | ~0% | > 2% of Invocations | > 5% of Invocations | Investigate service-side failures / problematic content |
| InvocationClientErrors | Sum | < 1% | > 5% of Invocations | > 15% of Invocations | Bad or blocked input reaching models; add guardrails |
| InvocationLatency | Average | task-dependent | rising trend | sustained high | Use smaller model / streaming; deploy in-region |
| InvocationLatency | p95 | task-dependent | > 2000 ms | > 3000 ms | Latency-sensitive UX risk; distillation candidate |
| TimeToFirstToken | Average, p95 | low | rising | high | Perceived latency — prefer streaming |
| InputTokenCount | Sum | baseline | high | very high | Input-heavy = prompt-caching / distillation candidate |
| OutputTokenCount | Sum | baseline | — | — | Output volume; cost driver |
| InvocationsIntervened | Sum | 5–15% of Invocations | < 5% (under-config) | > 15% (over-tuned) | Review guardrail policy configuration |
| TextUnitCount | Sum | tracked | approaching quota | at quota | Guardrail consumption maps to billing + quota |
| CacheReadInputTokenCount | Sum | present | writes without reads | — | Prompt-cache reads; used for offload % |
| CacheWriteInputTokenCount | Sum | present | high vs reads | — | Prompt-cache writes; used for offload % |

**Derived rates:**
- Throttle rate = `InvocationThrottles / Invocations`
- Server error rate = `InvocationServerErrors / Invocations`
- Client error rate = `InvocationClientErrors / Invocations`
- Intervened rate = `InvocationsIntervened / Invocations`
- Cache offload % = `CacheReadInputTokenCount / (CacheReadInputTokenCount + CacheWriteInputTokenCount) * 100`
- CRIS adoption % = `Profile InputTokenCount / (Profile InputTokenCount + Base Model InputTokenCount) * 100`

## Prompt Caching Status (derived)

| Status | Condition |
|---|---|
| Not Supported | Model not eligible for prompt caching |
| Not Enabled | Caching supported but no cache activity detected |
| Misconfigured | Cache writes occurring but no cache reads (prompts may not share a common prefix) |
| Underutilized | Cache offload % below 50% |
| Optimized | Cache offload % 50% or higher |

`cachingPotential` from output/input ratio: High (< 0.2), Medium (0.2–0.5), Low (> 0.5).

## Service Quota Utilization

| Quota family | Source | Warning | Critical | Finding |
|---|---|---|---|---|
| Model RPM / CRIS RPM / Global CRIS RPM | `servicequotas.GetServiceQuota` vs observed P95 invocations/min | > 75% | sustained near 100% | Request increase; spread load; adopt CRIS |
| Model TPM / CRIS TPM / Global CRIS TPM | `servicequotas.GetServiceQuota` vs observed P95 tokens/min | > 75% | sustained near 100% | Request increase; optimize token usage |
| Guardrail ApplyGuardrail + policy text units | quota vs text-unit consumption | > 75% | near 100% | Request increase; optimize policies; distribute regionally |

> Guardrail CloudWatch metrics do not distinguish Classic vs Standard policy versions.
> When both are configured, utilization can be inaccurate — review manually.

## EC2 GPU Utilization (Namespace: `CWAgent`, NVIDIA DCGM plugin, 7-day min window)

| Signal | Metric | Threshold | Recommendation |
|---|---|---|---|
| Low GPU utilization | `nvidia_smi_utilization_gpu` avg | < 40% | Migrate training to Trainium2 (~30–50% savings) |
| High GPU memory pressure | `nvidia_smi_memory_util` avg | > 90% | Model/tensor parallelism or larger instance (p5en/p6) |
| Bursty / intermittent usage | CV = StdDev/Mean of GPU util | > 0.5 | EC2 Spot Instances (up to 90%) / Capacity Blocks |
| High sustained utilization | `nvidia_smi_utilization_gpu` avg | > 80% | ML Savings Plans / Reserved Instances (up to 64%) |

If "CloudWatch Agent Installed" = No, `nvidia_smi_*` metrics are absent and GPU signals cannot be evaluated.

## Data Automation

| Metric | Production target | Finding |
|---|---|---|
| Success rate | > 95% | Below target → investigate IAM/input/limits |
| Error rate | < 5% | Above → check permissions and input format |
| Throttle rate | < 1% | Above → client-side rate limiting / quota increase |
| Average latency | baseline | Degradation → reduce input size, use async |

## Model Lifecycle (Invoked Model Versions)

| State | Meaning | Action |
|---|---|---|
| Active | Fully supported | No action |
| Legacy | Retirement announced | MEDIUM — plan upgrade, test new version in non-prod, migrate before EOL |
| End-of-Life | No longer available | HIGH — migrate immediately unless private arrangement with model provider exists |