agentcore-ops-review
Comprehensive operational review procedures for Amazon Bedrock
Install
npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/agentcore-ops-review
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart
git clone https://github.com/aws/tools-for-devops-agent.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.
README
Bedrock AgentCore Operational Review — AWS DevOps Agent Skill
A comprehensive Amazon Bedrock AgentCore operational review skill for AWS DevOps Agent. Conducts best-practices assessments aligned with the Amazon Bedrock AgentCore Developer Guide and the AWS Well-Architected Framework. Generates a shareable report artifact for the review.
What It Does
When activated via Chat, this skill instructs the DevOps Agent to:
- Discover AgentCore resources in the configured account/regions — agent runtimes, endpoints, versions, memories (and their strategies), gateways and gateway targets, browsers, code interpreters, and workload identities.
- Collect CloudWatch metrics from the
AWS/Bedrock-AgentCorenamespace for session counts, invocations, throttles, vCPU-hours, GB-hours, and memory ingestion errors. - Resolve VPC subnet placement to availability zones (via
ec2:DescribeSubnets) to assess runtime AZ fault tolerance. - Analyze against four check areas mapped to the Well-Architected Framework — Runtime Resilience (Reliability), Gateway Health (Reliability), Memory & Knowledge Effectiveness (Performance Efficiency), and Resource Utilization & Operational Hygiene (Operational Excellence) — plus cross-pillar runtime observability signals.
- Generate a shareable report artifact, titled
Bedrock AgentCore Operational Review — <account-id> — <YYYY-MM-DD>.
All data is gathered through native AWS APIs (bedrock-agentcore, cloudwatch, ec2, ce, health). The skill never invokes an agent (InvokeAgentRuntime) and reads no prompts or responses. It makes one data-plane call — bedrock-agentcore:ListMemoryRecords — solely to count long-term memory records; that response can include potentially PII-bearing content, which the skill never reads, stores, or reproduces (see Memory record data handling). It does not depend on any internal tooling.
Agent Types
This skill is intended for the following agent types (selected in the Operator Web App at upload time):
- On-demand — conversational invocation in Chat ("review my AgentCore runtimes", "AgentCore health check").
- Evaluation — proactive operational improvement recommendations.
Select Generic instead if you want the skill available to all agent types.
Prerequisites
1. An AWS DevOps Agent Space with the target AWS account
You need an existing Agent Space with the target AWS account configured as a cloud source.
2. IAM permissions — two modes
The standard AIDevOpsAgentAccessPolicy covers bedrock:* read actions but does NOT include the bedrock-agentcore:* namespace. (Note: bedrock-agentcore-control is the SDK client name, not an IAM prefix — every control-plane action such as ListAgentRuntimes, GetMemory, and ListGateways authorizes under the single service prefix bedrock-agentcore:.) The skill supports two modes:
Runtime-observability-only mode — relies exclusively on CloudWatch metrics against namespace AWS/Bedrock-AgentCore. Requires no bedrock-agentcore: additions. See references/iam-policy-observability-only.json:
cloudwatch:ListMetrics,cloudwatch:GetMetricDatahealth:DescribeEvents,health:DescribeEventDetailssts:GetCallerIdentityce:GetCostAndUsage,ce:GetDimensionValues(region discovery)
Full control-plane mode — adds the read-only bedrock-agentcore: set plus ec2:DescribeSubnets to enable the runtime/gateway/memory/utilization checks. See references/iam-policy-linked-account.json (and references/iam-policy-management-account.json for the payer/management account):
bedrock-agentcore:ListAgentRuntimes,GetAgentRuntime,ListAgentRuntimeEndpoints,ListAgentRuntimeVersionsbedrock-agentcore:ListMemories,GetMemory(control-plane),ListMemoryRecords(data-plane; record count only — see Memory record data handling)bedrock-agentcore:ListGateways,GetGateway,ListGatewayTargets,GetGatewayTargetbedrock-agentcore:ListBrowsers,ListCodeInterpreters,ListWorkloadIdentitiesec2:DescribeSubnets(AC-RUN-2 multi-AZ check)
The skill operates in read-only mode: it never calls Create*, Update*, Delete*, or InvokeAgentRuntime. Its only data-plane call is bedrock-agentcore:ListMemoryRecords, used for a record count and nothing else (see Memory record data handling). If a required permission is missing, the affected check degrades to a visibility limit ("signal unavailable — check skipped") rather than failing the review or producing a false finding.
3. AgentCore workloads with activity (recommended)
Most checks rely on AWS/Bedrock-AgentCore CloudWatch metrics and control-plane inventory, which only exist for accounts actively using AgentCore. Reviewing an account with no AgentCore activity produces "No AgentCore usage detected" rather than false findings.
4. Relationship to agentcore-observability-setup
This skill assesses operational posture from existing telemetry; it does not configure or validate observability wiring. Where telemetry is absent, it reports a visibility limit and defers the configuration gap to the agentcore-observability-setup skill, which owns the observability-wiring finding.
Uploading to AWS DevOps Agent
Reference: Uploading a skill
1. Package the skill
From the skills/ directory in this repo, build the archive from inside the
skill directory so SKILL.md sits at the archive root (nesting it under a
subdirectory causes Failed to get skill resource errors at load time):
cd skills/agentcore-ops-review
zip -qrD ../agentcore-ops-review.zip . \
-x 'README.md' 'CHANGELOG.md' '.skilleval.yaml' 'evals/*'
The resulting agentcore-ops-review.zip contains SKILL.md at the root plus the
references/ files:
SKILL.md # frontmatter + skill instructions (required, at root)
references/
├── pillar-checks.md
├── report-template.md
├── iam-policy.json
├── iam-policy-observability-only.json
├── iam-policy-linked-account.json
└── iam-policy-management-account.json
README.md, CHANGELOG.md, .skilleval.yaml, and evals/ are excluded from the
upload — they are repo/offline-evaluation artifacts, not part of the runtime skill.
Constraints (enforced at upload time):
- Total zip size ≤ 6 MB.
SKILL.mdis required and must includenameanddescriptionfrontmatter.- A
scripts/directory is not allowed — uploads containing scripts are rejected.
2. Upload via the Operator Web App
- Navigate to the Skills page in your Agent Space Operator Web App.
- Click Add skill → Upload skill.
- Drag and drop
agentcore-ops-review.zip(or browse to it). - Select agent types: On-demand and Evaluation (or leave Generic to make it available to all agent types).
- Review the validation results.
- Click Upload.
Usage
In the DevOps Agent Chat, use natural language:
- "Run an AgentCore operational review for all regions."
- "Review my AgentCore runtimes in
us-east-1for resilience." - "Audit my agent memory pipelines for extraction errors."
- "Check my AgentCore gateway health and target redundancy."
- "ORR for our AgentCore workloads."
The agent will:
- Collect all data automatically (no prompts for confirmation).
- Use only AWS read APIs — no agent invocation, no prompts/responses read; the sole data-plane call (
ListMemoryRecords) is used for a record count only, never forcontent. - Generate a report artifact titled
Bedrock AgentCore Operational Review — <account-id> — <YYYY-MM-DD>.
Skill Contents
agentcore-ops-review/
├── SKILL.md # main skill instructions (with frontmatter)
├── README.md # this file
├── CHANGELOG.md # version history
├── references/
│ ├── pillar-checks.md # detailed check definitions, rules, thresholds
│ ├── report-template.md # artifact report structure
│ ├── iam-policy.json # policy index / notes
│ ├── iam-policy-observability-only.json
│ ├── iam-policy-linked-account.json
│ └── iam-policy-management-account.json
└── evals/ # evaluation data (not included in upload zip)
Check Areas Covered
| Pillar (WA) | Check Area | Rule IDs | Reference |
|---|---|---|---|
| Reliability | Runtime Resilience & Production Readiness | AC-RUN-1..4 | AgentCore runtime |
| Reliability | Gateway Health & Target Redundancy | GW-01 | AgentCore gateway |
| Performance Efficiency | Memory & Knowledge Effectiveness | AC-MEM-1..4 | AgentCore memory |
| Operational Excellence | Resource Utilization & Operational Hygiene | AC-UTIL-1..3 | AgentCore observability |
| Cross-pillar | Runtime Observability (session count, invocations, throttles, vCPU/GB-hours) | — | AgentCore observability |
Severity Definitions
| Severity | Definition | SLA |
|---|---|---|
| CRITICAL | Immediate risk to availability, security, or data integrity | 24–48 hours |
| HIGH | Significant gap that could lead to incidents | 1 week |
| MEDIUM | Notable improvement opportunity | 30 days |
| LOW | Minor optimization or hardening | When convenient |
| INFO | Observation, no action required | N/A |
Memory record data handling
The skill's only data-plane call is bedrock-agentcore:ListMemoryRecords, used to count records for long-term memories (check AC-MEM-2). Each MemoryRecordSummary in that response includes a required content field — the extracted facts, preferences, and summaries a long-term memory has stored, which can contain end-user PII.
This skill uses only the record count. It does not read, parse, log, store, transform, transmit, or reproduce the content field anywhere — not in findings, artifacts, recommendations, or logs.
If you prefer the skill make zero data-plane calls, omit bedrock-agentcore:ListMemoryRecords from the IAM policy. AC-MEM-2 (empty/near-empty long-term memory) then degrades to a documented visibility limit, while all other memory checks — which rely on CloudWatch ingestion metrics (Errors, Invocations, NumberOfMemoryRecords) — continue to work.
Non-production disclaimer
⚠️ This skill is sample code, not intended for production use without additional review and testing. Users should validate it in a non-production environment first. It performs read-only operational analysis and makes no changes to your AWS resources, but you are responsible for reviewing the IAM permissions you grant and the findings it produces before acting on them.
Skill manifest
Bedrock AgentCore Operational Review
Use this skill when performing an operational review of Amazon Bedrock AgentCore resources, investigating agent runtime failures, auditing memory/knowledge pipeline health, or reviewing gateway resilience and resource hygiene.
Overview
This skill provides procedures for a review of AgentCore resources across four check areas mapped to the AWS Well-Architected Framework:
- Runtime Resilience (Reliability) — Failed/stuck runtimes and endpoints, single-AZ VPC placement, endpoint version drift, DEFAULT-endpoint-only deployments
- Gateway Health (Reliability) — Gateway status, target redundancy and health, policy engine attachment, target synchronization freshness
- Memory & Knowledge Effectiveness (Performance Efficiency) — Extraction pipeline errors, empty long-term memory, high ingestion error rate, provisioned-but-never-populated memories
- Resource Utilization & Operational Hygiene (Operational Excellence) — Idle resources, consolidation opportunities, low overall utilization
- Runtime Observability (cross-pillar) — Session counts, invocations, throttles, vCPU-hours, GB-hours per runtime
This is a READ-ONLY review. No modifications are made to any resource, and it never invokes an agent (InvokeAgentRuntime), reads no prompts or responses, and makes no other data-plane calls. One exception: for long-term memories the skill calls bedrock-agentcore:ListMemoryRecords (a data-plane API) solely to obtain a record count. That response can include a content field carrying extracted, potentially PII-bearing memory content; the skill uses only the count and never parses, logs, stores, or reproduces content. See "Memory record data handling" below.
Data Source Boundaries (IMPORTANT — IAM footprint)
The standard AIDevOpsAgentAccessPolicy covers bedrock:* read actions but does NOT include the bedrock-agentcore:* namespace. (Note: bedrock-agentcore-control is the SDK client name, not an IAM prefix — all control-plane actions such as ListAgentRuntimes, GetMemory, and ListGateways authorize under the single service prefix bedrock-agentcore:.) Two modes:
- Runtime-observability-only mode — relies exclusively on CloudWatch metrics (
cloudwatch:GetMetricData,cloudwatch:ListMetrics) against namespaceAWS/Bedrock-AgentCore. Requires nobedrock-agentcore:additions. When an account isn't using AgentCore, the namespace is simply empty ("no activity detected") — this is not a failure. - Full control-plane mode — adds read-only
bedrock-agentcore:control-plane List/Get actions andec2:DescribeSubnetsto enable the runtime/gateway/memory/utilization checks. It also uses one data-plane action,bedrock-agentcore:ListMemoryRecords, for the long-term-memory record count only (see "Memory record data handling" below). Seereferences/iam-policy-linked-account.json.
If a required permission is missing, the affected check degrades to a visibility limit (reported as "signal unavailable / check skipped") rather than failing the review or producing a false finding.
Memory record data handling: bedrock-agentcore:ListMemoryRecords is a data-plane API whose MemoryRecordSummary entries include a required content field — the extracted facts/preferences/summaries a long-term memory has stored, which can contain end-user PII. This skill calls it only to count records for AC-MEM-2 and never reads, parses, logs, stores, transforms, or reproduces the content field. If you prefer zero data-plane access, omit this action from the IAM policy: AC-MEM-2 then degrades to a visibility limit while all other memory checks (which use CloudWatch ingestion metrics) continue to work.
Seam with agentcore-observability-setup: This skill assesses operational posture from existing telemetry; it does not configure or validate observability wiring. Where telemetry is absent, this skill reports a visibility limit and defers the configuration gap to agentcore-observability-setup — that skill owns the observability-wiring finding (Transaction Search, OTEL/ADOT, log delivery, X-Ray resource policy), while this skill reports only the posture consequence. A customer running both should not see two overlapping findings on the same resource.
Step 1: Discover Account and Regions
- Call
sts:GetCallerIdentityto determine the account. - Discover AgentCore-active regions:
- Query Cost Explorer:
ce:GetDimensionValues(dimension SERVICE, last 30 days), match "Amazon Bedrock AgentCore". - Then
ce:GetCostAndUsagefor that service grouped by REGION; select top regions by UnblendedCost. - If no AgentCore spend is found, fall back to probing
cloudwatch:ListMetrics(namespaceAWS/Bedrock-AgentCore) in the customer's primary regions to detect activity.
- Query Cost Explorer:
- If no AgentCore activity is detected in any region, stop: "No AgentCore usage detected in the last 30 days."
Step 2: Runtime Resilience Pillar (Reliability)
Discover runtimes and enrich each with endpoints and versions:
bedrock-agentcore:ListAgentRuntimes→GetAgentRuntime(status,failureReason,networkConfiguration)bedrock-agentcore:ListAgentRuntimeEndpoints(status,liveVersion,targetVersion, name)bedrock-agentcore:ListAgentRuntimeVersions(to compute latest version + drift)ec2:DescribeSubnets(resolve VPC subnet IDs to AZs — endpoints carry NO AZ data)
| Check | Rule | Severity |
|---|---|---|
| AC-RUN-1 | Runtime/endpoint in CREATE_FAILED/UPDATE_FAILED, or stuck in CREATING/UPDATING > 1 hour |
Critical |
| AC-RUN-2 | VPC-mode runtime whose subnets all resolve to a single AZ (PUBLIC mode exempt) | High |
| AC-RUN-3 | Live endpoint ≥ 3 versions behind latest runtime version | Medium |
| AC-RUN-4 | Actively-updated runtime (2+ versions) served only by the auto-updating DEFAULT endpoint | Low |
Key fact: Endpoints are a versioning/traffic-routing construct, not a redundancy mechanism. AZ fault tolerance is derived exclusively from the runtime's VPC subnet configuration. If ec2:DescribeSubnets is unavailable, AC-RUN-2 degrades to "AZ distribution unknown".
Step 3: Gateway Health (Reliability)
Discover gateways and their targets:
bedrock-agentcore:ListGateways→GetGateway(status, protocolType, authorizerType,policyEngineConfiguration)bedrock-agentcore:ListGatewayTargets→GetGatewayTarget(status,targetConfiguration,lastSynchronizedAt, credential providers)
Target type derived from targetConfiguration.mcp: Lambda, MCP Server, API Gateway, OpenAPI Schema, Smithy Model.
| Risk Level | Condition |
|---|---|
| Critical | Gateway not READY, zero targets, OR all targets unhealthy |
| Warning | Single target (no redundancy), no policy engine attached, some unhealthy targets, OR stale sync (> 7 days) |
| Healthy | Multiple READY targets, policy engine attached, recent synchronization |
Unhealthy target statuses: FAILED, UPDATE_UNSUCCESSFUL, SYNCHRONIZE_UNSUCCESSFUL. Stale sync threshold: lastSynchronizedAt older than 7 days.
Step 4: Memory & Knowledge Effectiveness (Performance Efficiency)
Discover memories and their strategies, then query CloudWatch ingestion metrics:
bedrock-agentcore:ListMemories→GetMemory(status, createdAt, configured strategies)bedrock-agentcore:ListMemoryRecords— data-plane call, count only (long-term-strategy memories). Read the returned record count; do not read thecontentfield (see "Memory record data handling")cloudwatch:GetMetricData(namespaceAWS/Bedrock-AgentCore): per-memoryInvocations/Errorsfor theIngestionoperation, andInvocationsfor theCreateEventoperation (30-day window)
All rules are strategy-aware — read GetMemory strategies first. Record-count rules apply ONLY to memories with a long-term strategy; short-term-only memories are never flagged as empty.
| Check | Rule | Severity |
|---|---|---|
| AC-MEM-1 | Long-term memory with CloudWatch ingestion Errors > 0 |
High |
| AC-MEM-2 | Long-term memory with < 10 records (escalates to High if 0 records AND > 7 days old) | Medium→High |
| AC-MEM-3 | Ingestion error rate > 20% (Errors / Invocations) |
High |
| AC-MEM-4 | Provisioned but never populated: zero CreateEvent activity AND > 7 days old |
Medium |
Safety rule: CreateEvent activity is queried for EVERY memory (short-term memories receive events too). When the CloudWatch signal is unreadable, event count is None (not 0), so AC-MEM-4 is skipped rather than firing a false "never populated" finding.
Step 5: Resource Utilization & Operational Hygiene (Operational Excellence)
Enumerate all provisionable resource types and collect a 30-day activity signal:
- Inventory:
ListAgentRuntimes,ListMemories,ListGateways,ListBrowsers,ListCodeInterpreters,ListWorkloadIdentities - Activity:
cloudwatch:GetMetricData(Invocations,AWS/Bedrock-AgentCore) keyed by per-type dimension (AgentRuntimeId,MemoryId,GatewayId)
| Check | Rule | Severity |
|---|---|---|
| AC-UTIL-1 | Idle resource: zero activity over 30-day window, > 7 days old | Medium |
| AC-UTIL-2 | Consolidation: duplicate configs in a region, OR regions holding resources with < 5% of total activity (3+ regions) | Informational→Medium |
| AC-UTIL-3 | Overall utilization < 60% (active / assessable, excluding recently-created) | Informational |
For the AC-UTIL-1 idle signal, prefer the real-time ActiveSessionCount gauge (a currently-running-sessions gauge, filterable by the Service dimension) where available; fall back to the cumulative SessionCount / Invocations counters over the 30-day window when ActiveSessionCount is not present.
Framing: AgentCore runtime billing is consumption-based — idle time is free. Frame findings as operational hygiene and security surface (unmanaged IAM roles, stale config), NOT wasted spend. Exception: memories holding stored long-term records DO accrue storage cost — call this out in the finding.
Classification safety: A resource is only classified Idle when its activity signal is complete for the full window. Types with no queryable per-resource metric (Browser, CodeInterpreter, WorkloadIdentity), and every resource when the CloudWatch call fails, are classified Active (partial) — never Idle. Resources younger than 7 days are RecentlyCreated and excluded from both idle flagging and the utilization ratio.
Step 6: Runtime Observability (cross-pillar signals)
For each runtime, collect CloudWatch metrics from namespace AWS/Bedrock-AgentCore:
| Metric | Dimensions | Period | Statistic |
|---|---|---|---|
| CPUUsed-vCPUHours | Resource (ARN), Service=AgentCore.Runtime, Name |
3600s | Sum |
| MemoryUsed-GBHours | Resource (ARN), Service=AgentCore.Runtime, Name |
3600s | Sum |
| SessionCount | Resource (ARN), Operation=InvokeAgentRuntime, Name |
300s | Sum |
| Invocations | Resource (ARN), Operation=InvokeAgentRuntime, Name |
300s | Sum |
| Throttles | Resource (ARN), Operation=InvokeAgentRuntime, Name |
300s | Sum |
SessionCount is a cumulative counter of new sessions per period. Where a real-time view is needed, ActiveSessionCount is a gauge of currently-running sessions (filterable by the Service dimension).
Discover resources first via cloudwatch:ListMetrics (namespace AWS/Bedrock-AgentCore), extract unique Resource ARN + Name dimensions, then batch GetMetricData per resource.
Derived signals:
- Throttle rate =
Throttles / Invocations × 100 - Health status = "Throttling Detected" when
Throttles > 0, else "Healthy" - Only report runtimes with any activity (CPU-hours, GB-hours, invocations, or session count > 0).
Step 7: Additional Context
- Health Events:
health:DescribeEventsfiltered for Bedrock/AgentCore service, last 14 days. - Documentation: Cross-reference findings against AgentCore best-practices docs for remediation links.
Step 8: Produce Report and Recommendations
Produce an artifact with the structure defined in references/report-template.md.
For each finding with severity Warning/Medium or higher, create a recommendation with:
- Title:
[Check ID] [Check name] — [Resource identifier] - Summary: Current state, expected state, business/operational impact, remediation steps, and a TAM conversation starter.
Related Skills / Boundaries
Boundaries are deliberate and non-overlapping:
agentcore-observability-setup— sets up/validates observability wiring. This skill consumes that telemetry for posture assessment and defers observability-configuration gaps to it (see the seam described in Data Source Boundaries).agentcore-runtime-diag(queued submission — referenced for scope, not yet published) — reactive triage of a single failing invocation. This skill is proactive, estate-wide posture; it does not diagnose individual call failures or evaluate authorizer/credential correctness. Gateway checks here are redundancy/staleness/health only, not auth evaluation.aiml-access-diagnostics(published) — generic AI/ML AccessDenied chain viaiam:SimulatePrincipalPolicy. Not overlapping — this skill makes no IAM-simulation calls.bedrock-adoption-readiness(published) — foundation-model workload readiness (IAM governance, ZDR, quotas, observability). Different resource surface — this skill reviews the AgentCore agent runtime / memory / gateway layer, not foundation-model inference.
Error Handling
| Error | Action |
|---|---|
AccessDenied on bedrock-agentcore:* |
Log as visibility limit "signal unavailable — check skipped", continue in observability-only mode |
AccessDenied on ec2:DescribeSubnets |
AC-RUN-2 degrades to "AZ distribution unknown" |
| CloudWatch returns no data | Report "no activity detected" for that resource; classify utilization as Active (partial), never Idle |
| No AgentCore usage anywhere | Stop early: "No AgentCore usage detected" |
| Throttled by AWS API | Retry with exponential backoff (3 attempts) |
| Check raises an exception | Isolate the failure — degrade that check to a visibility limit and continue the others |
Important Notes
- ALL API calls with pagination (NextToken) MUST be paginated to completion.
- Cost Explorer queries run against us-east-1 (global endpoint).
- Findings are only produced when the underlying signal is complete — incomplete signals become visibility limits, never false positives.
- Batch CloudWatch
GetMetricDatarequests where possible. - This is a READ-ONLY review — no modifications, and no data-plane calls other than
ListMemoryRecordsfor record count (count only;contentis never read).
Files (tools-for-devops-agent)
-
evals
-
files
-
agentcore-context.json 342 B
{ "resources": [ { "type": "runtime", "name": "customer-support-agent", "region": "us-east-1", "account": "$accountId" }, { "type": "memory", "name": "support-longterm-memory", "region": "us-east-1", "account": "$accountId" }, { "type": "gateway", "name": "tools-gateway", "region": "us-east-1", "account": "$accountId" } ] }
-
-
evals.json 3.7 KB
[ { "id": "agentcore-review-smoke-test", "prompt": "Read agentcore-context.json. List each resource's name, type (runtime, memory, or gateway), region, and account. No analysis needed.", "expected_output": "Lists every resource from files/agentcore-context.json with its name, type, region, and account exactly as defined in the file.", "files": ["files/agentcore-context.json"], "assertions": [ "matches regex /[a-z]{2}-[a-z]+-\\d/", "contains 'runtime' or contains 'memory' or contains 'gateway'", "contains 'region'", "contains 'account'" ] }, { "id": "agentcore-review-artifact-naming", "prompt": "Read agentcore-context.json. Identify the FIRST resource listed and its region, then state the report artifact title you would generate for this review (use today's date in YYYY-MM-DD form). Do not run the audit.", "expected_output": "Identifies the first resource and region from files/agentcore-context.json and proposes an artifact title of the form 'Bedrock AgentCore Operational Review — <account-id> — <YYYY-MM-DD>'.", "files": ["files/agentcore-context.json"], "assertions": [ "contains 'Bedrock AgentCore Operational Review' or contains 'AgentCore Operational Review'", "matches regex /\\d{4}-\\d{2}-\\d{2}/", "matches regex /[a-z]{2}-[a-z]+-\\d/" ] }, { "id": "agentcore-review-data-source-priority", "prompt": "According to the skill, which AWS APIs does the AgentCore operational review use to collect data? Name at least three specific API calls or namespaces the skill mentions. Does the skill invoke any agent, and how does it handle memory record content? No account access required.", "expected_output": "States the skill uses the bedrock-agentcore, CloudWatch, and EC2 APIs; names specific calls such as bedrock-agentcore:ListAgentRuntimes, bedrock-agentcore:ListMemories, cloudwatch:GetMetricData, cloudwatch:ListMetrics, or ec2:DescribeSubnets. Confirms it never invokes an agent (InvokeAgentRuntime) and that its one data-plane call, ListMemoryRecords, is used only for a record count — it never reads, stores, or reproduces memory record content.", "files": [], "assertions": [ "contains 'bedrock-agentcore' or contains 'AgentCore'", "contains 'CloudWatch' or contains 'cloudwatch'", "contains 'ListAgentRuntimes' or contains 'ListMemories' or contains 'GetMetricData' or contains 'ListMetrics' or contains 'DescribeSubnets'", "contains 'count' or contains 'ListMemoryRecords'", "contains 'not' or contains 'never' or contains 'no '" ] }, { "id": "agentcore-review-severity-definitions", "prompt": "List the five severity levels the skill uses for findings, in order from most to least urgent. No account access required.", "expected_output": "Lists CRITICAL, HIGH, MEDIUM, LOW, INFO in that order.", "files": [], "assertions": [ "contains 'CRITICAL'", "contains 'HIGH'", "contains 'MEDIUM'", "contains 'LOW'", "contains 'INFO'" ] }, { "id": "agentcore-review-pillars", "prompt": "List the check areas the skill evaluates during an AgentCore operational review. No account access required.", "expected_output": "Mentions Runtime Resilience, Gateway Health, Memory & Knowledge Effectiveness, and Resource Utilization & Operational Hygiene (plus cross-pillar Runtime Observability).", "files": [], "assertions": [ "contains 'Runtime Resilience' or contains 'runtime resilience'", "contains 'Gateway' or contains 'gateway'", "contains 'Memory' or contains 'memory'", "contains 'Utilization' or contains 'utilization'", "contains 'Observability' or contains 'observability'" ] } ] -
eval_queries.json 797 B
[ {"query": "Which skill would help me run an AgentCore operational review? Just name it; do not run it.", "should_trigger": true}, {"query": "Is there a skill available for auditing Amazon Bedrock AgentCore runtimes and memory pipelines against best practices? Answer yes or no with the skill name; do not execute it.", "should_trigger": true}, {"query": "Name the skill that covers AgentCore runtime resilience, gateway health, and memory effectiveness reviews. Do not run any audit.", "should_trigger": true}, {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false}, {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false}, {"query": "Create a CloudFormation template for an S3 bucket", "should_trigger": false} ]
-
-
references
-
iam-policy-linked-account.json 1.8 KB
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AgentCoreOpsReviewControlPlane", "Effect": "Allow", "Action": [ "bedrock-agentcore:ListAgentRuntimes", "bedrock-agentcore:GetAgentRuntime", "bedrock-agentcore:ListAgentRuntimeEndpoints", "bedrock-agentcore:ListAgentRuntimeVersions", "bedrock-agentcore:ListMemories", "bedrock-agentcore:GetMemory", "bedrock-agentcore:ListGateways", "bedrock-agentcore:GetGateway", "bedrock-agentcore:ListGatewayTargets", "bedrock-agentcore:GetGatewayTarget", "bedrock-agentcore:ListBrowsers", "bedrock-agentcore:ListCodeInterpreters", "bedrock-agentcore:ListWorkloadIdentities" ], "Resource": "*" }, { "_comment": "Data-plane: counts long-term memory records for AC-MEM-2. Response can include PII-bearing memory content; the skill reads the count only and never parses/stores/reproduces content. Omit this statement for zero data-plane access (AC-MEM-2 degrades to a visibility limit; other memory checks use CloudWatch metrics).", "Sid": "AgentCoreOpsReviewMemoryRecordCount", "Effect": "Allow", "Action": [ "bedrock-agentcore:ListMemoryRecords" ], "Resource": "*" }, { "Sid": "AgentCoreOpsReviewObservability", "Effect": "Allow", "Action": [ "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "ec2:DescribeSubnets", "health:DescribeEvents", "health:DescribeEventDetails", "sts:GetCallerIdentity" ], "Resource": "*" }, { "Sid": "CostExplorerAccountLevel", "Effect": "Allow", "Action": [ "ce:GetCostAndUsage", "ce:GetDimensionValues" ], "Resource": "*" } ] } -
iam-policy-management-account.json 1.8 KB
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AgentCoreOpsReviewControlPlane", "Effect": "Allow", "Action": [ "bedrock-agentcore:ListAgentRuntimes", "bedrock-agentcore:GetAgentRuntime", "bedrock-agentcore:ListAgentRuntimeEndpoints", "bedrock-agentcore:ListAgentRuntimeVersions", "bedrock-agentcore:ListMemories", "bedrock-agentcore:GetMemory", "bedrock-agentcore:ListGateways", "bedrock-agentcore:GetGateway", "bedrock-agentcore:ListGatewayTargets", "bedrock-agentcore:GetGatewayTarget", "bedrock-agentcore:ListBrowsers", "bedrock-agentcore:ListCodeInterpreters", "bedrock-agentcore:ListWorkloadIdentities" ], "Resource": "*" }, { "_comment": "Data-plane: counts long-term memory records for AC-MEM-2. Response can include PII-bearing memory content; the skill reads the count only and never parses/stores/reproduces content. Omit this statement for zero data-plane access (AC-MEM-2 degrades to a visibility limit; other memory checks use CloudWatch metrics).", "Sid": "AgentCoreOpsReviewMemoryRecordCount", "Effect": "Allow", "Action": [ "bedrock-agentcore:ListMemoryRecords" ], "Resource": "*" }, { "Sid": "AgentCoreOpsReviewObservability", "Effect": "Allow", "Action": [ "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "ec2:DescribeSubnets", "health:DescribeEvents", "health:DescribeEventDetails", "sts:GetCallerIdentity" ], "Resource": "*" }, { "Sid": "CostExplorerOrgWide", "Effect": "Allow", "Action": [ "ce:GetCostAndUsage", "ce:GetDimensionValues" ], "Resource": "*" } ] } -
iam-policy-observability-only.json 545 B
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AgentCoreObservabilityOnly", "Effect": "Allow", "Action": [ "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "health:DescribeEvents", "health:DescribeEventDetails", "sts:GetCallerIdentity" ], "Resource": "*" }, { "Sid": "CostExplorerAccountLevel", "Effect": "Allow", "Action": [ "ce:GetCostAndUsage", "ce:GetDimensionValues" ], "Resource": "*" } ] } -
iam-policy.json 642 B
{ "Description": "IAM policy for the AgentCore Ops Review skill. Use iam-policy-management-account.json for the management/payer account and iam-policy-linked-account.json for each linked account. NOTE: the bedrock-agentcore namespace is NOT part of the standard AIDevOpsAgentAccessPolicy and must be added explicitly for full control-plane mode. (bedrock-agentcore-control is the SDK client name, not an IAM prefix — all control-plane actions authorize under bedrock-agentcore:.) Observability-only mode needs just the cloudwatch actions.", "See": [ "iam-policy-management-account.json", "iam-policy-linked-account.json" ] } -
pillar-checks.md 11.8 KB
# AgentCore Check Definitions — Detailed Reference All checks are read-only. They derive from control-plane APIs and CloudWatch metrics, plus a single data-plane call — `bedrock-agentcore:ListMemoryRecords` — used **only** to count long-term memory records (its `content` field is never read; see "Memory record data handling" in `SKILL.md`). No agent is invoked (`InvokeAgentRuntime`), and no prompts or responses are read. A finding is produced ONLY when the underlying signal is complete; incomplete signals become visibility limits, never false positives. APIs authorize under the single IAM service prefix `bedrock-agentcore:` (this covers control-plane actions such as `ListAgentRuntimes`/`GetGateway` and the data-plane `ListMemoryRecords`). `bedrock-agentcore-control` is the SDK client name, not an IAM prefix. This namespace is NOT part of the standard `AIDevOpsAgentAccessPolicy`. See `iam-policy-linked-account.json`. --- ## Runtime Resilience Pillar — Reliability (4 checks) Data collection: - `bedrock-agentcore:ListAgentRuntimes` → `GetAgentRuntime` (status, `failureReason`, `networkConfiguration`, timestamps) - `bedrock-agentcore:ListAgentRuntimeEndpoints` (status, `liveVersion`, `targetVersion`, name) - `bedrock-agentcore:ListAgentRuntimeVersions` (latest version + count) - `ec2:DescribeSubnets` (map VPC subnet IDs → AZ) ### AC-RUN-1: Runtime or Endpoint in Failed State - **Source**: runtime `status`; endpoint `status`; `lastUpdatedAt`/`createdAt` - **Failed states**: `CREATE_FAILED`, `UPDATE_FAILED`, `DELETE_FAILED`, `FAILED` - **Transitional states**: `CREATING`, `UPDATING`, `DELETING` - **Critical if**: runtime or any endpoint is in a failed state; OR stuck in a transitional state for > 1 hour - **Evidence**: include `failureReason` when present - **Remediation**: Investigate the failure reason, fix the root cause, redeploy or roll back to the last healthy version. ### AC-RUN-2: VPC Runtime Without Multi-AZ Subnets - **Source**: `networkConfiguration.networkMode`; `networkModeConfig.subnets`; `ec2:DescribeSubnets` → `availabilityZone` - **Applies to**: runtimes with `networkMode == VPC` and at least one subnet - **N/A if**: `networkMode == PUBLIC` (AWS manages availability) - **High if**: all resolved subnets are in a single AZ - **Visibility limit if**: subnet details unreadable, or a subnet has no resolvable AZ - **Key fact**: endpoints carry NO AZ data — AZ fault tolerance comes exclusively from VPC subnet placement - **Remediation**: Add subnets in ≥ 1 additional AZ to the runtime's network configuration. ### AC-RUN-3: Endpoint Version Drift - **Source**: endpoint `liveVersion` vs latest runtime version (from `ListAgentRuntimeVersions`) - **Calculation**: `drift = latestVersion - liveVersion` - **Medium if**: any live endpoint is `drift >= 3` - **N/A if**: latest version cannot be determined (versions call incomplete) - **Remediation**: Adopt a promotion cadence — validate new versions on a staging endpoint, promote production endpoints regularly to pick up fixes. ### AC-RUN-4: DEFAULT-Endpoint-Only Deployment - **Source**: endpoint names; runtime `versionCount` - **Low if**: the runtime's only endpoint is `DEFAULT` (auto-tracks latest) AND the runtime has 2+ versions (i.e. actively updated) - **Rationale**: no environment-promotion strategy — every published version goes live to consumers immediately - **Remediation**: Create pinned endpoints per environment (dev/staging/prod) so new versions are validated before serving production traffic. --- ## Gateway Health — Reliability (1 check, multi-signal) Data collection: - `bedrock-agentcore:ListGateways` → `GetGateway` (status, protocolType, authorizerType, `policyEngineConfiguration`) - `bedrock-agentcore:ListGatewayTargets` → `GetGatewayTarget` (status, `targetConfiguration`, `lastSynchronizedAt`, `credentialProviderConfigurations`) Target type derived from `targetConfiguration.mcp`: `lambda` → LAMBDA, `mcpServer` → MCP_SERVER, `apiGateway` → API_GATEWAY, `openApiSchema` → OPEN_API_SCHEMA, `smithyModel` → SMITHY_MODEL. ### GW-01: Gateway Health & Target Redundancy - **Unhealthy target statuses**: `FAILED`, `UPDATE_UNSUCCESSFUL`, `SYNCHRONIZE_UNSUCCESSFUL` - **Healthy target status**: `READY` - **Stale sync**: `lastSynchronizedAt` older than 7 days - **Policy engine attached**: `policyEngineConfiguration.arn` is present Risk levels: - **Critical**: gateway status != `READY`, OR total targets == 0, OR healthy targets == 0 - **Warning**: any unhealthy target, OR single target (no redundancy), OR no policy engine attached, OR stale sync detected - **Healthy**: multiple READY targets, policy engine attached, recent synchronization Best-practice signals: - ≥ 2 targets per gateway to avoid a single point of failure - Policy engine attached for authorization/rate limiting - Target type diversity (e.g. Lambda + MCP Server) for failover - Short-lived credentials with rotation on credential providers **Remediation**: add a second target (ideally a different target type), attach a policy engine, and investigate stale-sync targets (broken connectivity or expired credentials). --- ## Memory & Knowledge Effectiveness — Performance Efficiency (4 checks) **Strategy-aware:** read `GetMemory` strategies FIRST. Record-count rules (AC-MEM-2) apply only to memories with ≥ 1 long-term strategy. Short-term-only memories are exempt and reported as a "ShortTermOnly" observation, never a finding. Data collection: - `bedrock-agentcore:ListMemories` → `GetMemory` (status, createdAt, strategies) - `bedrock-agentcore:ListMemoryRecords` — **data-plane, count only** (long-term-strategy memories; read the record count, never the `content` field) - `cloudwatch:GetMetricData` (namespace `AWS/Bedrock-AgentCore`, 30-day window): - `Ingestion` operation: `Invocations`, `Errors` (long-term memories) - `CreateEvent` operation: `Invocations` (ALL memories — short-term memories receive events too) Thresholds: `ERROR_RATE_THRESHOLD = 0.20`, `NEAR_EMPTY_THRESHOLD = 10 records`, `MIN_AGE_DAYS = 7`. ### AC-MEM-1: Extraction / Consolidation Errors - **Applies to**: memories with ≥ 1 long-term strategy - **High if**: ingestion `Errors > 0` over the lookback window - **Confidence**: High if metrics complete, else Medium - **Remediation**: Check ingestion application logs (if log delivery enabled). Common causes: incompatible event format, extraction-model permission errors, strategy misconfiguration. ### AC-MEM-2: Empty or Near-Empty Long-Term Memory - **Applies to**: memories with ≥ 1 long-term strategy - **Medium if**: record count < 10 - **Escalates to High if**: record count == 0 AND age > 7 days - **Short-term-only**: emit `ShortTermOnly` observation instead (zero long-term records is expected) - **Remediation**: Verify events are flowing and extraction is producing records. Check ingestion metrics and logs. ### AC-MEM-3: High Extraction Error Rate - **Applies to**: long-term memories with ingestion `Invocations > 0` - **Calculation**: `error_rate = Errors / Invocations` - **High if**: `error_rate > 0.20` - **Remediation**: A significant portion of knowledge is being lost. Review extraction logs and the event shapes causing failures. ### AC-MEM-4: Provisioned but Never Populated - **Applies to**: ALL memories (evaluates event ingestion, not records) - **Medium if**: `CreateEvent` invocations == 0 AND age > 7 days - **Skipped if**: the `CreateEvent` metric was unreadable — event count is set to `None` (not 0), so an unreadable signal never reports "never populated" - **Remediation**: Confirm the memory is still intended for use. If abandoned, decommission it to reduce operational surface. --- ## Resource Utilization & Operational Hygiene — Operational Excellence (3 checks) **Framing:** AgentCore runtime billing is consumption-based (idle time is free). Findings are framed as operational hygiene and security surface, NOT wasted spend. Exception: memories holding stored long-term records DO accrue standing storage cost — noted per-resource via `hasStandingCost`. Data collection: - Inventory: `ListAgentRuntimes`, `ListMemories`, `ListGateways`, `ListBrowsers`, `ListCodeInterpreters`, `ListWorkloadIdentities` - Activity: `cloudwatch:GetMetricData` (`Invocations`, namespace `AWS/Bedrock-AgentCore`, 30-day window) keyed by per-type dimension: - Runtime → `AgentRuntimeId`, Memory → `MemoryId`, Gateway → `GatewayId` - Browser / CodeInterpreter / WorkloadIdentity → no queryable per-resource metric (activity signal incomplete) For the runtime idle signal, prefer the real-time `ActiveSessionCount` gauge where available; fall back to the cumulative `SessionCount` / `Invocations` counters over the 30-day window otherwise. Thresholds: `UTILIZATION_THRESHOLD = 0.60`, `SPRAWL_REGION_ACTIVITY_THRESHOLD = 0.05`, `MIN_AGE_DAYS = 7`, `RECENTLY_CREATED_DAYS = 7`. Classification (per resource): - **Active** — has activity, complete signal - **Idle** — zero activity, complete signal - **Active (partial)** — signal incomplete (no queryable metric, or CloudWatch call failed) — NEVER classified Idle - **RecentlyCreated** — age < 7 days; excluded from idle flagging AND from the utilization ratio ### AC-UTIL-1: Idle Provisioned Resources - **Medium if**: zero activity over the 30-day window AND age ≥ 7 days AND signal complete (status == Idle) - **Escalate messaging** (not severity) where standing cost exists (e.g. idle memory holding long-term records) - **Remediation**: Review whether the resource is still needed. Idle resources carry IAM roles and configuration that expand operational and security surface. ### AC-UTIL-2: Consolidation Opportunities - **Duplicates**: within a region, group by (type, region, config hash); flag clusters of 2+ near-identical resources - **Regional residue**: when resources span 3+ regions, flag regions with < 5% of total activity - **Informational** by default; **Medium** when broad (3+ duplicate clusters, or multiple issues) - **Remediation**: Consolidating duplicates and retiring low-activity regions simplifies architecture and reduces operational overhead. ### AC-UTIL-3: Low Overall Utilization - **Calculation**: `active / assessable`, where assessable excludes RecentlyCreated resources - **Informational if**: ratio < 0.60 - **Reported summary**: total, assessable, recentlyCreated, active, idle, ratio - **Remediation**: Review the estate for resources that can be retired. An accurate inventory makes incidents and audits easier. --- ## Runtime Observability — cross-pillar signals Data collection (namespace `AWS/Bedrock-AgentCore`): 1. `cloudwatch:ListMetrics` to discover resources — extract unique `Resource` (ARN) + `Name` dimensions 2. `cloudwatch:GetMetricData` per resource: | Metric | Dimensions | Period | Statistic | |--------|-----------|--------|-----------| | CPUUsed-vCPUHours | Resource, Service=`AgentCore.Runtime`, Name | 3600s | Sum | | MemoryUsed-GBHours | Resource, Service=`AgentCore.Runtime`, Name | 3600s | Sum | | SessionCount | Resource, Operation=`InvokeAgentRuntime`, Name | 300s | Sum | | Invocations | Resource, Operation=`InvokeAgentRuntime`, Name | 300s | Sum | | Throttles | Resource, Operation=`InvokeAgentRuntime`, Name | 300s | Sum | `SessionCount` is a cumulative counter of new sessions per period. `ActiveSessionCount` (added June 2026) is a real-time gauge of currently-running sessions, filterable by the `Service` dimension — prefer it where a live "is anything running" read is needed. Derived: - **Throttle rate** = `Throttles / Invocations × 100` (`0.00%` when no invocations) - **Health status** = "Throttling Detected" if `Throttles > 0`, else "Healthy" - Report only resources with activity (any of vCPU-hours, GB-hours, invocations, session count > 0) This pillar requires only `cloudwatch:*` — it works in observability-only mode with no `bedrock-agentcore:` permissions. When an account isn't using AgentCore, the namespace is empty → "no activity detected" (not a failure). -
report-template.md 6.3 KB
# Artifact Report Template The artifact produced by this agent should be titled: **"Bedrock AgentCore Operational Review — [Account ID] — [YYYY-MM-DD]"** ## Required Sections (in order) All sections MUST appear in the artifact. If no data is available for a section, include the section header with text: "No data available — check was not executed or returned no results." ### 1. Executive Summary **Table format:** | Pillar | Status | Key Finding | Severity | |--------|--------|-------------|----------| | Runtime Resilience | ✅ / ⚠️ / ❌ | One-line summary | Critical/Warning/Pass | | Gateway Health | ✅ / ⚠️ / ❌ | One-line summary | Critical/Warning/Pass | | Memory & Knowledge Effectiveness | ✅ / ⚠️ / ❌ | One-line summary | Critical/Warning/Pass | | Resource Utilization & Hygiene | ✅ / ⚠️ / ❌ | One-line summary | Critical/Warning/Pass | Status logic: - ✅ Pass: All checks in pillar pass - ⚠️ Warning: At least one Warning/Medium/Low, no Critical/High - ❌ Critical: At least one Critical or High finding ### 2. Prioritized Findings **Table format:** | # | Priority | Rule | Action | Pillar | Impact | Effort | |---|----------|------|--------|--------|--------|--------| | 1 | Critical | AC-RUN-1 | [Remediation action] | Runtime Resilience | High | Low | | 2 | High | AC-MEM-3 | [Remediation action] | Memory Effectiveness | High | Medium | Sort by severity: Critical → High → Medium → Low → Informational, then by Impact. ### 3. KPI Summary **Table format:** | Metric | Value | |--------|-------| | Accounts scanned | N | | Regions active | N | | Runtimes | N | | Gateways | N | | Memories | N | | Total invocations (30d) | N | | Throttle rate | N% | | Idle resources | N | | Overall utilization | N% | | Total findings | N (X Critical, Y High, Z Medium) | ### 4. Runtime Inventory & Observability **Table format:** | Runtime Name | Runtime ID | Status | Network Mode | Subnet AZs | Endpoints | Latest Version | Max Drift | Session Count (30d) | Invocations (30d) | Throttle Rate | vCPU-Hours | GB-Hours | Health | |--------------|-----------|--------|--------------|-----------|-----------|----------------|-----------|---------------------|-------------------|---------------|-----------|----------|--------| ### 5. Gateway Health **Table format:** | Gateway ID | Status | Protocol | Authorizer | Total Targets | Healthy | Unhealthy | Single Point of Failure | Policy Engine | Stale Sync | Risk Level | |------------|--------|----------|------------|---------------|---------|-----------|-------------------------|---------------|------------|------------| ### 6. Memory & Knowledge Effectiveness **Table format:** | Memory ID | Memory Name | Status | Strategies | Total Records | Days Since Creation | Ingestion Invocations | Ingestion Errors | Error Rate | Findings | Severity | |-----------|-------------|--------|-----------|---------------|---------------------|-----------------------|------------------|-----------|----------|----------| Note: "Total Records" = N/A for short-term-only memories. ### 7. Resource Utilization Summary **Per-type table:** | Resource Type | Total Provisioned | Active | Idle | Active (partial) | Recently Created | Utilization % | Regions Used | |---------------|-------------------|--------|------|------------------|------------------|---------------|--------------| **Per-resource inventory (idle/flagged only):** | Resource Type | Resource ID | Region | Age (days) | Activity Count | Signal Source | Status | Standing Cost? | Findings | |---------------|------------|--------|-----------|----------------|---------------|--------|----------------|----------| ### 8. Per-Pillar Detailed Findings For each pillar, include a sub-section: #### [Pillar Emoji] [Pillar Name] | Check ID | Check Name | Status | Current State | Expected State | Remediation | |----------|------------|--------|---------------|----------------|-------------| Pillar emojis: Runtime Resilience=🛡️, Gateway Health=🚪, Memory & Knowledge Effectiveness=🧠, Resource Utilization=♻️ ### 9. Visibility Limits **Table format:** | Scope | Summary | Affected Resources | |-------|---------|--------------------| List every check that was skipped or degraded due to missing permissions or incomplete signals (e.g. `bedrock-agentcore` access denied, `ec2:DescribeSubnets` missing, CloudWatch signal incomplete for Browser/CodeInterpreter). This section makes the "not covered" scope explicit and prevents incomplete signals from being read as passes. ### 10. Methodology **Text block:** ``` Assessment conducted: [Date] Account(s): [List] Regions: [List] Pillars: Runtime Resilience, Gateway Health, Memory & Knowledge Effectiveness, Resource Utilization Mode: [Full control-plane / Observability-only] Time windows: 30-day activity + memory ingestion, 14-day health events Tools: use_aws (bedrock-agentcore, cloudwatch, ec2, ce, health) Checks executed: [N] / [Total] Checks skipped: [List any skipped due to access issues — cross-reference Visibility Limits] ``` ## Recommendation Format For each finding with severity Warning/Medium or higher, create a separate recommendation: **Title**: `[RULE-ID] [Check Name] — [Resource identifier]` **Summary**: ``` Pillar: [Name] Check: [Rule ID — Check Name] Resource: [ARN or identifier] Current state: [What was found] Expected state: [What should be] Impact: [Operational/reliability impact of the gap] Remediation: 1. [Step 1] 2. [Step 2] 3. [Step 3] Conversation starter: [TAM-ready, plain-language framing of the finding] Priority: [Critical/High/Medium/Low] Effort: [Low/Medium/High] ``` ### Example conversation starters (from the check design) - **AC-RUN-1**: "One of your agent runtimes is in an UPDATE_FAILED state. Are you aware of this? Let's investigate and get it back to healthy." - **AC-RUN-2**: "Your runtime reaches your VPC through subnets in a single availability zone. An AZ disruption would cut the agent off from your VPC resources. Adding a subnet in a second AZ restores fault tolerance." - **AC-MEM-1**: "CloudWatch shows extraction errors on this memory — your agent isn't learning from that data. Have you reviewed the source event format or extraction-model permissions?" - **AC-UTIL-1**: "This runtime shows no activity in 30 days. Since AgentCore only bills active consumption it's not a direct cost, but it still carries IAM roles and config nobody is maintaining. Can we decommission it?"
-
-
.skilleval.yaml 390 B
# skill-eval audit configuration # See: https://github.com/aws-samples/sample-agent-skill-eval audit: ignore: # README.md alongside SKILL.md is intentional and required by this repo's # contribution guide (README carries the non-production disclaimer, # prerequisites, and upload steps). Matches the convention used by the # other skills in this repository. - STR-016 -
CHANGELOG.md 2.2 KB
# Changelog ## [1.0.1] - 2026-09-16 ### Changed - Clarified the data-boundary documentation: `bedrock-agentcore:ListMemoryRecords` is a data-plane call and its response can include PII-bearing memory `content`. The skill uses it **only** to count records (AC-MEM-2) and never reads, stores, or reproduces `content`. Updated SKILL.md, README (new "Memory record data handling" section), references, sample IAM policies, and the CloudFormation block to state this accurately, and separated the action into its own clearly-labeled IAM statement so it can be omitted for zero data-plane access. ## [1.0.0] - 2026-09-10 ### Added - Initial release of the `agentcore-ops-review` skill for AWS DevOps Agent. - Runtime Resilience checks (AC-RUN-1..4): failed/stuck runtime and endpoint states, single-AZ VPC placement, endpoint version drift, and DEFAULT-endpoint-only deployments. - Gateway Health check (GW-01): gateway status, target redundancy and health, policy engine attachment, and target synchronization freshness. - Memory & Knowledge Effectiveness checks (AC-MEM-1..4): strategy-aware extraction-error, empty long-term memory, high ingestion error rate, and provisioned-but-never-populated detection. - Resource Utilization & Operational Hygiene checks (AC-UTIL-1..3): idle resources, consolidation opportunities, and low overall utilization. Estate discovery covers runtimes, memories, gateways, browsers, code interpreters, and workload identities. - Cross-pillar runtime observability from the `AWS/Bedrock-AgentCore` CloudWatch namespace (`SessionCount`, `Invocations`, `Throttles`, `CPUUsed-vCPUHours`, `MemoryUsed-GBHours`), with `ActiveSessionCount` preferred for the idle-resource signal where available. - Two IAM modes: runtime-observability-only (CloudWatch only) and full control-plane (read-only `bedrock-agentcore:` set + `ec2:DescribeSubnets`). Missing permissions degrade affected checks to visibility limits rather than failing the review. - Structured report artifact with executive summary, prioritized findings, KPI summary, per-pillar detail, visibility limits, and methodology (`references/report-template.md`). - Sample IAM policies for observability-only, linked, and management accounts. -
README.md 11.6 KB
# Bedrock AgentCore Operational Review — AWS DevOps Agent Skill A comprehensive Amazon Bedrock AgentCore operational review skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html). Conducts best-practices assessments aligned with the [Amazon Bedrock AgentCore Developer Guide](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html) and the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html). Generates a shareable report artifact for the review. ## What It Does When activated via Chat, this skill instructs the DevOps Agent to: 1. Discover AgentCore resources in the configured account/regions — agent runtimes, endpoints, versions, memories (and their strategies), gateways and gateway targets, browsers, code interpreters, and workload identities. 2. Collect CloudWatch metrics from the `AWS/Bedrock-AgentCore` namespace for session counts, invocations, throttles, vCPU-hours, GB-hours, and memory ingestion errors. 3. Resolve VPC subnet placement to availability zones (via `ec2:DescribeSubnets`) to assess runtime AZ fault tolerance. 4. Analyze against four check areas mapped to the Well-Architected Framework — **Runtime Resilience** (Reliability), **Gateway Health** (Reliability), **Memory & Knowledge Effectiveness** (Performance Efficiency), and **Resource Utilization & Operational Hygiene** (Operational Excellence) — plus cross-pillar runtime observability signals. 5. Generate a shareable report artifact, titled `Bedrock AgentCore Operational Review — <account-id> — <YYYY-MM-DD>`. All data is gathered through native AWS APIs (`bedrock-agentcore`, `cloudwatch`, `ec2`, `ce`, `health`). The skill never invokes an agent (`InvokeAgentRuntime`) and reads no prompts or responses. It makes **one** data-plane call — `bedrock-agentcore:ListMemoryRecords` — solely to count long-term memory records; that response can include potentially PII-bearing `content`, which the skill never reads, stores, or reproduces (see [Memory record data handling](#memory-record-data-handling)). It does not depend on any internal tooling. ## Agent Types This skill is intended for the following agent types (selected in the Operator Web App at upload time): - **On-demand** — conversational invocation in Chat ("review my AgentCore runtimes", "AgentCore health check"). - **Evaluation** — proactive operational improvement recommendations. Select **Generic** instead if you want the skill available to all agent types. ## Prerequisites ### 1. An AWS DevOps Agent Space with the target AWS account You need an existing [Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html) with the target AWS account configured as a cloud source. ### 2. IAM permissions — two modes The standard `AIDevOpsAgentAccessPolicy` covers `bedrock:*` read actions but does **NOT** include the `bedrock-agentcore:*` namespace. (Note: `bedrock-agentcore-control` is the SDK client name, not an IAM prefix — every control-plane action such as `ListAgentRuntimes`, `GetMemory`, and `ListGateways` authorizes under the single service prefix `bedrock-agentcore:`.) The skill supports two modes: **Runtime-observability-only mode** — relies exclusively on CloudWatch metrics against namespace `AWS/Bedrock-AgentCore`. Requires no `bedrock-agentcore:` additions. See `references/iam-policy-observability-only.json`: - `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData` - `health:DescribeEvents`, `health:DescribeEventDetails` - `sts:GetCallerIdentity` - `ce:GetCostAndUsage`, `ce:GetDimensionValues` (region discovery) **Full control-plane mode** — adds the read-only `bedrock-agentcore:` set plus `ec2:DescribeSubnets` to enable the runtime/gateway/memory/utilization checks. See `references/iam-policy-linked-account.json` (and `references/iam-policy-management-account.json` for the payer/management account): - `bedrock-agentcore:ListAgentRuntimes`, `GetAgentRuntime`, `ListAgentRuntimeEndpoints`, `ListAgentRuntimeVersions` - `bedrock-agentcore:ListMemories`, `GetMemory` (control-plane), `ListMemoryRecords` (data-plane; record count only — see [Memory record data handling](#memory-record-data-handling)) - `bedrock-agentcore:ListGateways`, `GetGateway`, `ListGatewayTargets`, `GetGatewayTarget` - `bedrock-agentcore:ListBrowsers`, `ListCodeInterpreters`, `ListWorkloadIdentities` - `ec2:DescribeSubnets` (AC-RUN-2 multi-AZ check) The skill operates in **read-only** mode: it never calls `Create*`, `Update*`, `Delete*`, or `InvokeAgentRuntime`. Its only data-plane call is `bedrock-agentcore:ListMemoryRecords`, used for a record count and nothing else (see [Memory record data handling](#memory-record-data-handling)). If a required permission is missing, the affected check degrades to a **visibility limit** ("signal unavailable — check skipped") rather than failing the review or producing a false finding. ### 3. AgentCore workloads with activity (recommended) Most checks rely on `AWS/Bedrock-AgentCore` CloudWatch metrics and control-plane inventory, which only exist for accounts actively using AgentCore. Reviewing an account with no AgentCore activity produces "No AgentCore usage detected" rather than false findings. ### 4. Relationship to `agentcore-observability-setup` This skill assesses operational posture from *existing* telemetry; it does not configure or validate observability wiring. Where telemetry is absent, it reports a visibility limit and defers the configuration gap to the `agentcore-observability-setup` skill, which owns the observability-wiring finding. ## Uploading to AWS DevOps Agent > Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) ### 1. Package the skill From the `skills/` directory in this repo, build the archive from **inside** the skill directory so `SKILL.md` sits at the archive root (nesting it under a subdirectory causes `Failed to get skill resource` errors at load time): ```bash cd skills/agentcore-ops-review zip -qrD ../agentcore-ops-review.zip . \ -x 'README.md' 'CHANGELOG.md' '.skilleval.yaml' 'evals/*' ``` The resulting `agentcore-ops-review.zip` contains `SKILL.md` at the root plus the `references/` files: ``` SKILL.md # frontmatter + skill instructions (required, at root) references/ ├── pillar-checks.md ├── report-template.md ├── iam-policy.json ├── iam-policy-observability-only.json ├── iam-policy-linked-account.json └── iam-policy-management-account.json ``` `README.md`, `CHANGELOG.md`, `.skilleval.yaml`, and `evals/` are excluded from the upload — they are repo/offline-evaluation artifacts, not part of the runtime skill. Constraints (enforced at upload time): - Total zip size ≤ **6 MB**. - `SKILL.md` is required and must include `name` and `description` frontmatter. - A `scripts/` directory is **not** allowed — uploads containing scripts are rejected. ### 2. Upload via the Operator Web App 1. Navigate to the **Skills** page in your Agent Space Operator Web App. 2. Click **Add skill** → **Upload skill**. 3. Drag and drop `agentcore-ops-review.zip` (or browse to it). 4. Select agent types: **On-demand** and **Evaluation** (or leave **Generic** to make it available to all agent types). 5. Review the validation results. 6. Click **Upload**. ## Usage In the DevOps Agent Chat, use natural language: - *"Run an AgentCore operational review for all regions."* - *"Review my AgentCore runtimes in `us-east-1` for resilience."* - *"Audit my agent memory pipelines for extraction errors."* - *"Check my AgentCore gateway health and target redundancy."* - *"ORR for our AgentCore workloads."* The agent will: - Collect all data automatically (no prompts for confirmation). - Use only AWS read APIs — no agent invocation, no prompts/responses read; the sole data-plane call (`ListMemoryRecords`) is used for a record count only, never for `content`. - Generate a report artifact titled `Bedrock AgentCore Operational Review — <account-id> — <YYYY-MM-DD>`. ## Skill Contents ``` agentcore-ops-review/ ├── SKILL.md # main skill instructions (with frontmatter) ├── README.md # this file ├── CHANGELOG.md # version history ├── references/ │ ├── pillar-checks.md # detailed check definitions, rules, thresholds │ ├── report-template.md # artifact report structure │ ├── iam-policy.json # policy index / notes │ ├── iam-policy-observability-only.json │ ├── iam-policy-linked-account.json │ └── iam-policy-management-account.json └── evals/ # evaluation data (not included in upload zip) ``` ## Check Areas Covered | Pillar (WA) | Check Area | Rule IDs | Reference | |-------------|-----------|----------|-----------| | Reliability | Runtime Resilience & Production Readiness | AC-RUN-1..4 | [AgentCore runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime.html) | | Reliability | Gateway Health & Target Redundancy | GW-01 | [AgentCore gateway](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway.html) | | Performance Efficiency | Memory & Knowledge Effectiveness | AC-MEM-1..4 | [AgentCore memory](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html) | | Operational Excellence | Resource Utilization & Operational Hygiene | AC-UTIL-1..3 | [AgentCore observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability.html) | | Cross-pillar | Runtime Observability (session count, invocations, throttles, vCPU/GB-hours) | — | [AgentCore observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability.html) | ## Severity Definitions | Severity | Definition | SLA | |----------|------------|-----| | CRITICAL | Immediate risk to availability, security, or data integrity | 24–48 hours | | HIGH | Significant gap that could lead to incidents | 1 week | | MEDIUM | Notable improvement opportunity | 30 days | | LOW | Minor optimization or hardening | When convenient | | INFO | Observation, no action required | N/A | ## Memory record data handling The skill's only data-plane call is `bedrock-agentcore:ListMemoryRecords`, used to count records for long-term memories (check AC-MEM-2). Each `MemoryRecordSummary` in that response includes a required `content` field — the extracted facts, preferences, and summaries a long-term memory has stored, which can contain end-user PII. This skill uses **only the record count**. It does not read, parse, log, store, transform, transmit, or reproduce the `content` field anywhere — not in findings, artifacts, recommendations, or logs. If you prefer the skill make **zero** data-plane calls, omit `bedrock-agentcore:ListMemoryRecords` from the IAM policy. AC-MEM-2 (empty/near-empty long-term memory) then degrades to a documented visibility limit, while all other memory checks — which rely on CloudWatch ingestion metrics (`Errors`, `Invocations`, `NumberOfMemoryRecords`) — continue to work. ## Non-production disclaimer > ⚠️ This skill is sample code, not intended for production use without additional review and testing. Users should validate it in a non-production environment first. It performs read-only operational analysis and makes no changes to your AWS resources, but you are responsible for reviewing the IAM permissions you grant and the findings it produces before acting on them. -
SKILL.md 15.7 KB
--- name: agentcore-ops-review description: Comprehensive operational review procedures for Amazon Bedrock AgentCore resources aligned with the AWS Well-Architected Framework. Covers four check areas — Runtime Resilience, Gateway Health, Memory & Knowledge Effectiveness, and Resource Utilization & Operational Hygiene — plus runtime observability signals from CloudWatch. Use this skill when a user asks to review, audit, or assess Amazon Bedrock AgentCore workloads, perform an AgentCore operational readiness review, investigate agent runtime failures, audit memory/knowledge pipeline health, or review gateway resilience and resource hygiene. Triggers on requests like "AgentCore review", "AgentCore best practices audit", "review my AgentCore runtimes", "AgentCore health check", "audit my agent memory pipelines", or "ORR for AgentCore". metadata: author: pamvas version: "1.0.1" aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" aws-devops-agent-skills.aws-services: "Amazon Bedrock AgentCore" aws-devops-agent-skills.technical-domains: "Machine Learning, GenAI" --- # Bedrock AgentCore Operational Review Use this skill when performing an operational review of Amazon Bedrock AgentCore resources, investigating agent runtime failures, auditing memory/knowledge pipeline health, or reviewing gateway resilience and resource hygiene. ## Overview This skill provides procedures for a review of AgentCore resources across four check areas mapped to the AWS Well-Architected Framework: - **Runtime Resilience** (Reliability) — Failed/stuck runtimes and endpoints, single-AZ VPC placement, endpoint version drift, DEFAULT-endpoint-only deployments - **Gateway Health** (Reliability) — Gateway status, target redundancy and health, policy engine attachment, target synchronization freshness - **Memory & Knowledge Effectiveness** (Performance Efficiency) — Extraction pipeline errors, empty long-term memory, high ingestion error rate, provisioned-but-never-populated memories - **Resource Utilization & Operational Hygiene** (Operational Excellence) — Idle resources, consolidation opportunities, low overall utilization - **Runtime Observability** (cross-pillar) — Session counts, invocations, throttles, vCPU-hours, GB-hours per runtime This is a **READ-ONLY** review. No modifications are made to any resource, and it never invokes an agent (`InvokeAgentRuntime`), reads no prompts or responses, and makes no other data-plane calls. **One exception:** for long-term memories the skill calls `bedrock-agentcore:ListMemoryRecords` (a data-plane API) solely to obtain a **record count**. That response can include a `content` field carrying extracted, potentially PII-bearing memory content; the skill uses **only the count** and never parses, logs, stores, or reproduces `content`. See "Memory record data handling" below. ## Data Source Boundaries (IMPORTANT — IAM footprint) The standard `AIDevOpsAgentAccessPolicy` covers `bedrock:*` read actions but does **NOT** include the `bedrock-agentcore:*` namespace. (Note: `bedrock-agentcore-control` is the SDK client name, not an IAM prefix — all control-plane actions such as `ListAgentRuntimes`, `GetMemory`, and `ListGateways` authorize under the single service prefix `bedrock-agentcore:`.) Two modes: 1. **Runtime-observability-only mode** — relies exclusively on CloudWatch metrics (`cloudwatch:GetMetricData`, `cloudwatch:ListMetrics`) against namespace `AWS/Bedrock-AgentCore`. Requires no `bedrock-agentcore:` additions. When an account isn't using AgentCore, the namespace is simply empty ("no activity detected") — this is not a failure. 2. **Full control-plane mode** — adds read-only `bedrock-agentcore:` control-plane List/Get actions and `ec2:DescribeSubnets` to enable the runtime/gateway/memory/utilization checks. It also uses one **data-plane** action, `bedrock-agentcore:ListMemoryRecords`, for the long-term-memory record count only (see "Memory record data handling" below). See `references/iam-policy-linked-account.json`. If a required permission is missing, the affected check degrades to a **visibility limit** (reported as "signal unavailable / check skipped") rather than failing the review or producing a false finding. **Memory record data handling:** `bedrock-agentcore:ListMemoryRecords` is a data-plane API whose `MemoryRecordSummary` entries include a required `content` field — the extracted facts/preferences/summaries a long-term memory has stored, which can contain end-user PII. This skill calls it **only** to count records for AC-MEM-2 and never reads, parses, logs, stores, transforms, or reproduces the `content` field. If you prefer zero data-plane access, omit this action from the IAM policy: AC-MEM-2 then degrades to a visibility limit while all other memory checks (which use CloudWatch ingestion metrics) continue to work. **Seam with `agentcore-observability-setup`:** This skill assesses operational posture from *existing* telemetry; it does not configure or validate observability wiring. Where telemetry is absent, this skill reports a visibility limit and defers the configuration gap to `agentcore-observability-setup` — that skill owns the observability-wiring finding (Transaction Search, OTEL/ADOT, log delivery, X-Ray resource policy), while this skill reports only the posture consequence. A customer running both should not see two overlapping findings on the same resource. ## Step 1: Discover Account and Regions 1. Call `sts:GetCallerIdentity` to determine the account. 2. Discover AgentCore-active regions: - Query Cost Explorer: `ce:GetDimensionValues` (dimension SERVICE, last 30 days), match "Amazon Bedrock AgentCore". - Then `ce:GetCostAndUsage` for that service grouped by REGION; select top regions by UnblendedCost. - If no AgentCore spend is found, fall back to probing `cloudwatch:ListMetrics` (namespace `AWS/Bedrock-AgentCore`) in the customer's primary regions to detect activity. 3. If no AgentCore activity is detected in any region, stop: "No AgentCore usage detected in the last 30 days." ## Step 2: Runtime Resilience Pillar (Reliability) Discover runtimes and enrich each with endpoints and versions: - `bedrock-agentcore:ListAgentRuntimes` → `GetAgentRuntime` (status, `failureReason`, `networkConfiguration`) - `bedrock-agentcore:ListAgentRuntimeEndpoints` (status, `liveVersion`, `targetVersion`, name) - `bedrock-agentcore:ListAgentRuntimeVersions` (to compute latest version + drift) - `ec2:DescribeSubnets` (resolve VPC subnet IDs to AZs — endpoints carry NO AZ data) | Check | Rule | Severity | |-------|------|----------| | AC-RUN-1 | Runtime/endpoint in `CREATE_FAILED`/`UPDATE_FAILED`, or stuck in `CREATING`/`UPDATING` > 1 hour | **Critical** | | AC-RUN-2 | VPC-mode runtime whose subnets all resolve to a single AZ (PUBLIC mode exempt) | **High** | | AC-RUN-3 | Live endpoint ≥ 3 versions behind latest runtime version | **Medium** | | AC-RUN-4 | Actively-updated runtime (2+ versions) served only by the auto-updating DEFAULT endpoint | **Low** | **Key fact:** Endpoints are a versioning/traffic-routing construct, not a redundancy mechanism. AZ fault tolerance is derived exclusively from the runtime's VPC subnet configuration. If `ec2:DescribeSubnets` is unavailable, AC-RUN-2 degrades to "AZ distribution unknown". ## Step 3: Gateway Health (Reliability) Discover gateways and their targets: - `bedrock-agentcore:ListGateways` → `GetGateway` (status, protocolType, authorizerType, `policyEngineConfiguration`) - `bedrock-agentcore:ListGatewayTargets` → `GetGatewayTarget` (status, `targetConfiguration`, `lastSynchronizedAt`, credential providers) Target type derived from `targetConfiguration.mcp`: Lambda, MCP Server, API Gateway, OpenAPI Schema, Smithy Model. | Risk Level | Condition | |------------|-----------| | **Critical** | Gateway not `READY`, zero targets, OR all targets unhealthy | | **Warning** | Single target (no redundancy), no policy engine attached, some unhealthy targets, OR stale sync (> 7 days) | | **Healthy** | Multiple READY targets, policy engine attached, recent synchronization | Unhealthy target statuses: `FAILED`, `UPDATE_UNSUCCESSFUL`, `SYNCHRONIZE_UNSUCCESSFUL`. Stale sync threshold: `lastSynchronizedAt` older than 7 days. ## Step 4: Memory & Knowledge Effectiveness (Performance Efficiency) Discover memories and their strategies, then query CloudWatch ingestion metrics: - `bedrock-agentcore:ListMemories` → `GetMemory` (status, createdAt, **configured strategies**) - `bedrock-agentcore:ListMemoryRecords` — **data-plane call, count only** (long-term-strategy memories). Read the returned record *count*; do not read the `content` field (see "Memory record data handling") - `cloudwatch:GetMetricData` (namespace `AWS/Bedrock-AgentCore`): per-memory `Invocations`/`Errors` for the `Ingestion` operation, and `Invocations` for the `CreateEvent` operation (30-day window) **All rules are strategy-aware** — read `GetMemory` strategies first. Record-count rules apply ONLY to memories with a long-term strategy; short-term-only memories are never flagged as empty. | Check | Rule | Severity | |-------|------|----------| | AC-MEM-1 | Long-term memory with CloudWatch ingestion `Errors > 0` | **High** | | AC-MEM-2 | Long-term memory with < 10 records (escalates to High if 0 records AND > 7 days old) | **Medium→High** | | AC-MEM-3 | Ingestion error rate > 20% (`Errors / Invocations`) | **High** | | AC-MEM-4 | Provisioned but never populated: zero `CreateEvent` activity AND > 7 days old | **Medium** | **Safety rule:** `CreateEvent` activity is queried for EVERY memory (short-term memories receive events too). When the CloudWatch signal is unreadable, event count is `None` (not 0), so AC-MEM-4 is skipped rather than firing a false "never populated" finding. ## Step 5: Resource Utilization & Operational Hygiene (Operational Excellence) Enumerate all provisionable resource types and collect a 30-day activity signal: - Inventory: `ListAgentRuntimes`, `ListMemories`, `ListGateways`, `ListBrowsers`, `ListCodeInterpreters`, `ListWorkloadIdentities` - Activity: `cloudwatch:GetMetricData` (`Invocations`, `AWS/Bedrock-AgentCore`) keyed by per-type dimension (`AgentRuntimeId`, `MemoryId`, `GatewayId`) | Check | Rule | Severity | |-------|------|----------| | AC-UTIL-1 | Idle resource: zero activity over 30-day window, > 7 days old | **Medium** | | AC-UTIL-2 | Consolidation: duplicate configs in a region, OR regions holding resources with < 5% of total activity (3+ regions) | **Informational→Medium** | | AC-UTIL-3 | Overall utilization < 60% (active / assessable, excluding recently-created) | **Informational** | For the AC-UTIL-1 idle signal, prefer the real-time `ActiveSessionCount` gauge (a currently-running-sessions gauge, filterable by the `Service` dimension) where available; fall back to the cumulative `SessionCount` / `Invocations` counters over the 30-day window when `ActiveSessionCount` is not present. **Framing:** AgentCore runtime billing is consumption-based — idle time is free. Frame findings as operational hygiene and security surface (unmanaged IAM roles, stale config), NOT wasted spend. Exception: memories holding stored long-term records DO accrue storage cost — call this out in the finding. **Classification safety:** A resource is only classified **Idle** when its activity signal is complete for the full window. Types with no queryable per-resource metric (Browser, CodeInterpreter, WorkloadIdentity), and every resource when the CloudWatch call fails, are classified **Active (partial)** — never Idle. Resources younger than 7 days are **RecentlyCreated** and excluded from both idle flagging and the utilization ratio. ## Step 6: Runtime Observability (cross-pillar signals) For each runtime, collect CloudWatch metrics from namespace `AWS/Bedrock-AgentCore`: | Metric | Dimensions | Period | Statistic | |--------|-----------|--------|-----------| | CPUUsed-vCPUHours | Resource (ARN), Service=`AgentCore.Runtime`, Name | 3600s | Sum | | MemoryUsed-GBHours | Resource (ARN), Service=`AgentCore.Runtime`, Name | 3600s | Sum | | SessionCount | Resource (ARN), Operation=`InvokeAgentRuntime`, Name | 300s | Sum | | Invocations | Resource (ARN), Operation=`InvokeAgentRuntime`, Name | 300s | Sum | | Throttles | Resource (ARN), Operation=`InvokeAgentRuntime`, Name | 300s | Sum | `SessionCount` is a cumulative counter of new sessions per period. Where a real-time view is needed, `ActiveSessionCount` is a gauge of currently-running sessions (filterable by the `Service` dimension). Discover resources first via `cloudwatch:ListMetrics` (namespace `AWS/Bedrock-AgentCore`), extract unique `Resource` ARN + `Name` dimensions, then batch `GetMetricData` per resource. Derived signals: - **Throttle rate** = `Throttles / Invocations × 100` - **Health status** = "Throttling Detected" when `Throttles > 0`, else "Healthy" - Only report runtimes with any activity (CPU-hours, GB-hours, invocations, or session count > 0). ## Step 7: Additional Context 1. **Health Events**: `health:DescribeEvents` filtered for Bedrock/AgentCore service, last 14 days. 2. **Documentation**: Cross-reference findings against AgentCore best-practices docs for remediation links. ## Step 8: Produce Report and Recommendations Produce an artifact with the structure defined in `references/report-template.md`. For each finding with severity Warning/Medium or higher, create a recommendation with: - Title: `[Check ID] [Check name] — [Resource identifier]` - Summary: Current state, expected state, business/operational impact, remediation steps, and a TAM conversation starter. ## Related Skills / Boundaries Boundaries are deliberate and non-overlapping: - **`agentcore-observability-setup`** — sets up/validates observability *wiring*. This skill consumes that telemetry for posture assessment and defers observability-*configuration* gaps to it (see the seam described in Data Source Boundaries). - **`agentcore-runtime-diag`** (queued submission — referenced for scope, not yet published) — reactive triage of a *single failing invocation*. This skill is proactive, estate-wide posture; it does not diagnose individual call failures or evaluate authorizer/credential correctness. Gateway checks here are redundancy/staleness/health only, not auth evaluation. - **`aiml-access-diagnostics`** (published) — generic AI/ML AccessDenied chain via `iam:SimulatePrincipalPolicy`. Not overlapping — this skill makes no IAM-simulation calls. - **`bedrock-adoption-readiness`** (published) — foundation-model workload readiness (IAM governance, ZDR, quotas, observability). Different resource surface — this skill reviews the AgentCore agent runtime / memory / gateway layer, not foundation-model inference. ## Error Handling | Error | Action | |-------|--------| | AccessDenied on `bedrock-agentcore:*` | Log as visibility limit "signal unavailable — check skipped", continue in observability-only mode | | AccessDenied on `ec2:DescribeSubnets` | AC-RUN-2 degrades to "AZ distribution unknown" | | CloudWatch returns no data | Report "no activity detected" for that resource; classify utilization as Active (partial), never Idle | | No AgentCore usage anywhere | Stop early: "No AgentCore usage detected" | | Throttled by AWS API | Retry with exponential backoff (3 attempts) | | Check raises an exception | Isolate the failure — degrade that check to a visibility limit and continue the others | ## Important Notes - ALL API calls with pagination (NextToken) MUST be paginated to completion. - Cost Explorer queries run against us-east-1 (global endpoint). - Findings are only produced when the underlying signal is complete — incomplete signals become visibility limits, never false positives. - Batch CloudWatch `GetMetricData` requests where possible. - This is a READ-ONLY review — no modifications, and no data-plane calls other than `ListMemoryRecords` for record count (count only; `content` is never read).
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.