Claude Skill

enrich-with-aws-security-agent

Automatically load this skill when investigating application outages, service degradation, or errors that could have security-related root causes — including unexplained downtime, authentication or authorization failures, injection attacks, data exposure, or suspicious applicatio

LLM Mart · 0 points · 0 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download aws-tools-for-devops-agent-skills_enrich-with-aws-security-agent-1c971c7.zip · 13 KB

aws/tools-for-devops-agent 67 53 forks Apache-2.0 Updated 8d ago

Part of aws/tools-for-devops-agent — 21 skills

Install

skills CLI npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/enrich-with-aws-security-agent

Claude Code

claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart

Git git clone https://github.com/aws/tools-for-devops-agent.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.

README

Enrich with AWS Security Agent

This skill enables the AWS DevOps Agent to query AWS Security Agent CloudWatch logs during incident investigations, surfacing code-level security findings that may be the root cause of application outages.

⚠️ Important Notice

This skill is provided as sample code. If you intend to deploy it in production, start with a non-production environment first. Review IAM permissions and security configurations against your organization's policies, and validate that skill behavior meets your operational requirements before production use.

Purpose

When an application outage or degradation occurs without a clear infrastructure cause, the root cause may be a security vulnerability — such as SSRF, injection attacks, or compromised dependencies. This skill queries AWS Security Agent findings stored in CloudWatch Logs to retrieve actionable details (file path, line number, vulnerability type, severity) that customers can directly fix.

Key Capabilities

Query AWS Security Agent CloudWatch log groups for security findings
Extract actionable vulnerability details: file path, line number, vulnerability type, severity
Correlate security findings with current incident symptoms and timeframes
Present remediation guidance with exact code locations
Automatically activate when application errors have no obvious infrastructure cause

Prerequisites

AWS Security Agent configured and writing findings to CloudWatch Logs
Agent permissions for logs:DescribeLogGroups, logs:StartQuery, logs:GetQueryResults on Security Agent log groups (e.g., /aws/securityagent/*)

Limitations

Requires AWS Security Agent to be configured and actively scanning — no findings will exist if the service is not enabled
Log retention depends on CloudWatch Logs retention settings for the Security Agent log groups
Findings are only as current as the last Security Agent scan

Agent Types

This skill is used by the following agent types:

Chat tasks — conversational security finding lookup and analysis
Incident RCA — automated root cause analysis when security issues may cause outages

Uploading to AWS DevOps Agent

To deploy this skill to your Agent Space:

Zip the enrich-with-aws-security-agent/ directory (only including allowed extensions):

cd skills
zip -r enrich-with-aws-security-agent.zip enrich-with-aws-security-agent/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'

In the AWS DevOps Agent Operator Web App, navigate to the Skills page.
Click Add skill → Upload skill.
Drag and drop the enrich-with-aws-security-agent.zip file (max 6 MB).
Select the agent types: Chat tasks and Incident RCA.
Click Upload.

For more details, see Uploading a skill in the AWS DevOps Agent User Guide.

How to Use This Skill

Chat

"Check AWS Security Agent findings for my application — are there any critical vulnerabilities?"
"Show me security findings related to SSRF or injection attacks in the last 7 days."
"What security vulnerabilities has AWS Security Agent found in the orders service?"
"Are there any hardcoded secrets or credential exposure findings?"

Investigation

"My application is returning 500 errors but there were no recent deployments. Investigate."
"We're seeing unexplained outages on the checkout service — could it be a security issue?"
"The API started failing with authentication errors. Check if there are security findings related to this."
"Our web application is behaving unexpectedly and we suspect injection attacks. Investigate."

Skill manifest

Enriching Investigations with AWS Security Agent Findings

When to Use — AUTOMATIC ACTIVATION

Load this skill automatically when ANY of these conditions apply:

Application-Level Symptoms (even without explicit security context)

Application pages or endpoints are not loading or returning errors
Unexplained service outages with no recent deployments
Application behaving unexpectedly without clear infrastructure cause
Services returning 4xx/5xx errors without obvious resource exhaustion

Explicit Security Signals

Security vulnerabilities or misconfigurations mentioned
Suspicious code patterns or behaviors observed
Authentication or authorization failures
Injection vulnerabilities suspected (SQL, command, SSRF, etc.)
Sensitive data exposure concerns
Security-related errors in application logs
Findings that suggest code-level security issues

Key Insight

Many security issues (like SSRF, injection attacks, or compromised dependencies) manifest as application outages or errors rather than obvious security alerts. When an app stops working and there's no clear infrastructure cause (no deployments, no resource exhaustion, no config changes), always check for security issues.

Overview

AWS Security Agent is an AI-powered security service that performs automated code security reviews and penetration testing. Its findings are stored in CloudWatch Logs and provide detailed, actionable information about vulnerabilities including exact file locations, line numbers, and remediation guidance.

Steps

Identify the relevant CloudWatch log group for AWS Security Agent findings. Look for log groups with patterns like:
- /aws/securityagent/*
- Log groups containing "security-agent" or "securityagent" in the name

Query the Security Agent logs using CloudWatch Logs Insights:

fields @timestamp, @message
| filter @message like /finding|vulnerability|issue|risk/
| sort @timestamp desc
| limit 100

Extract actionable details from the findings:
- File path: The exact source file containing the vulnerability
- Line number: The specific line(s) of code affected
- Vulnerability type: Classification (e.g., SQL injection, XSS, SSRF, hardcoded secrets)
- Severity: Critical, High, Medium, Low
- Description: What the vulnerability is and why it's a risk
- Remediation: Suggested fix or mitigation
Correlate findings with the incident by matching:
- Affected services or components
- Timeframes (when the vulnerability was introduced vs. when the incident occurred)
- Code paths involved in the failing functionality
Present findings to the customer with:
- Clear identification of the vulnerable code location
- Explanation of how this vulnerability may relate to the incident
- Specific remediation steps they can take to fix the code

Example Log Query for Specific Application

If investigating a specific application (e.g., "ssrf-demo", "orders", "account"):

fields @timestamp, @message
| filter @message like /ssrf|orders|account|vulnerability|finding|issue/
| sort @timestamp desc
| limit 50

Important Notes

AWS Security Agent API access may require additional IAM permissions. If API calls fail, fall back to the CloudWatch logs approach.
Always provide the exact file and line number when available — this is the key value for customers.
Link the security finding to the observed incident behavior when possible.
When in doubt, check security logs — it's better to rule out security issues than to miss them.

Files (tools-for-devops-agent)

evals

benchmark.json 27.6 KB

{
  "skill_name": "enrich-with-aws-security-agent",
  "skill_path": "/Users/yakiratz/git/sample-code-for-devops-agent-skills/skills/enrich-with-aws-security-agent",
  "eval_count": 5,
  "runs_per_eval": 1,
  "metadata": {
    "timestamp": "2026-06-14T14:43:00Z"
  },
  "runs": [
    {
      "eval_id": "query-security-agent-logs",
      "run_index": 0,
      "with_skill": {
        "eval_id": "query-security-agent-logs",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output attempts to query CloudWatch Logs for Security Agent findings",
            "passed": true,
            "evidence": "output explicitly attempts to query CloudWatch Logs for Security Agent findings across multiple regions",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          },
          {
            "text": "The output looks for log groups matching /aws/securityagent/ or similar patterns",
            "passed": true,
            "evidence": "output describes searching for Security Agent log groups; mentions checking for non-standard names; implies pattern-based search though exact pattern not shown",
            "method": "llm",
            "confidence": 0.75,
            "uncertain": false
          },
          {
            "text": "The output does not produce an unhandled error or permission denied without explanation",
            "passed": true,
            "evidence": "no unhandled errors; clearly explains that no log groups were found and offers next steps",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          }
        ],
        "pass_rate": 1.0,
        "summary": "With skill: 100% assertions passed",
        "execution_metrics": {
          "tool_calls": 12,
          "token_counts": {
            "input_tokens": 30,
            "output_tokens": 2982,
            "cache_read_input_tokens": 685430,
            "cache_creation_input_tokens": 117250
          }
        },
        "timing": {
          "elapsed_seconds": 93.99542250000013
        },
        "raw_output": "I'll query CloudWatch Logs for AWS Security Agent findings. Let me load the AWS tools first.I need the AWS account ID and region to query CloudWatch Logs. Let me also discover the right Mechanic tools to use.Mechanic isn't installed. Let me check if the aws-mcp server tools have loaded.Mechanic isn't installed and aws-mcp tools aren't yet available. Let me check if AWS CLI is configured locally so I can query CloudWatch Logs directly.I have AWS credentials. Let me find Security Agent log groups and check for findings.No Security Agent log groups in us-east-1. Let me check other common regions.I checked the AWS account (872879685372) across us-east-1, us-west-2, us-east-2, and eu-west-1 for AWS Security Agent log groups. **No Security Agent log groups were found in any of those regions.**\n\nThe only security-adjacent log groups are GuardDuty (`/aws/guardduty/malware-scan-events`), which is a different service.\n\nSo I can't surface Security Agent findings for your 500 errors \u2014 there aren't any in this account. A few possibilities:\n\n1. **AWS Security Agent isn't deployed** in this account, or hasn't run yet against your application code.\n2. **The findings are in a different region** I didn't check \u2014 let me know which region your application runs in and I'll look there.\n3. **The log group has a non-standard name** \u2014 if you know the prefix or a substring, I can search for it.\n\nIn the meantime, to investigate the 500s I'd suggest checking:\n- The application's own CloudWatch log group (which one is your app \u2014 Lambda? EKS? ECS?)\n- GuardDuty findings (`/aws/guardduty/malware-scan-events` exists in us-east-1 and us-west-2)\n- VPC Flow Logs / unusual outbound traffic (often an SSRF or compromised-dependency tell)\n\nWhich app and region should I dig into?"
      },
      "without_skill": {
        "eval_id": "query-security-agent-logs",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output attempts to query CloudWatch Logs for Security Agent findings",
            "passed": false,
            "evidence": "Output does not query CloudWatch Logs; it asks clarifying questions and proposes options instead",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output looks for log groups matching /aws/securityagent/ or similar patterns",
            "passed": false,
            "evidence": "Output does not reference any log group patterns like /aws/securityagent/",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output does not produce an unhandled error or permission denied without explanation",
            "passed": true,
            "evidence": "Output is a coherent recommendation with no errors or unexplained permission denials",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          }
        ],
        "pass_rate": 0.3333333333333333,
        "summary": "Without skill: 33% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 6,
            "output_tokens": 954,
            "cache_read_input_tokens": 21163,
            "cache_creation_input_tokens": 21794
          }
        },
        "timing": {
          "elapsed_seconds": 38.80243029199846
        },
        "raw_output": "Security vulnerability findings (Inspector, Security Hub, GuardDuty) almost never cause 500 errors \u2014 they flag misconfigurations and CVEs, not runtime failures. With no recent deploys, 500s usually come from downstream issues: dependency outages, resource exhaustion (DB connections, memory, disk), expired certs, quota throttling, or upstream incidents.\n\nI'd recommend a root-cause investigation via the AWS DevOps Agent rather than a security sweep. Want me to:\n\n1. **Run `aws-devops-agent:investigate`** on the 500 errors (deep, traces alarms/logs/metrics) \u2014 best fit for \"service is broken, why?\"\n2. **Run a security findings check** anyway (Security Hub / Inspector / GuardDuty) \u2014 if you have a specific reason to suspect security\n3. **Both in parallel**\n\nWhich app/service is impacted, and do you have an alarm name, log group, or service ARN to anchor the investigation?"
      },
      "delta_pass_rate": 0.6666666666666667
    },
    {
      "eval_id": "extract-vulnerability-details",
      "run_index": 0,
      "with_skill": {
        "eval_id": "extract-vulnerability-details",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output searches for SSRF or injection-related findings",
            "passed": true,
            "evidence": "output explicitly searches for SSRF/injection patterns in log groups including 'ssrf' and 'inject'",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          },
          {
            "text": "The output attempts to extract file path and line number from findings",
            "passed": false,
            "evidence": "no log groups were found, so no findings were retrieved and no file path/line extraction was attempted",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          },
          {
            "text": "The output includes severity information or states it was not available",
            "passed": false,
            "evidence": "output does not mention severity at all, neither providing it nor stating it was unavailable",
            "method": "llm",
            "confidence": 0.85,
            "uncertain": false
          }
        ],
        "pass_rate": 0.3333333333333333,
        "summary": "With skill: 33% assertions passed",
        "execution_metrics": {
          "tool_calls": 9,
          "token_counts": {
            "input_tokens": 40,
            "output_tokens": 3347,
            "cache_read_input_tokens": 706549,
            "cache_creation_input_tokens": 102739
          }
        },
        "timing": {
          "elapsed_seconds": 112.79524504100118
        },
        "raw_output": "I'll query AWS Security Agent CloudWatch logs for SSRF/injection findings. Let me load the AWS MCP tools first.I'll first locate the AWS Security Agent CloudWatch log groups, then query them for SSRF/injection findings.I searched for AWS Security Agent CloudWatch log groups but none exist in this account.\n\n## Search results\n\nI queried CloudWatch in `us-east-1`, `us-west-2`, `us-east-2`, and `eu-west-1` for log groups matching `/aws/securityagent/*`, `security-agent`, `securityagent`, `security_agent`, and broader patterns (`security`, `vuln`, `finding`, `scan`, `ssrf`, `inject`). The only security-adjacent groups found are:\n\n- `/aws/guardduty/malware-scan-events` (us-west-2, eu-west-1) \u2014 GuardDuty, not Security Agent; covers malware scanning, not SSRF/injection code review\n- `/aws/bedrock-agentcore/runtimes/...` \u2014 Bedrock AgentCore runtimes (matched on the word \"agent\"), unrelated to AWS Security Agent\n\nThere is no `/aws/securityagent/*` log group in this account.\n\n## Possible reasons\n\n1. **AWS Security Agent isn't enabled** in this account \u2014 its CloudWatch logging integration may not be configured, or the service hasn't run a review yet.\n2. **Findings live in a different region** that I didn't check \u2014 let me know if there's a specific region to look in.\n3. **Findings are written to a non-standard log group / S3 / Security Hub** \u2014 some integrations route findings to AWS Security Hub or a customer-managed log group instead of `/aws/securityagent/*`.\n4. **You're in the wrong account** \u2014 Security Agent findings may be in a centralized security/audit account rather than this workload account.\n\n## What would help\n\n- Confirm which AWS account / region the Security Agent runs in.\n- Or I can pivot to **AWS Security Hub** (`aws securityhub get-findings`) and **Inspector** (`aws inspector2 list-findings`) which often hold SSRF/injection findings \u2014 want me to query those instead?\n\nI won't fabricate findings \u2014 I'll only report what's actually in the logs."
      },
      "without_skill": {
        "eval_id": "extract-vulnerability-details",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output searches for SSRF or injection-related findings",
            "passed": false,
            "evidence": "output asks for clarification rather than searching for SSRF/injection findings; only mentions them as examples of what could be pulled",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          },
          {
            "text": "The output attempts to extract file path and line number from findings",
            "passed": false,
            "evidence": "no attempt to extract file paths or line numbers; output declines to act without more context",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output includes severity information or states it was not available",
            "passed": false,
            "evidence": "no severity information provided and no statement that severity was unavailable",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "Without skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 12,
            "output_tokens": 816,
            "cache_read_input_tokens": 42422,
            "cache_creation_input_tokens": 56904
          }
        },
        "timing": {
          "elapsed_seconds": 34.88966312499906
        },
        "raw_output": "I don't have context about which application or where AWS Security Agent findings are stored for you. The current working directory is an empty temp folder, and I haven't been pointed at any specific finding source.\n\nA few things I'd need to actually answer this:\n\n1. **Which application/service?** \u2014 name, package, or AWS account.\n2. **Where are the findings?** \u2014 e.g., Mirador (`mirador.security.aws.dev`), Shepherd (`shepherd.a2z.com`), Talos, a SIM/Taskei ticket, or a report file on disk.\n\nI'm not going to guess or fabricate findings. If you can share an app name or a link (Mirador URL, Shepherd issue, ticket ID), I'll pull the actual SSRF/injection items and detail them."
      },
      "delta_pass_rate": 0.3333333333333333
    },
    {
      "eval_id": "correlate-findings-with-incident",
      "run_index": 0,
      "with_skill": {
        "eval_id": "correlate-findings-with-incident",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output attempts to correlate findings with the checkout service",
            "passed": false,
            "evidence": "output is empty, no correlation attempted",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output considers the timeframe of the incident",
            "passed": false,
            "evidence": "output is empty, no timeframe consideration",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output explains how any found vulnerability could cause the observed symptoms",
            "passed": false,
            "evidence": "output is empty, no vulnerability explanation",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "With skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.01514049999969
        },
        "raw_output": ""
      },
      "without_skill": {
        "eval_id": "correlate-findings-with-incident",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output attempts to correlate findings with the checkout service",
            "passed": false,
            "evidence": "Output mentions checkout service in clarifying questions but does not attempt to correlate any findings with it; instead pushes back on the premise and asks for service name",
            "method": "llm",
            "confidence": 0.85,
            "uncertain": false
          },
          {
            "text": "The output considers the timeframe of the incident",
            "passed": true,
            "evidence": "Output references the '2am UTC checkout outage' timeframe and notes that vulnerabilities are latent vs outages being time-bound events",
            "method": "llm",
            "confidence": 0.7,
            "uncertain": false
          },
          {
            "text": "The output explains how any found vulnerability could cause the observed symptoms",
            "passed": false,
            "evidence": "Output argues vulnerabilities and outages are usually unrelated and does not explain how any specific vulnerability could cause the symptoms",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          }
        ],
        "pass_rate": 0.3333333333333333,
        "summary": "Without skill: 33% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 12,
            "output_tokens": 1447,
            "cache_read_input_tokens": 42543,
            "cache_creation_input_tokens": 54624
          }
        },
        "timing": {
          "elapsed_seconds": 42.51593774999856
        },
        "raw_output": "I don't see an \"AWS Security Agent\" in my available tools. Let me check if anything matching that is available before responding.I don't have a tool called \"AWS Security Agent\" \u2014 I want to flag that before pretending to query one.\n\nWhat I do have that's adjacent:\n- **Shepherd** (`shepherd.a2z.com`) \u2014 security risks/findings per service\n- **Mirador** (`mirador.security.aws.dev`) \u2014 security findings and exceptions\n- **Aristotle** \u2014 AWS security knowledge base\n\nBut also worth pushing back: security vulnerabilities and an outage starting at a specific time are usually unrelated. Vulnerabilities are latent issues; outages are typically caused by deploys, infra failures, dependency issues, or traffic events. Better tools for a 2am UTC checkout outage:\n- `aws-devops-agent:investigate` \u2014 deep root-cause investigation for incidents\n- `alarm-investigation` skill \u2014 trace the breaching alarm\n- `log-investigation` skill \u2014 find the code emitting the failing metric and search logs\n- `pipeline-troubleshoot` if a recent deploy is suspect\n\nA few questions before I dig in:\n1. What's the **service name** (or pipeline name) for the checkout service?\n2. Is there an **alarm or ticket** I should start from?\n3. Did anything **deploy recently** to that service?\n4. Do you actually want me to check security findings too, or was the assumption that they caused the outage?"
      },
      "delta_pass_rate": -0.3333333333333333
    },
    {
      "eval_id": "no-findings-scenario",
      "run_index": 0,
      "with_skill": {
        "eval_id": "no-findings-scenario",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output indicates whether findings were found or not",
            "passed": false,
            "evidence": "output asks for inputs before querying; never states whether findings were found or not",
            "method": "llm",
            "confidence": 0.9,
            "uncertain": false
          },
          {
            "text": "The output does not fabricate security findings that do not exist",
            "passed": true,
            "evidence": "output contains no fabricated findings; it only requests info to run a query",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output suggests continuing investigation with other approaches if no findings exist",
            "passed": false,
            "evidence": "output only proposes running the same CloudWatch Logs Insights query once info is provided; does not suggest alternative investigation approaches if no findings exist",
            "method": "llm",
            "confidence": 0.85,
            "uncertain": false
          }
        ],
        "pass_rate": 0.3333333333333333,
        "summary": "With skill: 33% assertions passed",
        "execution_metrics": {
          "tool_calls": 3,
          "token_counts": {
            "input_tokens": 19,
            "output_tokens": 1420,
            "cache_read_input_tokens": 164497,
            "cache_creation_input_tokens": 69767
          }
        },
        "timing": {
          "elapsed_seconds": 38.47547720900184
        },
        "raw_output": "I need a few details before I can query CloudWatch Logs for AWS Security Agent findings on your payments service.The user declined to answer. I'll need that information to proceed \u2014 without it I can't safely query CloudWatch.\n\nTo run this check I need three pieces of info:\n\n1. **AWS account ID** (or confirmation to use the current profile) \u2014 required for any CloudWatch query.\n2. **Region** (airport code like `iad`, `pdx`, `dub`) \u2014 log groups are regional.\n3. **Log group name** \u2014 typically `/aws/securityagent/*`. I can auto-discover it once I have account + region.\n\nReply with those (e.g. *\"account 123456789012, pdx, auto-discover\"*) and I'll run a Logs Insights query for critical findings on the payments service over the last 24 hours, then surface file/line/severity details for anything that comes back."
      },
      "without_skill": {
        "eval_id": "no-findings-scenario",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output indicates whether findings were found or not",
            "passed": false,
            "evidence": "output does not indicate whether findings were found or not; instead asks clarifying questions before attempting to look",
            "method": "llm",
            "confidence": 0.85,
            "uncertain": false
          },
          {
            "text": "The output does not fabricate security findings that do not exist",
            "passed": true,
            "evidence": "no fabricated findings; output explicitly declines to invent results and asks for clarification",
            "method": "llm",
            "confidence": 0.98,
            "uncertain": false
          },
          {
            "text": "The output suggests continuing investigation with other approaches if no findings exist",
            "passed": true,
            "evidence": "output suggests alternative systems (GuardDuty, Security Hub, Inspector, internal services) and offers to pull findings once clarified, implying continued investigation paths",
            "method": "llm",
            "confidence": 0.7,
            "uncertain": false
          }
        ],
        "pass_rate": 0.6666666666666666,
        "summary": "Without skill: 67% assertions passed",
        "execution_metrics": {
          "tool_calls": 1,
          "token_counts": {
            "input_tokens": 12,
            "output_tokens": 478,
            "cache_read_input_tokens": 42543,
            "cache_creation_input_tokens": 56981
          }
        },
        "timing": {
          "elapsed_seconds": 27.095395667001867
        },
        "raw_output": "I don't have a tool called \"AWS Security Agent\" available in this environment. Before I go look for one \u2014 a couple of clarifications would help me route this correctly.\n\nCould you tell me:\n1. **What system holds those logs?** \"AWS Security Agent\" isn't a name I recognize as a standard tool. Possibilities: AWS GuardDuty, AWS Security Hub, Amazon Inspector, an internal Amazon security service (e.g., Shepherd, Mirador, SAS), or a homegrown agent specific to your team.\n2. **What identifies the payments service?** A service name in RIP, a pipeline name, a Brazil package, an AWS account ID, or a CloudWatch log group?\n\nIf you can point me to the system + the service identifier, I can pull the findings. If \"AWS Security Agent\" is shorthand for something specific in your team, just let me know what it maps to."
      },
      "delta_pass_rate": -0.3333333333333333
    },
    {
      "eval_id": "log-group-discovery",
      "run_index": 0,
      "with_skill": {
        "eval_id": "log-group-discovery",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output searches for log groups with patterns like /aws/securityagent/",
            "passed": false,
            "evidence": "LLM grading failed (rc=1): Error getting AWS credentials from awsCredentialExport (in settings or ~/.claude.json): awsCredentialExport did not return a valid value\n",
            "method": "llm",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "The output uses DescribeLogGroups or similar API to discover log groups",
            "passed": false,
            "evidence": "LLM grading failed (rc=1): Error getting AWS credentials from awsCredentialExport (in settings or ~/.claude.json): awsCredentialExport did not return a valid value\n",
            "method": "llm",
            "confidence": 1.0,
            "uncertain": false
          },
          {
            "text": "The output identifies a specific log group or explains how to locate it",
            "passed": false,
            "evidence": "LLM grading failed (rc=1): Error getting AWS credentials from awsCredentialExport (in settings or ~/.claude.json): awsCredentialExport did not return a valid value\n",
            "method": "llm",
            "confidence": 1.0,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "With skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.10490408300029
        },
        "raw_output": ""
      },
      "without_skill": {
        "eval_id": "log-group-discovery",
        "run_index": 0,
        "assertion_results": [
          {
            "text": "The output searches for log groups with patterns like /aws/securityagent/",
            "passed": false,
            "evidence": "output is empty, no search performed",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output uses DescribeLogGroups or similar API to discover log groups",
            "passed": false,
            "evidence": "output is empty, no API mentioned",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          },
          {
            "text": "The output identifies a specific log group or explains how to locate it",
            "passed": false,
            "evidence": "output is empty, no log group identified",
            "method": "llm",
            "confidence": 0.95,
            "uncertain": false
          }
        ],
        "pass_rate": 0.0,
        "summary": "Without skill: 0% assertions passed",
        "execution_metrics": {
          "tool_calls": 0,
          "token_counts": {
            "input_tokens": 0,
            "output_tokens": 0,
            "cache_read_input_tokens": 0,
            "cache_creation_input_tokens": 0
          }
        },
        "timing": {
          "elapsed_seconds": 120.04507625000042
        },
        "raw_output": ""
      },
      "delta_pass_rate": 0.0
    }
  ],
  "run_summary": {
    "with_skill": {
      "mean_pass_rate": 0.3333,
      "stddev_pass_rate": 0.4082,
      "mean_tokens": 1549.8,
      "mean_input_tokens": 17.8,
      "mean_output_tokens": 1549.8,
      "mean_total_tokens": 1567.6,
      "mean_tool_calls": 4.8
    },
    "without_skill": {
      "mean_pass_rate": 0.2667,
      "stddev_pass_rate": 0.2789,
      "mean_tokens": 739.0,
      "mean_input_tokens": 8.4,
      "mean_output_tokens": 739.0,
      "mean_total_tokens": 747.4,
      "mean_tool_calls": 0.6
    },
    "delta": {
      "pass_rate": 0.0667,
      "tokens": 810.8,
      "total_tokens": 820.2,
      "input_tokens": 9.4,
      "tool_calls": 4.2
    },
    "cost_efficiency": {
      "quality_delta": 0.0667,
      "cost_delta_pct": 109.7,
      "classification": "TRADEOFF",
      "emoji": "\ud83d\udfe1",
      "description": "Skill improves quality but increases cost"
    },
    "estimated_cost": {
      "with_skill_per_run": {
        "input_cost": 5.3e-05,
        "output_cost": 0.023247,
        "total_cost": 0.0233,
        "model": "sonnet",
        "currency": "USD"
      },
      "without_skill_per_run": {
        "input_cost": 2.5e-05,
        "output_cost": 0.011085,
        "total_cost": 0.01111,
        "model": "sonnet",
        "currency": "USD"
      },
      "per_eval_pair": 0.03441,
      "total_runs": 5,
      "total_cost": 0.1721,
      "model": "sonnet",
      "currency": "USD"
    }
  },
  "scores": {
    "outcome": 0.3333,
    "process": 1.0,
    "style": 0.3333,
    "efficiency": 0.7468,
    "overall": 0.6034
  },
  "passed": true
}

evals.json 3.1 KB

[
  {
    "id": "query-security-agent-logs",
    "prompt": "My application is returning 500 errors but there were no recent deployments or infrastructure changes. Check AWS Security Agent findings for any vulnerabilities.",
    "expected_output": "The output queries CloudWatch Logs for AWS Security Agent findings and returns any vulnerabilities found, including file path, line number, and vulnerability type.",
    "assertions": [
      "The output attempts to query CloudWatch Logs for Security Agent findings",
      "The output looks for log groups matching /aws/securityagent/ or similar patterns",
      "The output does not produce an unhandled error or permission denied without explanation"
    ]
  },
  {
    "id": "extract-vulnerability-details",
    "prompt": "Show me the details of any SSRF or injection vulnerabilities found by AWS Security Agent in our application.",
    "expected_output": "The output extracts and presents vulnerability details including file path, line number, vulnerability type, severity, and remediation guidance.",
    "assertions": [
      "The output searches for SSRF or injection-related findings",
      "The output attempts to extract file path and line number from findings",
      "The output includes severity information or states it was not available"
    ]
  },
  {
    "id": "correlate-findings-with-incident",
    "prompt": "We have an outage on our checkout service starting at 2am UTC. Check if AWS Security Agent found any vulnerabilities that could explain this outage.",
    "expected_output": "The output correlates Security Agent findings with the incident timeframe and affected service, explaining how vulnerabilities may relate to the outage.",
    "assertions": [
      "The output attempts to correlate findings with the checkout service",
      "The output considers the timeframe of the incident",
      "The output explains how any found vulnerability could cause the observed symptoms"
    ]
  },
  {
    "id": "no-findings-scenario",
    "prompt": "Check AWS Security Agent logs for any critical vulnerabilities in our payments service in the last 24 hours.",
    "expected_output": "The output reports whether Security Agent findings exist for the payments service, and suggests alternative investigation paths if none are found.",
    "assertions": [
      "The output indicates whether findings were found or not",
      "The output does not fabricate security findings that do not exist",
      "The output suggests continuing investigation with other approaches if no findings exist"
    ]
  },
  {
    "id": "log-group-discovery",
    "prompt": "I need to check AWS Security Agent findings but I'm not sure which log group to look at. Help me find the right CloudWatch log group.",
    "expected_output": "The output searches for CloudWatch log groups matching Security Agent patterns and identifies the correct log group to query.",
    "assertions": [
      "The output searches for log groups with patterns like /aws/securityagent/",
      "The output uses DescribeLogGroups or similar API to discover log groups",
      "The output identifies a specific log group or explains how to locate it"
    ]
  }
]

eval_queries.json 570 B

[
  {
    "query": "How do I set up an S3 bucket lifecycle policy?",
    "should_trigger": false
  },
  {
    "query": "What are the best practices for DynamoDB table design?",
    "should_trigger": false
  },
  {
    "query": "Explain how to configure CloudFront distributions with custom origins.",
    "should_trigger": false
  },
  {
    "query": "How do I create a new IAM role for my Lambda function?",
    "should_trigger": false
  },
  {
    "query": "What are the differences between ECS and EKS for container orchestration?",
    "should_trigger": false
  }
]

report.json 1.6 KB

{
  "skill_name": "enrich-with-aws-security-agent",
  "skill_path": "/Users/yakiratz/git/sample-code-for-devops-agent-skills/skills/enrich-with-aws-security-agent",
  "timestamp": "2026-06-14T17:55:46Z",
  "overall_score": 0.8414,
  "overall_grade": "B",
  "passed": false,
  "sections": {
    "audit": {
      "score": 100,
      "grade": "A",
      "passed": true,
      "normalized": 1.0,
      "critical": 0,
      "warning": 0,
      "info": 0
    },
    "functional": {
      "overall": 0.6034,
      "grade": "D",
      "passed": false,
      "scores": {
        "outcome": 0.3333,
        "process": 1.0,
        "style": 0.3333,
        "efficiency": 0.7468,
        "overall": 0.6034
      },
      "cost_efficiency": {
        "quality_delta": 0.0667,
        "cost_delta_pct": 109.7,
        "classification": "TRADEOFF",
        "emoji": "\ud83d\udfe1",
        "description": "Skill improves quality but increases cost"
      },
      "estimated_cost": {
        "with_skill_per_run": {
          "input_cost": 5.3e-05,
          "output_cost": 0.023247,
          "total_cost": 0.0233,
          "model": "sonnet",
          "currency": "USD"
        },
        "without_skill_per_run": {
          "input_cost": 2.5e-05,
          "output_cost": 0.011085,
          "total_cost": 0.01111,
          "model": "sonnet",
          "currency": "USD"
        },
        "per_eval_pair": 0.03441,
        "total_runs": 5,
        "total_cost": 0.1721,
        "model": "sonnet",
        "currency": "USD"
      }
    },
    "trigger": {
      "pass_rate": 1.0,
      "grade": "A",
      "passed": true,
      "total_queries": 5
    }
  }
}

trigger_report.json 2.2 KB

{
  "skill_name": "enrich-with-aws-security-agent",
  "skill_path": "/Users/yakiratz/git/sample-code-for-devops-agent-skills/skills/enrich-with-aws-security-agent",
  "query_results": [
    {
      "query": "How do I set up an S3 bucket lifecycle policy?",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 4.0,
      "mean_output_tokens": 575.7,
      "mean_total_tokens": 579.7
    },
    {
      "query": "What are the best practices for DynamoDB table design?",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 6.0,
      "mean_output_tokens": 660.7,
      "mean_total_tokens": 666.7
    },
    {
      "query": "Explain how to configure CloudFront distributions with custom origins.",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 6.0,
      "mean_output_tokens": 1262.0,
      "mean_total_tokens": 1268.0
    },
    {
      "query": "How do I create a new IAM role for my Lambda function?",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 4.0,
      "mean_output_tokens": 568.7,
      "mean_total_tokens": 572.7
    },
    {
      "query": "What are the differences between ECS and EKS for container orchestration?",
      "should_trigger": false,
      "trigger_count": 0,
      "run_count": 3,
      "trigger_rate": 0.0,
      "passed": true,
      "mean_input_tokens": 4.0,
      "mean_output_tokens": 409.0,
      "mean_total_tokens": 413.0
    }
  ],
  "summary": {
    "total_queries": 5,
    "passed": 5,
    "failed": 0,
    "trigger_precision": 1.0,
    "no_trigger_precision": 1.0,
    "mean_total_tokens_per_run": 700.0,
    "estimated_cost": {
      "per_run": {
        "input_cost": 1.4e-05,
        "output_cost": 0.010428,
        "total_cost": 0.010443,
        "model": "sonnet",
        "currency": "USD"
      },
      "total_runs": 5,
      "total_cost": 0.0522,
      "model": "sonnet",
      "currency": "USD"
    }
  },
  "passed": true
}

.skilleval.yaml 77 B

audit:
  ignore:
    - STR-016    # README alongside SKILL.md is intentional

CHANGELOG.md 41 B

# Changelog

## 1.0.0

- Initial version

README.md 3.9 KB

# Enrich with AWS Security Agent

This skill enables the AWS DevOps Agent to query AWS Security Agent CloudWatch logs during incident investigations, surfacing code-level security findings that may be the root cause of application outages.

## ⚠️ Important Notice

This skill is provided as sample code. If you intend to deploy it in production, start with a non-production environment first. Review IAM permissions and security configurations against your organization's policies, and validate that skill behavior meets your operational requirements before production use.

## Purpose

When an application outage or degradation occurs without a clear infrastructure cause, the root cause may be a security vulnerability — such as SSRF, injection attacks, or compromised dependencies. This skill queries AWS Security Agent findings stored in CloudWatch Logs to retrieve actionable details (file path, line number, vulnerability type, severity) that customers can directly fix.

## Key Capabilities

- Query AWS Security Agent CloudWatch log groups for security findings
- Extract actionable vulnerability details: file path, line number, vulnerability type, severity
- Correlate security findings with current incident symptoms and timeframes
- Present remediation guidance with exact code locations
- Automatically activate when application errors have no obvious infrastructure cause

## Prerequisites

- AWS Security Agent configured and writing findings to CloudWatch Logs
- Agent permissions for `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults` on Security Agent log groups (e.g., `/aws/securityagent/*`)

## Limitations

- Requires AWS Security Agent to be configured and actively scanning — no findings will exist if the service is not enabled
- Log retention depends on CloudWatch Logs retention settings for the Security Agent log groups
- Findings are only as current as the last Security Agent scan

## Agent Types

This skill is used by the following agent types:

- **Chat tasks** — conversational security finding lookup and analysis
- **Incident RCA** — automated root cause analysis when security issues may cause outages

## Uploading to AWS DevOps Agent

To deploy this skill to your Agent Space:

1. Zip the `enrich-with-aws-security-agent/` directory (only including allowed extensions):

   ```bash
   cd skills
   zip -r enrich-with-aws-security-agent.zip enrich-with-aws-security-agent/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'
   ```

2. In the AWS DevOps Agent Operator Web App, navigate to the **Skills** page.
3. Click **Add skill** → **Upload skill**.
4. Drag and drop the `enrich-with-aws-security-agent.zip` file (max 6 MB).
5. Select the agent types: **Chat tasks** and **Incident RCA**.
6. Click **Upload**.

For more details, see [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#creating-skills) in the AWS DevOps Agent User Guide.

## How to Use This Skill

### Chat

- "Check AWS Security Agent findings for my application — are there any critical vulnerabilities?"
- "Show me security findings related to SSRF or injection attacks in the last 7 days."
- "What security vulnerabilities has AWS Security Agent found in the orders service?"
- "Are there any hardcoded secrets or credential exposure findings?"

### Investigation

- "My application is returning 500 errors but there were no recent deployments. Investigate."
- "We're seeing unexplained outages on the checkout service — could it be a security issue?"
- "The API started failing with authentication errors. Check if there are security findings related to this."
- "Our web application is behaving unexpectedly and we suspect injection attacks. Investigate."

SKILL.md 4.3 KB

---
name: enrich-with-aws-security-agent
description: Automatically load this skill when investigating application outages, service degradation, or errors that could have security-related root causes — including unexplained downtime, authentication or authorization failures, injection attacks, data exposure, or suspicious application behavior. Query AWS Security Agent CloudWatch logs to retrieve detailed code review findings with actionable, low-level details (file, line number, vulnerability type) that customers can directly fix.
metadata:
  author: yakiratz-aws
  version: "1.0.0"
  aws-devops-agent-skills.agent-types: "Chat tasks, Incident RCA"
  aws-devops-agent-skills.aws-services: "Amazon CloudWatch, AWS Security Agent"
  aws-devops-agent-skills.technical-domains: "Security"
---

# Enriching Investigations with AWS Security Agent Findings

## When to Use — AUTOMATIC ACTIVATION

**Load this skill automatically when ANY of these conditions apply:**

### Application-Level Symptoms (even without explicit security context)
- Application pages or endpoints are not loading or returning errors
- Unexplained service outages with no recent deployments
- Application behaving unexpectedly without clear infrastructure cause
- Services returning 4xx/5xx errors without obvious resource exhaustion

### Explicit Security Signals
- Security vulnerabilities or misconfigurations mentioned
- Suspicious code patterns or behaviors observed
- Authentication or authorization failures
- Injection vulnerabilities suspected (SQL, command, SSRF, etc.)
- Sensitive data exposure concerns
- Security-related errors in application logs
- Findings that suggest code-level security issues

### Key Insight
Many security issues (like SSRF, injection attacks, or compromised dependencies) manifest as **application outages or errors** rather than obvious security alerts. When an app stops working and there's no clear infrastructure cause (no deployments, no resource exhaustion, no config changes), **always check for security issues**.

## Overview

AWS Security Agent is an AI-powered security service that performs automated code security reviews and penetration testing. Its findings are stored in CloudWatch Logs and provide detailed, actionable information about vulnerabilities including exact file locations, line numbers, and remediation guidance.

## Steps

1. **Identify the relevant CloudWatch log group** for AWS Security Agent findings. Look for log groups with patterns like:
   - `/aws/securityagent/*`
   - Log groups containing "security-agent" or "securityagent" in the name

2. **Query the Security Agent logs** using CloudWatch Logs Insights:
   ```
   fields @timestamp, @message
   | filter @message like /finding|vulnerability|issue|risk/
   | sort @timestamp desc
   | limit 100
   ```

3. **Extract actionable details** from the findings:
   - **File path**: The exact source file containing the vulnerability
   - **Line number**: The specific line(s) of code affected
   - **Vulnerability type**: Classification (e.g., SQL injection, XSS, SSRF, hardcoded secrets)
   - **Severity**: Critical, High, Medium, Low
   - **Description**: What the vulnerability is and why it's a risk
   - **Remediation**: Suggested fix or mitigation

4. **Correlate findings with the incident** by matching:
   - Affected services or components
   - Timeframes (when the vulnerability was introduced vs. when the incident occurred)
   - Code paths involved in the failing functionality

5. **Present findings to the customer** with:
   - Clear identification of the vulnerable code location
   - Explanation of how this vulnerability may relate to the incident
   - Specific remediation steps they can take to fix the code

## Example Log Query for Specific Application

If investigating a specific application (e.g., "ssrf-demo", "orders", "account"):
```
fields @timestamp, @message
| filter @message like /ssrf|orders|account|vulnerability|finding|issue/
| sort @timestamp desc
| limit 50
```

## Important Notes

- AWS Security Agent API access may require additional IAM permissions. If API calls fail, fall back to the CloudWatch logs approach.
- Always provide the exact file and line number when available — this is the key value for customers.
- Link the security finding to the observed incident behavior when possible.
- **When in doubt, check security logs** — it's better to rule out security issues than to miss them.