Claude Skill

msk-operations

Amazon MSK Provisioned operations, troubleshooting, and health

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Download aws-tools-for-devops-agent-skills_msk-operations-d8f96e1.zip · 47 KB

Install

skills CLI npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/msk-operations
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart
Git git clone https://github.com/aws/tools-for-devops-agent.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.

README

MSK Operations — AWS DevOps Agent Skill

An Amazon MSK Provisioned operations, troubleshooting, and health-assessment skill for AWS DevOps Agent. Covers Standard and Express brokers and both proactive operational reviews and ad-hoc incident response (performance, consumer lag, storage, maintenance, client tuning).

⚠️ Non-production disclaimer. This skill is sample code, not intended for production use without additional review and testing. Users should validate in a non-production environment first.

Purpose

Give AWS DevOps Agent the domain knowledge to answer MSK questions accurately without falling back on training data (which routinely conflates Standard and Express broker behavior). The skill activates in two shapes:

  1. Operational review — assess an MSK cluster's health against best practices and produce a prioritized findings report.
  2. Ad-hoc troubleshooting — investigate a specific MSK symptom (high CPU, consumer lag, disk full, TrafficShaping, unexpected broker restart, etc.) and recommend the correct next action.

Key Capabilities

  • Determine broker type (Standard vs Express) and route to the correct diagnostic path — many CloudWatch metrics and behaviors differ between the two.
  • Troubleshoot broker performance issues (CpuUser, CpuSystem, RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercent, ProduceTotalTimeMsMean, TrafficShaping).
  • Diagnose consumer lag using SumOffsetLag, MaxOffsetLag, and (when PER_TOPIC_PER_PARTITION is enabled) per-partition OffsetLag.
  • Manage broker storage — EBS scaling for Standard, StorageUsed monitoring for Express, tiered storage, retention planning.
  • Recommend CloudWatch alarms in the AWS/Kafka namespace and validate that the right monitoring level is enabled.
  • Explain rolling restart, patching, and version upgrade behavior; call out operations that are unsafe during UnderReplicatedPartitions > 0.
  • Advise on Kafka client (producer / consumer) configuration — batch sizing, linger.ms, compression, acks, min.insync.replicas — and on authentication choices (IAM, SCRAM, mTLS).

Prerequisites

IAM permissions

The AWS DevOps Agent's primary cloud-source role needs read access to MSK and CloudWatch. All calls except kafka:GetBootstrapBrokers are covered by AIDevOpsAgentAccessPolicy. kafka:GetBootstrapBrokers is granted by the opt-in EnableMskOperations parameter (default true) in cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml. The full set of actions the skill uses in practice:

kafka:DescribeClusterV2
kafka:ListClustersV2
kafka:ListNodes
kafka:GetBootstrapBrokers
kafka:ListClusterOperationsV2
kafka:DescribeConfigurationRevision
kafka:ListConfigurations
cloudwatch:GetMetricData
cloudwatch:GetMetricStatistics
cloudwatch:ListMetrics
cloudwatch:DescribeAlarms
logs:DescribeLogGroups
logs:FilterLogEvents

Write actions (UpdateBrokerStorage, CreateConfiguration, PutMetricAlarm, etc.) are only recommended in the skill output — the operator is expected to run them after review.

Cluster access

  • The MSK cluster must live in an account configured as a cloud source in your Agent Space.
  • No data-plane / Kafka-protocol access is required. Everything the skill uses comes from the AWS control-plane APIs and CloudWatch.

Limitations

  • Covers MSK Provisioned only — does not cover MSK Connect, MSK Serverless, or MSK Replicator.
  • No Kafka data-plane visibility. The skill cannot read topic contents, run kafka-consumer-groups.sh, or otherwise interact with the Kafka protocol.
  • Some checks require monitoring level PER_BROKER or higher (thread pool idle %, VolumeQueueLength, BwInAllowanceExceeded, IAM connection metrics). At DEFAULT level the skill will explicitly note which checks are limited.
  • Sizing questions (broker count, instance type choice, monthly cost) are redirected to the AWS documentation — the skill does not include a bundled sizing calculator.

Agent Types

This skill is intended for the following agent types (selected in the Operator Web App at upload time):

  • Chat tasks — conversational invocation in Chat ("my MSK cluster is latent", "run an MSK health check on prod-cluster", "why did broker 2 restart last night?").

Select All agents instead if you want the skill available to all agent types.

Uploading to AWS DevOps Agent

Reference: Uploading a skill

1. Package the skill

From the skills/ directory in this repo:

cd skills
zip -r msk-operations.zip msk-operations/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'

The resulting msk-operations.zip contains:

msk-operations/
├── SKILL.md
└── references/
    ├── troubleshoot-performance.md
    ├── troubleshoot-consumer-lag.md
    ├── manage-storage.md
    ├── monitor-and-alarm.md
    ├── maintenance-operations.md
    └── configure-clients.md

Constraints (enforced at upload time):

  • Total zip size ≤ 6 MB.
  • SKILL.md is required and must include name and description frontmatter.
  • A scripts/ directory is not allowed — uploads containing scripts are rejected.

2. Upload via the Operator Web App

  1. Navigate to the Skills page in your Agent Space Operator Web App.
  2. Click Add skill → Upload skill.
  3. Drag and drop msk-operations.zip (or browse to it).
  4. Select agent types: Chat tasks (or leave All agents to make it available to all agent types).
  5. Review the validation results.
  6. Click Upload.

How to Use This Skill

In DevOps Agent Chat, use natural language. You do NOT need to mention the skill name — the agent activates it based on the MSK / Kafka triggers in the skill description.

Ad-hoc troubleshooting prompts

  • "My MSK cluster prod-orders is showing high produce latency in us-east-1. What should I check?"
  • "Consumer group payments-consumer on cluster prod-orders has growing lag. Diagnose it."
  • "UnderReplicatedPartitions is > 0 on broker 2 of prod-orders. What's the safe next step?"
  • "Why did broker 3 of prod-orders restart last night?"
  • "Storage on prod-orders is at 82%. What are my options?"
  • "TrafficShaping is firing on all three brokers of prod-orders."

Operational-review prompts

  • "Run an MSK operational review on cluster prod-orders."
  • "Audit prod-orders against MSK best practices."
  • "Are the CloudWatch alarms on prod-orders complete? Which ones am I missing?"
  • "Health-check every MSK Provisioned cluster in us-east-1."

Client tuning prompts

  • "My Java Kafka producer is getting NotEnoughReplicasException when writing to MSK. What producer settings should I check?"
  • "What should linger.ms and batch.size be for a producer writing 200 MB/s to a Standard broker cluster?"

Skill Contents

msk-operations/
├── SKILL.md
├── README.md
├── CHANGELOG.md
├── .skilleval.yaml
├── evals/
│   ├── evals.json
│   └── eval_queries.json
└── references/
    ├── troubleshoot-performance.md
    ├── troubleshoot-consumer-lag.md
    ├── manage-storage.md
    ├── monitor-and-alarm.md
    ├── maintenance-operations.md
    └── configure-clients.md

README.md, CHANGELOG.md, .skilleval.yaml, and evals/ are for repo maintenance and are not required at the cluster — the zip command above excludes them from the upload artifact.

Skill manifest

Amazon MSK Operations

Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots).

When to Use

Activate this skill when the user asks to:

  • Review, audit, or assess an MSK cluster for best practices, health, or operational readiness.
  • Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency, consumer lag, broker storage running out, TrafficShaping events, under-replicated partitions, or an unexpected broker restart.
  • Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch alarms and dashboards, understand the metrics available in the AWS/Kafka namespace.
  • Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security patching, broker instance type change.
  • Advise on Kafka client (producer / consumer) configuration when the client is connecting to an MSK cluster.

Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces.

Broker Type Determination

Determine the broker type first — many checks differ between Standard and Express.

aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Check ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType:

  • Starts with kafka. (e.g. kafka.m5.large, kafka.m7g.xlarge) → Standard broker.
  • Starts with express. (e.g. express.m7g.large) → Express broker.

Key Standard vs Express differences

Standard brokers use customer-managed EBS volumes for storage. You choose instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage scaling. Standard brokers have scheduled maintenance windows.

Express brokers provide fully managed, pay-as-you-go storage with no EBS provisioning. Instance types are prefixed with express.m7g.*. Express brokers offer up to 3× more throughput per broker than Standard, and have no maintenance windows. Express enforces a fixed replication factor of 3 and min.insync.replicas=2 — you cannot create topics with RF=1.

Critical Warnings

  • This skill is read-only. Every command in this file and in references/ that mutates cluster state — update-broker-storage, create-configuration, update-cluster-configuration, update-monitoring, put-metric-alarm, reboot-broker, and any partition reassignment — is a recommendation for the operator to run after review. Present these as proposed remediations with expected impact and preconditions; do NOT execute them, and do NOT imply that the agent will run them.
  • NEVER reboot brokers while UnderReplicatedPartitions > 0 (Standard only — Express brokers do not emit URP). This risks data loss and extended outages.
  • NEVER recommend partition reassignment without first checking replication status. Reassignment during URP compounds the problem.
  • linger.ms=0 is the #1 cause of "high CPU" on MSK. ALWAYS check client batch configuration before recommending broker scaling.
  • EBS throughput ceilings are invisible in Kafka metrics — ALWAYS check EBS volume metrics (VolumeReadBytes, VolumeWriteBytes, VolumeQueueLength) when diagnosing Standard broker latency.
  • Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned EBS throughput for Express clusters.
  • Express brokers enforce fixed RF=3 and min.insync.replicas=2 — do NOT attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard brokers.

Quick Diagnostics

These five checks cover the most common MSK issues. Use them before loading a reference file.

  1. CpuUser + CpuSystem > 60%: Check RequestHandlerAvgIdlePercent (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check client batch.size and linger.ms before recommending scaling.

  2. KafkaDataLogsDiskUsed > 85% (Standard only): Recommend to the operator that EBS be expanded via aws kafka update-broker-storage (do not execute). Identify high-growth topics via per-topic BytesInPerSec to size the increase. Express clusters use StorageUsed metric instead and storage is fully managed.

  3. UnderReplicatedPartitions > 0 (Standard only): Check if a maintenance operation or broker restart is in progress. If URP is decreasing, wait for recovery. Do NOT restart brokers or reassign partitions during URP. Express brokers do not emit this metric — monitor ProduceThrottleTime, FetchThrottleTime, and consumer lag instead.

  4. Consumer OffsetLag / MaxOffsetLag increasing: Determine if broker-side (high ProduceTotalTimeMsMean, CPU saturation) or client-side (slow processing, insufficient consumers). Per-partition lag from PER_TOPIC_PER_PARTITION monitoring level helps isolate hot partitions.

  5. BytesInPerSec near throughput ceiling: For Standard, check EBS volume type and calculate: BytesInPerSec × ReplicationFactor vs volume throughput limit. For Express, check against the per-broker sustained performance limits in the MSK quotas.

Which Reference Do You Need?

Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory.

Customer Intent Reference
High CPU, high produce/fetch latency, slow cluster, TrafficShaping references/troubleshoot-performance.md
Consumer lag increasing, rebalance storms, stuck consumer groups references/troubleshoot-consumer-lag.md
Disk filling up, retention planning, tiered storage, EBS scaling references/manage-storage.md
Setting up monitoring level, dashboards, recommended CloudWatch alarms references/monitor-and-alarm.md
Rolling restart impact, patching, Kafka version upgrades, maintenance resilience references/maintenance-operations.md
Producer / consumer configuration, IAM / SCRAM / TLS auth for clients references/configure-clients.md

For sizing questions (broker count, instance type choice, monthly cost), refer the user to the Amazon MSK best practices — right-size your cluster documentation. Do not size from memory.

Operational Review Workflow

Use this workflow when the user asks for a review, audit, health check, or assessment of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report.

Follow the steps in order for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check.

Step 1 — Identify Target Clusters

Ask the user which MSK clusters to review. Accept any of:

  • Specific cluster names or ARNs and regions
  • "all clusters" in specific regions
  • "all MSK clusters in all regions"

If no scope is given, default to all configured account regions. Enumerate clusters with aws kafka list-clusters-v2 per region.

Step 2 — Determine Broker Type Per Cluster

For each cluster:

aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard brokers (kafka.*) and Express brokers (express.*) require different checks in the later steps — some metrics only exist on one type.

Step 3 — Collect Cluster Configuration

For each cluster, gather:

aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn>   # last 30 days
aws kafka describe-configuration-revision \
     --arn <configuration-arn> --revision <revision>       # if a custom config is applied

Capture:

  • Cluster: state, Kafka version, number of broker nodes, AZ distribution (ZoneIds), storage mode (EBS / Tiered), current version.
  • Broker: instance type, EBS volume size (Standard), provisioned throughput (if any).
  • Encryption: EncryptionInTransit.ClientBroker (TLS / TLS_PLAINTEXT / PLAINTEXT), EncryptionInTransit.InCluster, EncryptionAtRest.DataVolumeKMSKeyId.
  • Auth: ClientAuthentication.Sasl.Iam.Enabled, ClientAuthentication.Sasl.Scram.Enabled, ClientAuthentication.Tls.Enabled, ClientAuthentication.Unauthenticated.Enabled.
  • Public access: BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type.
  • Monitoring level: EnhancedMonitoring (DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION).
  • Logging: LoggingInfo.BrokerLogs (CloudWatch / S3 / Firehose destinations and their Enabled flags).
  • Open monitoring: OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker, NodeExporter.EnabledInBroker.
  • Recent operations: From list-cluster-operations-v2, note any SECURITY_PATCHING, BROKER_UPDATE, UPDATE_CLUSTER_CONFIGURATION, UPDATE_STORAGE, or UPDATE_MONITORING events in the review window.

Step 4 — Detect Monitoring Level and Gaps

The EnhancedMonitoring value from Step 3 determines which checks are available. At DEFAULT, most per-broker health metrics are still available (CPU, disk, network, partitions, connections, memory, TrafficShaping), but the following checks are not possible without upgrading:

  • ReplicationBytesInPerSec / ReplicationBytesOutPerSec (inter-broker replication load)
  • RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent (thread pool saturation)
  • BwInAllowanceExceeded / BwOutAllowanceExceeded (detailed bandwidth breaches)
  • VolumeQueueLength (EBS I/O queue depth — Standard only)
  • VolumeReadBytes / VolumeWriteBytes (EBS throughput utilization — Standard only)
  • IAM connection metrics (IAMTooManyConnections)

Record the monitoring level and list any dimensions that will be scored partially or skipped. Recommend upgrading to PER_BROKER if any dimension is degraded by the current level.

Step 5 — Collect CloudWatch Metrics (7-Day Historical)

Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for per-broker metrics; Cluster Name and Consumer Group and Topic for consumer lag; Cluster Name only for cluster-wide metrics.

Use one cloudwatch.GetMetricData batch per cluster where possible. Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.

5.1 Cluster-wide (both broker types)

Metric Stat Purpose
ActiveControllerCount Sum Must be exactly 1
OfflinePartitionsCount Maximum Must be 0
GlobalPartitionCount Maximum Total leader partitions
GlobalTopicCount Maximum Total topics

5.2 Per-broker — Standard (kafka.*)

Metric Stat Threshold
CpuUser + CpuSystem Average, Maximum < 60% avg
KafkaDataLogsDiskUsed Average, Maximum < 70% avg, < 85% max
PartitionCount Maximum ≤ recommended limit for broker size
LeaderCount Maximum Compare across brokers; skew < 10%
UnderReplicatedPartitions Maximum 0 in steady state
UnderMinIsrPartitionCount Maximum 0
BytesInPerSec / BytesOutPerSec Average, Maximum vs baseline bandwidth
ConnectionCount Average, Maximum Compare across brokers
HeapMemoryAfterGC Maximum < 60%
TrafficShaping Sum Must be 0
NetworkRxDropped / NetworkTxDropped / NetworkRxErrors / NetworkTxErrors Sum Must be 0
ReplicationBytesInPerSec / ReplicationBytesOutPerSec Average Requires PER_BROKER
RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent Average > 30% (PER_BROKER)
VolumeQueueLength Average, Maximum Avg < 1 (PER_BROKER)
VolumeReadBytes + VolumeWriteBytes Sum vs EBS baseline throughput (PER_BROKER)

5.3 Per-broker — Express (express.*)

Metric Stat Threshold
CpuUser + CpuSystem Average, Maximum < 60% avg
StorageUsed Maximum Fully managed — flag if trending against per-broker quota
PartitionCount Maximum ≤ recommended limit for broker size
LeaderCount Maximum Compare across brokers
BytesInPerSec / BytesOutPerSec Average, Maximum vs Express per-broker ingress/egress quotas
ProduceThrottleTime / FetchThrottleTime Maximum Should be 0
ClientConnectionCount Average, Maximum vs listener quota (see references/monitor-and-alarm.md)

Express brokers do NOT emit: UnderReplicatedPartitions, UnderMinIsrPartitionCount, HeapMemoryAfterGC, TrafficShaping, Volume* metrics, KafkaDataLogsDiskUsed, ProduceMessageConversionsPerSec, FetchMessageConversionsPerSec.

5.4 Consumer lag (both broker types)

Per consumer group (identified from the customer or from the broker logs):

Metric Stat Notes
SumOffsetLag per (Consumer Group, Topic) Maximum DEFAULT level
MaxOffsetLag per (Consumer Group, Topic) Maximum DEFAULT level
EstimatedMaxTimeLag per (Consumer Group, Topic) Maximum DEFAULT level
OffsetLag per (Consumer Group, Topic, Partition) Maximum Requires PER_TOPIC_PER_PARTITION

Step 6 — Alarm Coverage

Collect existing alarms:

aws cloudwatch describe-alarms --namespace AWS/Kafka

Compare against the 13-alarm recommended set (details in references/monitor-and-alarm.md) and produce a coverage table (present / missing / firing).

Step 7 — Analyze Against Best Practices

Assign a severity to every finding: CRITICAL / HIGH / MEDIUM / LOW / INFO (see Severity Definitions later in this file).

Evaluate across seven dimensions. Some checks are skipped for Express — noted inline.

7.1 Cluster Configuration

  • Cluster state is ACTIVE. MAINTENANCE / UPDATING is transient; anything else is a finding.
  • Deployed across 3 AZs (Standard: broker count multiple of AZ count).
  • Kafka version within N-2 of the latest supported.
  • Enhanced monitoring at PER_BROKER or higher — DEFAULT → MEDIUM.
  • Storage mode: Tiered storage enabled for topics with long retention (Standard).

7.2 Security

  • Encryption in transit: TLS. TLS_PLAINTEXT → MEDIUM (prod: HIGH). PLAINTEXT → CRITICAL.
  • Encryption at rest: enabled with KMS. Customer-managed KMS preferred over AWS-managed for regulated workloads.
  • Authentication: at least one of IAM / SCRAM / mTLS enabled. Unauthenticated.Enabled=true in production → CRITICAL.
  • Public access: SERVICE_PROVIDED_EIPS in production → HIGH.

7.3 Logging & Monitoring

  • Broker logs enabled to at least one destination (CloudWatch / S3 / Firehose). All disabled → HIGH.
  • Open monitoring (Prometheus JMX + Node exporter) — informational.
  • Alarm coverage from Step 6: missing alarms → MEDIUM each; missing critical alarms (Active Controller, Offline Partitions, Disk, CPU) → HIGH.
  • Any alarm currently in ALARM state → HIGH (surface in report header).

7.4 Partition Health

  • PartitionCount per broker ≤ recommended limit for the broker instance type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks update operations).
  • LeaderCount variance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH.
  • UnderReplicatedPartitions > 0 sustained (Standard only) → HIGH. Transient during a SECURITY_PATCHING / BROKER_UPDATE operation from Step 3 → INFO — do NOT flag.
  • UnderMinIsrPartitionCount > 0 (Standard only) → CRITICAL.
  • Config-level ISR risk: topics with min.insync.replicas >= replication.factor are a configuration error the CloudWatch metric will not surface. If the user has provided topic configs, flag any such topic as HIGH.

7.5 Compute Health

  • CpuUser + CpuSystem avg < 60%. 60-70% → MEDIUM. > 70% → HIGH.
  • CPU variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
  • HeapMemoryAfterGC < 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH.
  • RequestHandlerAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.
  • NetworkProcessorAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.

7.6 Network Health

  • TrafficShaping = 0 (Standard only). Any non-zero → HIGH.
  • BwInAllowanceExceeded / BwOutAllowanceExceeded = 0 (PER_BROKER). Non-zero → HIGH.
  • Throughput variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
  • Total per-broker throughput < 60% of baseline bandwidth (see references/troubleshoot-performance.md for baseline table). 60-70% → MEDIUM. > 70% → HIGH.
  • Express: ProduceThrottleTime / FetchThrottleTime > 0 → HIGH.

7.7 Storage Health

Standard only (Express storage is managed — Express clusters only get the StorageUsed check):

  • KafkaDataLogsDiskUsed < 70%. 70-85% → HIGH. > 85% → CRITICAL.
  • VolumeQueueLength avg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH.
  • EBS auto-scaling configured, or disk headroom > 30% → PASS. No auto-scaling AND disk > 50% → MEDIUM. No auto-scaling AND disk > 70% → HIGH.
  • EBS throughput (VolumeReadBytes + VolumeWriteBytes) < 60% of instance baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH.

Express:

  • StorageUsed per broker vs Express per-broker storage quota (see MSK Express quotas). Approaching quota → MEDIUM.

Step 8 — Generate the Report

Generate a separate report artifact per cluster reviewed.

Artifact naming: msk-review-<cluster-name>-<YYYY-MM-DD>.md Example: msk-review-prod-orders-2026-04-29.md

Report structure:

Report Header

# MSK Operational Review — <cluster-name>
Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD>
Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n>
Kafka Version: <version> | Monitoring Level: <level>

Executive Summary

  • Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL
  • Finding counts by severity
  • Top 3 CRITICAL/HIGH items

Configuration Snapshot

| Item | Value | | Cluster state / version | … | | Broker type / instance / count / AZs | … | | Storage | mode, size (Standard), provisioned throughput (if any) | | Encryption | in-transit, in-cluster, at-rest (KMS) | | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags | | Public access | DISABLED / SERVICE_PROVIDED_EIPS | | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION | | Logging | destinations enabled |

Findings by Dimension

For each of the 7 dimensions (7.1-7.7):

| # | Finding | Severity | Current State | Recommendation |

If a dimension was skipped or partial due to monitoring level or broker type, say so explicitly in a note above the table for that dimension.

CloudWatch Metrics (7-Day)

| Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding |

Alarm Coverage

| # | Recommended Alarm | Metric | Threshold | Priority | Status |

Also list alarms currently in ALARM state with timestamps.

Recent Cluster Operations (Last 30 Days)

From list-cluster-operations-v2 in Step 3:

| Operation Type | Start Time | End Time | State |

Call out any operations that would explain transient metric anomalies.

Priority Matrix

| # | Finding | Severity | Dimension | Effort | Impact |

Sorted by severity.

Next Steps

  • Immediate (CRITICAL / HIGH — 7 days)
  • Short-term (MEDIUM — 30 days)
  • Long-term (LOW — 90 days)

Appendix — Reference Links

Common CLI Recipes

Describe cluster:

aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>

List brokers:

aws kafka list-nodes --cluster-arn <cluster-arn>

Get bootstrap brokers:

aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn>

List recent cluster operations (patching, config updates, storage changes):

aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn>

Expand Standard broker storage — Operator-run (recommend, do not execute):

aws kafka update-broker-storage \
  --cluster-arn <cluster-arn> \
  --current-version <cluster-version> \
  --target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]'

Get a CloudWatch metric (example: CpuUser per broker):

aws cloudwatch get-metric-statistics \
  --namespace AWS/Kafka \
  --metric-name CpuUser \
  --dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \
  --start-time <start> --end-time <end> --period 300 --statistics Average

Create a cluster configuration (server.properties) — Operator-run (recommend, do not execute):

The --server-properties argument MUST be a real Kafka properties file with one key=value per line, separated by actual newline characters — NOT the literal two-character escape sequence \n. The MSK API accepts the bytes as-is; if you pass "k1=v1\nk2=v2" as a single string with escaped newlines, MSK stores ONE invalid property line and the cluster will fail to apply it.

Recommended pattern: write the properties to a local file with real newlines, then pass it via fileb:// so the CLI uploads the raw bytes verbatim. Verify by reading the revision back with describe-configuration-revision and base64-decoding ServerProperties — you should see one property per line.

cat > server.properties <<'EOF'
auto.create.topics.enable=false
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
num.io.threads=32
num.network.threads=16
log.retention.hours=168
EOF

aws kafka create-configuration \
  --name <config-name> \
  --kafka-versions "3.6.0" \
  --server-properties fileb://server.properties

Common Error Reference

Error Cause Fix
aws kafka update-broker-storage returns "storage is optimizing" Previous storage expansion still in cool-down (minimum 6 hours) Wait for optimization to complete. Check cluster state with describe-cluster-v2.
ClusterState is MAINTENANCE Standard brokers undergoing patching. Express brokers stay ACTIVE during maintenance. Wait for cluster to return to ACTIVE. Do not perform update operations during MAINTENANCE.
Consumer receives GROUP_COORDINATOR_NOT_AVAILABLE Coordinator broker is temporarily unavailable during rolling restart or overloaded Retry with backoff. Check if maintenance is in progress via list-cluster-operations-v2.
NotEnoughReplicasException on produce Fewer brokers in ISR than min.insync.replicas (default: 2) Check UnderReplicatedPartitions (Standard only). For Express, check ProduceThrottleTime and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower min.insync.replicas to work around this.

Severity Definitions (for review-style reports)

Severity Definition SLA
CRITICAL Immediate risk to availability, security, or data integrity Fix within 24–48 hours
HIGH Significant gap that could lead to incidents Fix within 1 week
MEDIUM Notable improvement opportunity Plan within 30 days
LOW Minor optimization or hardening Address when convenient
INFO Observation, no action required N/A

Additional Resources

Files (tools-for-devops-agent)
  • evals
    • benchmark.json 26.6 KB
      Loading…
    • evals.json 3.1 KB
      Loading…
    • eval_queries.json 1 KB
      Loading…
    • report.json 1.6 KB
      Loading…
    • trigger_report.json 3.4 KB
      Loading…
  • references
    • configure-clients.md 6.7 KB
      Loading…
    • maintenance-operations.md 12.3 KB
      Loading…
    • manage-storage.md 7.1 KB
      Loading…
    • monitor-and-alarm.md 13.3 KB
      Loading…
    • troubleshoot-consumer-lag.md 11.1 KB
      Loading…
    • troubleshoot-performance.md 9.1 KB
      Loading…
  • .skilleval.yaml 77 B
    Loading…
  • CHANGELOG.md 1.9 KB
    Loading…
  • README.md 8 KB
    # MSK Operations — AWS DevOps Agent Skill
    
    An Amazon MSK Provisioned operations, troubleshooting, and health-assessment
    skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html).
    Covers Standard and Express brokers and both proactive operational reviews and
    ad-hoc incident response (performance, consumer lag, storage, maintenance,
    client tuning).
    
    > ⚠️ **Non-production disclaimer.** This skill is sample code, not intended for
    > production use without additional review and testing. Users should validate in
    > a non-production environment first.
    
    ## Purpose
    
    Give AWS DevOps Agent the domain knowledge to answer MSK questions accurately
    without falling back on training data (which routinely conflates Standard and
    Express broker behavior). The skill activates in two shapes:
    
    1. **Operational review** — assess an MSK cluster's health against best
       practices and produce a prioritized findings report.
    2. **Ad-hoc troubleshooting** — investigate a specific MSK symptom (high CPU,
       consumer lag, disk full, TrafficShaping, unexpected broker restart, etc.)
       and recommend the correct next action.
    
    ## Key Capabilities
    
    - Determine broker type (Standard vs Express) and route to the correct diagnostic
      path — many CloudWatch metrics and behaviors differ between the two.
    - Troubleshoot broker performance issues (`CpuUser`, `CpuSystem`,
      `RequestHandlerAvgIdlePercent`, `NetworkProcessorAvgIdlePercent`,
      `ProduceTotalTimeMsMean`, `TrafficShaping`).
    - Diagnose consumer lag using `SumOffsetLag`, `MaxOffsetLag`, and (when
      `PER_TOPIC_PER_PARTITION` is enabled) per-partition `OffsetLag`.
    - Manage broker storage — EBS scaling for Standard, `StorageUsed` monitoring for
      Express, tiered storage, retention planning.
    - Recommend CloudWatch alarms in the `AWS/Kafka` namespace and validate that
      the right monitoring level is enabled.
    - Explain rolling restart, patching, and version upgrade behavior; call out
      operations that are unsafe during `UnderReplicatedPartitions > 0`.
    - Advise on Kafka client (producer / consumer) configuration — batch sizing,
      `linger.ms`, compression, `acks`, `min.insync.replicas` — and on
      authentication choices (IAM, SCRAM, mTLS).
    
    ## Prerequisites
    
    ### IAM permissions
    
    The AWS DevOps Agent's primary cloud-source role needs read access to MSK and
    CloudWatch. All calls except `kafka:GetBootstrapBrokers` are covered by
    `AIDevOpsAgentAccessPolicy`. `kafka:GetBootstrapBrokers` is granted by the
    opt-in `EnableMskOperations` parameter (default `true`) in
    [`cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml`](https://github.com/aws/tools-for-devops-agent/blob/main/cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml).
    The full set of actions the skill uses in practice:
    
    ```
    kafka:DescribeClusterV2
    kafka:ListClustersV2
    kafka:ListNodes
    kafka:GetBootstrapBrokers
    kafka:ListClusterOperationsV2
    kafka:DescribeConfigurationRevision
    kafka:ListConfigurations
    cloudwatch:GetMetricData
    cloudwatch:GetMetricStatistics
    cloudwatch:ListMetrics
    cloudwatch:DescribeAlarms
    logs:DescribeLogGroups
    logs:FilterLogEvents
    ```
    
    Write actions (`UpdateBrokerStorage`, `CreateConfiguration`,
    `PutMetricAlarm`, etc.) are only recommended in the skill output — the operator
    is expected to run them after review.
    
    ### Cluster access
    
    - The MSK cluster must live in an account configured as a cloud source in your
      Agent Space.
    - No data-plane / Kafka-protocol access is required. Everything the skill uses
      comes from the AWS control-plane APIs and CloudWatch.
    
    ## Limitations
    
    - Covers **MSK Provisioned only** — does **not** cover MSK Connect, MSK
      Serverless, or MSK Replicator.
    - No Kafka data-plane visibility. The skill cannot read topic contents, run
      `kafka-consumer-groups.sh`, or otherwise interact with the Kafka protocol.
    - Some checks require monitoring level `PER_BROKER` or higher (thread pool idle
      %, `VolumeQueueLength`, `BwInAllowanceExceeded`, IAM connection metrics). At
      `DEFAULT` level the skill will explicitly note which checks are limited.
    - Sizing questions (broker count, instance type choice, monthly cost) are
      redirected to the AWS documentation — the skill does not include a bundled
      sizing calculator.
    
    ## Agent Types
    
    This skill is intended for the following agent types (selected in the Operator
    Web App at upload time):
    
    - **Chat tasks** — conversational invocation in Chat ("my MSK cluster is
      latent", "run an MSK health check on `prod-cluster`", "why did broker 2
      restart last night?").
    
    Select **All agents** instead if you want the skill available to all agent types.
    
    ## Uploading to AWS DevOps Agent
    
    > Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill)
    
    ### 1. Package the skill
    
    From the `skills/` directory in this repo:
    
    ```bash
    cd skills
    zip -r msk-operations.zip msk-operations/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'
    ```
    
    The resulting `msk-operations.zip` contains:
    
    ```
    msk-operations/
    ├── SKILL.md
    └── references/
        ├── troubleshoot-performance.md
        ├── troubleshoot-consumer-lag.md
        ├── manage-storage.md
        ├── monitor-and-alarm.md
        ├── maintenance-operations.md
        └── configure-clients.md
    ```
    
    Constraints (enforced at upload time):
    
    - Total zip size ≤ **6 MB**.
    - `SKILL.md` is required and must include `name` and `description` frontmatter.
    - A `scripts/` directory is **not** allowed — uploads containing scripts are rejected.
    
    ### 2. Upload via the Operator Web App
    
    1. Navigate to the **Skills** page in your Agent Space Operator Web App.
    2. Click **Add skill** → **Upload skill**.
    3. Drag and drop `msk-operations.zip` (or browse to it).
    4. Select agent types: **Chat tasks** (or leave **All agents**
       to make it available to all agent types).
    5. Review the validation results.
    6. Click **Upload**.
    
    ## How to Use This Skill
    
    In DevOps Agent Chat, use natural language. You do NOT need to mention the
    skill name — the agent activates it based on the MSK / Kafka triggers in the
    skill description.
    
    ### Ad-hoc troubleshooting prompts
    
    - *"My MSK cluster `prod-orders` is showing high produce latency in `us-east-1`.
      What should I check?"*
    - *"Consumer group `payments-consumer` on cluster `prod-orders` has growing
      lag. Diagnose it."*
    - *"UnderReplicatedPartitions is > 0 on broker 2 of `prod-orders`. What's the
      safe next step?"*
    - *"Why did broker 3 of `prod-orders` restart last night?"*
    - *"Storage on `prod-orders` is at 82%. What are my options?"*
    - *"TrafficShaping is firing on all three brokers of `prod-orders`."*
    
    ### Operational-review prompts
    
    - *"Run an MSK operational review on cluster `prod-orders`."*
    - *"Audit `prod-orders` against MSK best practices."*
    - *"Are the CloudWatch alarms on `prod-orders` complete? Which ones am I
      missing?"*
    - *"Health-check every MSK Provisioned cluster in `us-east-1`."*
    
    ### Client tuning prompts
    
    - *"My Java Kafka producer is getting `NotEnoughReplicasException` when writing
      to MSK. What producer settings should I check?"*
    - *"What should `linger.ms` and `batch.size` be for a producer writing 200 MB/s
      to a Standard broker cluster?"*
    
    ## Skill Contents
    
    ```
    msk-operations/
    ├── SKILL.md
    ├── README.md
    ├── CHANGELOG.md
    ├── .skilleval.yaml
    ├── evals/
    │   ├── evals.json
    │   └── eval_queries.json
    └── references/
        ├── troubleshoot-performance.md
        ├── troubleshoot-consumer-lag.md
        ├── manage-storage.md
        ├── monitor-and-alarm.md
        ├── maintenance-operations.md
        └── configure-clients.md
    ```
    
    `README.md`, `CHANGELOG.md`, `.skilleval.yaml`, and `evals/` are for repo
    maintenance and are not required at the cluster — the `zip` command above
    excludes them from the upload artifact.
    
  • SKILL.md 26 KB
    ---
    name: msk-operations
    description: Amazon MSK Provisioned operations, troubleshooting, and health
      assessment for Standard and Express brokers. Use whenever the user mentions
      Amazon MSK, MSK Provisioned, MSK Standard/Express brokers, Apache Kafka on
      AWS, `kafka.*` / `express.*` instance types, or the `AWS/Kafka` CloudWatch
      namespace. Covers MSK performance issues (high CPU, produce/fetch latency,
      TrafficShaping), consumer lag, storage/EBS issues, rolling restarts, Kafka
      version upgrades, SECURITY_PATCHING, BROKER_UPDATE, CloudWatch alarm design,
      Kafka client (producer/consumer) tuning, under-replicated partitions,
      unexpected broker reboots, and full MSK operational reviews / health checks
      / best-practices audits. Do NOT use for MSK Connect, MSK Serverless, or MSK
      Replicator. Do NOT use for authoring CloudFormation, CDK, or Terraform
      templates. Do NOT use for other AWS services (RDS, Aurora, S3, DynamoDB,
      Kinesis, Lambda, EC2) unless MSK is explicitly named.
    metadata:
      author: kjjanaki
      version: "1.0.5"
      aws-devops-agent-skills.agent-types: "CHAT"
      aws-devops-agent-skills.aws-services: "Amazon MSK"
      aws-devops-agent-skills.technical-domains: "Analytics"
    ---
    
    # Amazon MSK Operations
    
    Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka)
    Provisioned clusters — both Standard and Express broker types. This skill covers
    day-to-day operations (health assessments, monitoring setup) and ad-hoc incident
    response (performance degradation, consumer lag, storage full, unexpected broker
    reboots).
    
    ## When to Use
    
    Activate this skill when the user asks to:
    
    - Review, audit, or assess an MSK cluster for best practices, health, or
      operational readiness.
    - Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency,
      consumer lag, broker storage running out, TrafficShaping events, under-replicated
      partitions, or an unexpected broker restart.
    - Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch
      alarms and dashboards, understand the metrics available in the `AWS/Kafka`
      namespace.
    - Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security
      patching, broker instance type change.
    - Advise on Kafka client (producer / consumer) configuration when the client is
      connecting to an MSK cluster.
    
    Do **not** activate this skill for MSK Connect, MSK Serverless, or MSK Replicator
    — those are separate services with their own operational surfaces.
    
    ## Broker Type Determination
    
    Determine the broker type first — many checks differ between Standard and Express.
    
    ```
    aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
    ```
    
    Check `ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType`:
    
    - Starts with `kafka.` (e.g. `kafka.m5.large`, `kafka.m7g.xlarge`) → **Standard broker**.
    - Starts with `express.` (e.g. `express.m7g.large`) → **Express broker**.
    
    ### Key Standard vs Express differences
    
    **Standard brokers** use customer-managed EBS volumes for storage. You choose
    instance types (`kafka.m5.*`, `kafka.m7g.*`), provision EBS, and manage storage
    scaling. Standard brokers have scheduled maintenance windows.
    
    **Express brokers** provide fully managed, pay-as-you-go storage with no EBS
    provisioning. Instance types are prefixed with `express.m7g.*`. Express brokers
    offer up to 3× more throughput per broker than Standard, and have no maintenance
    windows. Express enforces a fixed replication factor of 3 and
    `min.insync.replicas=2` — you cannot create topics with RF=1.
    
    ## Critical Warnings
    
    - **This skill is read-only.** Every command in this file and in `references/`
      that mutates cluster state — `update-broker-storage`, `create-configuration`,
      `update-cluster-configuration`, `update-monitoring`, `put-metric-alarm`,
      `reboot-broker`, and any partition reassignment — is a **recommendation for
      the operator to run after review**. Present these as proposed remediations
      with expected impact and preconditions; do NOT execute them, and do NOT
      imply that the agent will run them.
    - **NEVER reboot brokers while `UnderReplicatedPartitions` > 0** (Standard only —
      Express brokers do not emit URP). This risks data loss and extended outages.
    - **NEVER recommend partition reassignment without first checking replication
      status.** Reassignment during URP compounds the problem.
    - **`linger.ms=0` is the #1 cause of "high CPU" on MSK.** ALWAYS check client
      batch configuration before recommending broker scaling.
    - **EBS throughput ceilings are invisible in Kafka metrics** — ALWAYS check EBS
      volume metrics (`VolumeReadBytes`, `VolumeWriteBytes`, `VolumeQueueLength`)
      when diagnosing Standard broker latency.
    - **Express brokers have NO customer-managed EBS** — do NOT recommend EBS
      expansion or provisioned EBS throughput for Express clusters.
    - **Express brokers enforce fixed RF=3 and `min.insync.replicas=2`** — do NOT
      attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard
      brokers.
    
    ## Quick Diagnostics
    
    These five checks cover the most common MSK issues. Use them before loading a
    reference file.
    
    1. **`CpuUser + CpuSystem` > 60%**: Check `RequestHandlerAvgIdlePercent`
       (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check
       client `batch.size` and `linger.ms` before recommending scaling.
    
    2. **`KafkaDataLogsDiskUsed` > 85%** (Standard only): **Recommend to the
       operator** that EBS be expanded via `aws kafka update-broker-storage` (do
       not execute). Identify high-growth topics via per-topic `BytesInPerSec` to
       size the increase. Express clusters use `StorageUsed` metric instead and
       storage is fully managed.
    
    3. **`UnderReplicatedPartitions` > 0** (Standard only): Check if a maintenance
       operation or broker restart is in progress. If URP is decreasing, wait for
       recovery. Do NOT restart brokers or reassign partitions during URP. Express
       brokers do not emit this metric — monitor `ProduceThrottleTime`,
       `FetchThrottleTime`, and consumer lag instead.
    
    4. **Consumer `OffsetLag` / `MaxOffsetLag` increasing**: Determine if broker-side
       (high `ProduceTotalTimeMsMean`, CPU saturation) or client-side (slow
       processing, insufficient consumers). Per-partition lag from
       `PER_TOPIC_PER_PARTITION` monitoring level helps isolate hot partitions.
    
    5. **`BytesInPerSec` near throughput ceiling**: For Standard, check EBS volume
       type and calculate: `BytesInPerSec × ReplicationFactor` vs volume throughput
       limit. For Express, check against the per-broker sustained performance limits
       in the MSK quotas.
    
    ## Which Reference Do You Need?
    
    Route to a reference file based on the customer intent. Read the reference in
    full before answering — do not paraphrase from memory.
    
    | Customer Intent | Reference |
    |---|---|
    | High CPU, high produce/fetch latency, slow cluster, TrafficShaping | `references/troubleshoot-performance.md` |
    | Consumer lag increasing, rebalance storms, stuck consumer groups | `references/troubleshoot-consumer-lag.md` |
    | Disk filling up, retention planning, tiered storage, EBS scaling | `references/manage-storage.md` |
    | Setting up monitoring level, dashboards, recommended CloudWatch alarms | `references/monitor-and-alarm.md` |
    | Rolling restart impact, patching, Kafka version upgrades, maintenance resilience | `references/maintenance-operations.md` |
    | Producer / consumer configuration, IAM / SCRAM / TLS auth for clients | `references/configure-clients.md` |
    
    For sizing questions (broker count, instance type choice, monthly cost), refer
    the user to the [Amazon MSK best practices — right-size your cluster](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-right-size-cluster)
    documentation. Do not size from memory.
    
    ## Operational Review Workflow
    
    Use this workflow when the user asks for a **review, audit, health check, or
    assessment** of an MSK cluster. The routing table above handles ad-hoc
    troubleshooting; this section produces a consistent, comprehensive report.
    
    Follow the steps **in order** for each target cluster. Do not skip steps. If a
    step cannot be completed (e.g. a metric requires a higher monitoring level
    than the cluster has enabled), record the gap in the report rather than
    silently omitting the check.
    
    ### Step 1 — Identify Target Clusters
    
    Ask the user which MSK clusters to review. Accept any of:
    
    - Specific cluster names or ARNs and regions
    - "all clusters" in specific regions
    - "all MSK clusters in all regions"
    
    If no scope is given, default to all configured account regions. Enumerate
    clusters with `aws kafka list-clusters-v2` per region.
    
    ### Step 2 — Determine Broker Type Per Cluster
    
    For each cluster:
    
    ```
    aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
    ```
    
    Read `ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType`. Standard
    brokers (`kafka.*`) and Express brokers (`express.*`) require different checks
    in the later steps — some metrics only exist on one type.
    
    ### Step 3 — Collect Cluster Configuration
    
    For each cluster, gather:
    
    ```
    aws kafka describe-cluster-v2 --cluster-arn <arn>
    aws kafka list-nodes --cluster-arn <arn>
    aws kafka get-bootstrap-brokers --cluster-arn <arn>
    aws kafka list-cluster-operations-v2 --cluster-arn <arn>   # last 30 days
    aws kafka describe-configuration-revision \
         --arn <configuration-arn> --revision <revision>       # if a custom config is applied
    ```
    
    Capture:
    
    - **Cluster:** state, Kafka version, number of broker nodes, AZ distribution
      (`ZoneIds`), storage mode (EBS / Tiered), current version.
    - **Broker:** instance type, EBS volume size (Standard), provisioned throughput
      (if any).
    - **Encryption:** `EncryptionInTransit.ClientBroker` (TLS / TLS_PLAINTEXT /
      PLAINTEXT), `EncryptionInTransit.InCluster`, `EncryptionAtRest.DataVolumeKMSKeyId`.
    - **Auth:** `ClientAuthentication.Sasl.Iam.Enabled`,
      `ClientAuthentication.Sasl.Scram.Enabled`, `ClientAuthentication.Tls.Enabled`,
      `ClientAuthentication.Unauthenticated.Enabled`.
    - **Public access:** `BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type`.
    - **Monitoring level:** `EnhancedMonitoring` (`DEFAULT` / `PER_BROKER` /
      `PER_TOPIC_PER_BROKER` / `PER_TOPIC_PER_PARTITION`).
    - **Logging:** `LoggingInfo.BrokerLogs` (CloudWatch / S3 / Firehose destinations
      and their `Enabled` flags).
    - **Open monitoring:** `OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker`,
      `NodeExporter.EnabledInBroker`.
    - **Recent operations:** From `list-cluster-operations-v2`, note any
      `SECURITY_PATCHING`, `BROKER_UPDATE`, `UPDATE_CLUSTER_CONFIGURATION`,
      `UPDATE_STORAGE`, or `UPDATE_MONITORING` events in the review window.
    
    ### Step 4 — Detect Monitoring Level and Gaps
    
    The `EnhancedMonitoring` value from Step 3 determines which checks are
    available. At `DEFAULT`, most per-broker health metrics are still available
    (CPU, disk, network, partitions, connections, memory, TrafficShaping), but the
    following checks are **not possible without upgrading**:
    
    - `ReplicationBytesInPerSec` / `ReplicationBytesOutPerSec` (inter-broker
      replication load)
    - `RequestHandlerAvgIdlePercent` / `NetworkProcessorAvgIdlePercent` (thread pool
      saturation)
    - `BwInAllowanceExceeded` / `BwOutAllowanceExceeded` (detailed bandwidth
      breaches)
    - `VolumeQueueLength` (EBS I/O queue depth — Standard only)
    - `VolumeReadBytes` / `VolumeWriteBytes` (EBS throughput utilization — Standard
      only)
    - IAM connection metrics (`IAMTooManyConnections`)
    
    Record the monitoring level and list any dimensions that will be scored
    partially or skipped. Recommend upgrading to `PER_BROKER` if any dimension is
    degraded by the current level.
    
    ### Step 5 — Collect CloudWatch Metrics (7-Day Historical)
    
    Namespace: `AWS/Kafka`. Dimensions: `Cluster Name` and `Broker ID` for
    per-broker metrics; `Cluster Name` and `Consumer Group` and `Topic` for
    consumer lag; `Cluster Name` only for cluster-wide metrics.
    
    Use one `cloudwatch.GetMetricData` batch per cluster where possible.
    `Period: 3600` (1 hour). `StartTime`: 7 days ago. `EndTime`: now.
    
    #### 5.1 Cluster-wide (both broker types)
    
    | Metric | Stat | Purpose |
    |---|---|---|
    | `ActiveControllerCount` | Sum | Must be exactly 1 |
    | `OfflinePartitionsCount` | Maximum | Must be 0 |
    | `GlobalPartitionCount` | Maximum | Total leader partitions |
    | `GlobalTopicCount` | Maximum | Total topics |
    
    #### 5.2 Per-broker — Standard (`kafka.*`)
    
    | Metric | Stat | Threshold |
    |---|---|---|
    | `CpuUser` + `CpuSystem` | Average, Maximum | < 60% avg |
    | `KafkaDataLogsDiskUsed` | Average, Maximum | < 70% avg, < 85% max |
    | `PartitionCount` | Maximum | ≤ recommended limit for broker size |
    | `LeaderCount` | Maximum | Compare across brokers; skew < 10% |
    | `UnderReplicatedPartitions` | Maximum | 0 in steady state |
    | `UnderMinIsrPartitionCount` | Maximum | 0 |
    | `BytesInPerSec` / `BytesOutPerSec` | Average, Maximum | vs baseline bandwidth |
    | `ConnectionCount` | Average, Maximum | Compare across brokers |
    | `HeapMemoryAfterGC` | Maximum | < 60% |
    | `TrafficShaping` | Sum | Must be 0 |
    | `NetworkRxDropped` / `NetworkTxDropped` / `NetworkRxErrors` / `NetworkTxErrors` | Sum | Must be 0 |
    | `ReplicationBytesInPerSec` / `ReplicationBytesOutPerSec` | Average | Requires PER_BROKER |
    | `RequestHandlerAvgIdlePercent` / `NetworkProcessorAvgIdlePercent` | Average | > 30% (PER_BROKER) |
    | `VolumeQueueLength` | Average, Maximum | Avg < 1 (PER_BROKER) |
    | `VolumeReadBytes` + `VolumeWriteBytes` | Sum | vs EBS baseline throughput (PER_BROKER) |
    
    #### 5.3 Per-broker — Express (`express.*`)
    
    | Metric | Stat | Threshold |
    |---|---|---|
    | `CpuUser` + `CpuSystem` | Average, Maximum | < 60% avg |
    | `StorageUsed` | Maximum | Fully managed — flag if trending against per-broker quota |
    | `PartitionCount` | Maximum | ≤ recommended limit for broker size |
    | `LeaderCount` | Maximum | Compare across brokers |
    | `BytesInPerSec` / `BytesOutPerSec` | Average, Maximum | vs Express per-broker ingress/egress quotas |
    | `ProduceThrottleTime` / `FetchThrottleTime` | Maximum | Should be 0 |
    | `ClientConnectionCount` | Average, Maximum | vs listener quota (see `references/monitor-and-alarm.md`) |
    
    **Express brokers do NOT emit:** `UnderReplicatedPartitions`,
    `UnderMinIsrPartitionCount`, `HeapMemoryAfterGC`, `TrafficShaping`, `Volume*`
    metrics, `KafkaDataLogsDiskUsed`, `ProduceMessageConversionsPerSec`,
    `FetchMessageConversionsPerSec`.
    
    #### 5.4 Consumer lag (both broker types)
    
    Per consumer group (identified from the customer or from the broker logs):
    
    | Metric | Stat | Notes |
    |---|---|---|
    | `SumOffsetLag` per (Consumer Group, Topic) | Maximum | DEFAULT level |
    | `MaxOffsetLag` per (Consumer Group, Topic) | Maximum | DEFAULT level |
    | `EstimatedMaxTimeLag` per (Consumer Group, Topic) | Maximum | DEFAULT level |
    | `OffsetLag` per (Consumer Group, Topic, Partition) | Maximum | Requires PER_TOPIC_PER_PARTITION |
    
    ### Step 6 — Alarm Coverage
    
    Collect existing alarms:
    
    ```
    aws cloudwatch describe-alarms --namespace AWS/Kafka
    ```
    
    Compare against the 13-alarm recommended set (details in
    `references/monitor-and-alarm.md`) and produce a coverage table (present / missing / firing).
    
    ### Step 7 — Analyze Against Best Practices
    
    Assign a severity to every finding: **CRITICAL / HIGH / MEDIUM / LOW / INFO**
    (see Severity Definitions later in this file).
    
    Evaluate across seven dimensions. Some checks are skipped for Express — noted
    inline.
    
    #### 7.1 Cluster Configuration
    
    - Cluster state is `ACTIVE`. `MAINTENANCE` / `UPDATING` is transient; anything
      else is a finding.
    - Deployed across **3 AZs** (Standard: broker count multiple of AZ count).
    - Kafka version within N-2 of the latest supported.
    - Enhanced monitoring at `PER_BROKER` or higher — `DEFAULT` → MEDIUM.
    - Storage mode: Tiered storage enabled for topics with long retention
      (Standard).
    
    #### 7.2 Security
    
    - Encryption in transit: `TLS`. `TLS_PLAINTEXT` → MEDIUM (prod: HIGH).
      `PLAINTEXT` → CRITICAL.
    - Encryption at rest: enabled with KMS. Customer-managed KMS preferred over
      AWS-managed for regulated workloads.
    - Authentication: at least one of IAM / SCRAM / mTLS enabled.
      `Unauthenticated.Enabled=true` in production → CRITICAL.
    - Public access: `SERVICE_PROVIDED_EIPS` in production → HIGH.
    
    #### 7.3 Logging & Monitoring
    
    - Broker logs enabled to at least one destination (CloudWatch / S3 / Firehose).
      All disabled → HIGH.
    - Open monitoring (Prometheus JMX + Node exporter) — informational.
    - Alarm coverage from Step 6: missing alarms → MEDIUM each; missing critical
      alarms (Active Controller, Offline Partitions, Disk, CPU) → HIGH.
    - Any alarm currently in `ALARM` state → HIGH (surface in report header).
    
    #### 7.4 Partition Health
    
    - `PartitionCount` per broker ≤ recommended limit for the broker instance
      type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks
      update operations).
    - `LeaderCount` variance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH.
    - `UnderReplicatedPartitions` > 0 sustained (Standard only) → HIGH. Transient
      during a `SECURITY_PATCHING` / `BROKER_UPDATE` operation from Step 3 →
      INFO — do NOT flag.
    - `UnderMinIsrPartitionCount` > 0 (Standard only) → CRITICAL.
    - **Config-level ISR risk:** topics with `min.insync.replicas >= replication.factor`
      are a configuration error the CloudWatch metric will not surface. If the
      user has provided topic configs, flag any such topic as HIGH.
    
    #### 7.5 Compute Health
    
    - `CpuUser + CpuSystem` avg < 60%. 60-70% → MEDIUM. > 70% → HIGH.
    - CPU variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
    - `HeapMemoryAfterGC` < 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH.
    - `RequestHandlerAvgIdlePercent` > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.
    - `NetworkProcessorAvgIdlePercent` > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.
    
    #### 7.6 Network Health
    
    - `TrafficShaping` = 0 (Standard only). Any non-zero → HIGH.
    - `BwInAllowanceExceeded` / `BwOutAllowanceExceeded` = 0 (PER_BROKER). Non-zero → HIGH.
    - Throughput variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
    - Total per-broker throughput < 60% of baseline bandwidth (see
      `references/troubleshoot-performance.md` for baseline table). 60-70% →
      MEDIUM. > 70% → HIGH.
    - Express: `ProduceThrottleTime` / `FetchThrottleTime` > 0 → HIGH.
    
    #### 7.7 Storage Health
    
    **Standard only** (Express storage is managed — Express clusters only get the
    `StorageUsed` check):
    
    - `KafkaDataLogsDiskUsed` < 70%. 70-85% → HIGH. > 85% → CRITICAL.
    - `VolumeQueueLength` avg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH.
    - EBS auto-scaling configured, or disk headroom > 30% → PASS. No
      auto-scaling AND disk > 50% → MEDIUM. No auto-scaling AND disk > 70% → HIGH.
    - EBS throughput (`VolumeReadBytes + VolumeWriteBytes`) < 60% of instance
      baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH.
    
    **Express:**
    
    - `StorageUsed` per broker vs Express per-broker storage quota (see MSK
      Express quotas). Approaching quota → MEDIUM.
    
    ### Step 8 — Generate the Report
    
    Generate a **separate report artifact per cluster reviewed**.
    
    Artifact naming: `msk-review-<cluster-name>-<YYYY-MM-DD>.md`
    Example: `msk-review-prod-orders-2026-04-29.md`
    
    Report structure:
    
    #### Report Header
    
    ```
    # MSK Operational Review — <cluster-name>
    Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD>
    Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n>
    Kafka Version: <version> | Monitoring Level: <level>
    ```
    
    #### Executive Summary
    
    - Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL
    - Finding counts by severity
    - Top 3 CRITICAL/HIGH items
    
    #### Configuration Snapshot
    
    | Item | Value |
    | Cluster state / version | … |
    | Broker type / instance / count / AZs | … |
    | Storage | mode, size (Standard), provisioned throughput (if any) |
    | Encryption | in-transit, in-cluster, at-rest (KMS) |
    | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags |
    | Public access | DISABLED / SERVICE_PROVIDED_EIPS |
    | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION |
    | Logging | destinations enabled |
    
    #### Findings by Dimension
    
    For each of the 7 dimensions (7.1-7.7):
    
    | # | Finding | Severity | Current State | Recommendation |
    
    If a dimension was skipped or partial due to monitoring level or broker type,
    say so explicitly in a note above the table for that dimension.
    
    #### CloudWatch Metrics (7-Day)
    
    | Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding |
    
    #### Alarm Coverage
    
    | # | Recommended Alarm | Metric | Threshold | Priority | Status |
    
    Also list alarms currently in `ALARM` state with timestamps.
    
    #### Recent Cluster Operations (Last 30 Days)
    
    From `list-cluster-operations-v2` in Step 3:
    
    | Operation Type | Start Time | End Time | State |
    
    Call out any operations that would explain transient metric anomalies.
    
    #### Priority Matrix
    
    | # | Finding | Severity | Dimension | Effort | Impact |
    
    Sorted by severity.
    
    #### Next Steps
    
    - Immediate (CRITICAL / HIGH — 7 days)
    - Short-term (MEDIUM — 30 days)
    - Long-term (LOW — 90 days)
    
    #### Appendix — Reference Links
    
    - [Amazon MSK best practices — Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html)
    - [Amazon MSK best practices — Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-express.html)
    - [Amazon MSK — Monitoring an MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/monitoring.html)
    - [Amazon MSK — CloudWatch metrics for Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html)
    - [Amazon MSK — Right-size your cluster](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-right-size-cluster)
    - [Amazon MSK — Service quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html)
    
    ## Common CLI Recipes
    
    **Describe cluster:**
    
    ```
    aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
    ```
    
    **List brokers:**
    
    ```
    aws kafka list-nodes --cluster-arn <cluster-arn>
    ```
    
    **Get bootstrap brokers:**
    
    ```
    aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn>
    ```
    
    **List recent cluster operations (patching, config updates, storage changes):**
    
    ```
    aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn>
    ```
    
    **Expand Standard broker storage** — *Operator-run (recommend, do not execute):*
    
    ```
    aws kafka update-broker-storage \
      --cluster-arn <cluster-arn> \
      --current-version <cluster-version> \
      --target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]'
    ```
    
    **Get a CloudWatch metric (example: `CpuUser` per broker):**
    
    ```
    aws cloudwatch get-metric-statistics \
      --namespace AWS/Kafka \
      --metric-name CpuUser \
      --dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \
      --start-time <start> --end-time <end> --period 300 --statistics Average
    ```
    
    **Create a cluster configuration (`server.properties`)** — *Operator-run (recommend, do not execute):*
    
    The `--server-properties` argument MUST be a real Kafka properties file with
    one `key=value` per line, separated by actual newline characters — NOT the
    literal two-character escape sequence `\n`. The MSK API accepts the bytes as-is;
    if you pass `"k1=v1\nk2=v2"` as a single string with escaped newlines, MSK
    stores ONE invalid property line and the cluster will fail to apply it.
    
    Recommended pattern: write the properties to a local file with real newlines,
    then pass it via `fileb://` so the CLI uploads the raw bytes verbatim. Verify by
    reading the revision back with `describe-configuration-revision` and
    base64-decoding `ServerProperties` — you should see one property per line.
    
    ```
    cat > server.properties <<'EOF'
    auto.create.topics.enable=false
    default.replication.factor=3
    min.insync.replicas=2
    unclean.leader.election.enable=false
    num.io.threads=32
    num.network.threads=16
    log.retention.hours=168
    EOF
    
    aws kafka create-configuration \
      --name <config-name> \
      --kafka-versions "3.6.0" \
      --server-properties fileb://server.properties
    ```
    
    ## Common Error Reference
    
    | Error | Cause | Fix |
    |---|---|---|
    | `aws kafka update-broker-storage` returns "storage is optimizing" | Previous storage expansion still in cool-down (minimum 6 hours) | Wait for optimization to complete. Check cluster state with `describe-cluster-v2`. |
    | `ClusterState` is `MAINTENANCE` | Standard brokers undergoing patching. Express brokers stay `ACTIVE` during maintenance. | Wait for cluster to return to `ACTIVE`. Do not perform update operations during `MAINTENANCE`. |
    | Consumer receives `GROUP_COORDINATOR_NOT_AVAILABLE` | Coordinator broker is temporarily unavailable during rolling restart or overloaded | Retry with backoff. Check if maintenance is in progress via `list-cluster-operations-v2`. |
    | `NotEnoughReplicasException` on produce | Fewer brokers in ISR than `min.insync.replicas` (default: 2) | Check `UnderReplicatedPartitions` (Standard only). For Express, check `ProduceThrottleTime` and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower `min.insync.replicas` to work around this. |
    
    ## Severity Definitions (for review-style reports)
    
    | Severity | Definition | SLA |
    |---|---|---|
    | CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours |
    | HIGH | Significant gap that could lead to incidents | Fix within 1 week |
    | MEDIUM | Notable improvement opportunity | Plan within 30 days |
    | LOW | Minor optimization or hardening | Address when convenient |
    | INFO | Observation, no action required | N/A |
    
    ## Additional Resources
    
    - [Amazon MSK best practices — Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html)
    - [Amazon MSK best practices — Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-express.html)
    - [Amazon MSK — Apache Kafka client best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-kafka-client.html)
    - [Amazon MSK — Monitoring an MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/monitoring.html)
    - [Amazon MSK — CloudWatch metrics for Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html)
    - [Amazon MSK — Service quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html)
    - [Amazon MSK — Custom MSK configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration.html)
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.