msk-operations
Amazon MSK Provisioned operations, troubleshooting, and health
Install
npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/msk-operations
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart
git clone https://github.com/aws/tools-for-devops-agent.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.
README
MSK Operations — AWS DevOps Agent Skill
An Amazon MSK Provisioned operations, troubleshooting, and health-assessment skill for AWS DevOps Agent. Covers Standard and Express brokers and both proactive operational reviews and ad-hoc incident response (performance, consumer lag, storage, maintenance, client tuning).
⚠️ Non-production disclaimer. This skill is sample code, not intended for production use without additional review and testing. Users should validate in a non-production environment first.
Purpose
Give AWS DevOps Agent the domain knowledge to answer MSK questions accurately without falling back on training data (which routinely conflates Standard and Express broker behavior). The skill activates in two shapes:
- Operational review — assess an MSK cluster's health against best practices and produce a prioritized findings report.
- Ad-hoc troubleshooting — investigate a specific MSK symptom (high CPU, consumer lag, disk full, TrafficShaping, unexpected broker restart, etc.) and recommend the correct next action.
Key Capabilities
- Determine broker type (Standard vs Express) and route to the correct diagnostic path — many CloudWatch metrics and behaviors differ between the two.
- Troubleshoot broker performance issues (
CpuUser,CpuSystem,RequestHandlerAvgIdlePercent,NetworkProcessorAvgIdlePercent,ProduceTotalTimeMsMean,TrafficShaping). - Diagnose consumer lag using
SumOffsetLag,MaxOffsetLag, and (whenPER_TOPIC_PER_PARTITIONis enabled) per-partitionOffsetLag. - Manage broker storage — EBS scaling for Standard,
StorageUsedmonitoring for Express, tiered storage, retention planning. - Recommend CloudWatch alarms in the
AWS/Kafkanamespace and validate that the right monitoring level is enabled. - Explain rolling restart, patching, and version upgrade behavior; call out
operations that are unsafe during
UnderReplicatedPartitions > 0. - Advise on Kafka client (producer / consumer) configuration — batch sizing,
linger.ms, compression,acks,min.insync.replicas— and on authentication choices (IAM, SCRAM, mTLS).
Prerequisites
IAM permissions
The AWS DevOps Agent's primary cloud-source role needs read access to MSK and
CloudWatch. All calls except kafka:GetBootstrapBrokers are covered by
AIDevOpsAgentAccessPolicy. kafka:GetBootstrapBrokers is granted by the
opt-in EnableMskOperations parameter (default true) in
cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml.
The full set of actions the skill uses in practice:
kafka:DescribeClusterV2
kafka:ListClustersV2
kafka:ListNodes
kafka:GetBootstrapBrokers
kafka:ListClusterOperationsV2
kafka:DescribeConfigurationRevision
kafka:ListConfigurations
cloudwatch:GetMetricData
cloudwatch:GetMetricStatistics
cloudwatch:ListMetrics
cloudwatch:DescribeAlarms
logs:DescribeLogGroups
logs:FilterLogEvents
Write actions (UpdateBrokerStorage, CreateConfiguration,
PutMetricAlarm, etc.) are only recommended in the skill output — the operator
is expected to run them after review.
Cluster access
- The MSK cluster must live in an account configured as a cloud source in your Agent Space.
- No data-plane / Kafka-protocol access is required. Everything the skill uses comes from the AWS control-plane APIs and CloudWatch.
Limitations
- Covers MSK Provisioned only — does not cover MSK Connect, MSK Serverless, or MSK Replicator.
- No Kafka data-plane visibility. The skill cannot read topic contents, run
kafka-consumer-groups.sh, or otherwise interact with the Kafka protocol. - Some checks require monitoring level
PER_BROKERor higher (thread pool idle %,VolumeQueueLength,BwInAllowanceExceeded, IAM connection metrics). AtDEFAULTlevel the skill will explicitly note which checks are limited. - Sizing questions (broker count, instance type choice, monthly cost) are redirected to the AWS documentation — the skill does not include a bundled sizing calculator.
Agent Types
This skill is intended for the following agent types (selected in the Operator Web App at upload time):
- Chat tasks — conversational invocation in Chat ("my MSK cluster is
latent", "run an MSK health check on
prod-cluster", "why did broker 2 restart last night?").
Select All agents instead if you want the skill available to all agent types.
Uploading to AWS DevOps Agent
Reference: Uploading a skill
1. Package the skill
From the skills/ directory in this repo:
cd skills
zip -r msk-operations.zip msk-operations/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'
The resulting msk-operations.zip contains:
msk-operations/
├── SKILL.md
└── references/
├── troubleshoot-performance.md
├── troubleshoot-consumer-lag.md
├── manage-storage.md
├── monitor-and-alarm.md
├── maintenance-operations.md
└── configure-clients.md
Constraints (enforced at upload time):
- Total zip size ≤ 6 MB.
SKILL.mdis required and must includenameanddescriptionfrontmatter.- A
scripts/directory is not allowed — uploads containing scripts are rejected.
2. Upload via the Operator Web App
- Navigate to the Skills page in your Agent Space Operator Web App.
- Click Add skill → Upload skill.
- Drag and drop
msk-operations.zip(or browse to it). - Select agent types: Chat tasks (or leave All agents to make it available to all agent types).
- Review the validation results.
- Click Upload.
How to Use This Skill
In DevOps Agent Chat, use natural language. You do NOT need to mention the skill name — the agent activates it based on the MSK / Kafka triggers in the skill description.
Ad-hoc troubleshooting prompts
- "My MSK cluster
prod-ordersis showing high produce latency inus-east-1. What should I check?" - "Consumer group
payments-consumeron clusterprod-ordershas growing lag. Diagnose it." - "UnderReplicatedPartitions is > 0 on broker 2 of
prod-orders. What's the safe next step?" - "Why did broker 3 of
prod-ordersrestart last night?" - "Storage on
prod-ordersis at 82%. What are my options?" - "TrafficShaping is firing on all three brokers of
prod-orders."
Operational-review prompts
- "Run an MSK operational review on cluster
prod-orders." - "Audit
prod-ordersagainst MSK best practices." - "Are the CloudWatch alarms on
prod-orderscomplete? Which ones am I missing?" - "Health-check every MSK Provisioned cluster in
us-east-1."
Client tuning prompts
- "My Java Kafka producer is getting
NotEnoughReplicasExceptionwhen writing to MSK. What producer settings should I check?" - "What should
linger.msandbatch.sizebe for a producer writing 200 MB/s to a Standard broker cluster?"
Skill Contents
msk-operations/
├── SKILL.md
├── README.md
├── CHANGELOG.md
├── .skilleval.yaml
├── evals/
│ ├── evals.json
│ └── eval_queries.json
└── references/
├── troubleshoot-performance.md
├── troubleshoot-consumer-lag.md
├── manage-storage.md
├── monitor-and-alarm.md
├── maintenance-operations.md
└── configure-clients.md
README.md, CHANGELOG.md, .skilleval.yaml, and evals/ are for repo
maintenance and are not required at the cluster — the zip command above
excludes them from the upload artifact.
Skill manifest
Amazon MSK Operations
Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots).
When to Use
Activate this skill when the user asks to:
- Review, audit, or assess an MSK cluster for best practices, health, or operational readiness.
- Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency, consumer lag, broker storage running out, TrafficShaping events, under-replicated partitions, or an unexpected broker restart.
- Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch
alarms and dashboards, understand the metrics available in the
AWS/Kafkanamespace. - Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security patching, broker instance type change.
- Advise on Kafka client (producer / consumer) configuration when the client is connecting to an MSK cluster.
Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces.
Broker Type Determination
Determine the broker type first — many checks differ between Standard and Express.
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
Check ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType:
- Starts with
kafka.(e.g.kafka.m5.large,kafka.m7g.xlarge) → Standard broker. - Starts with
express.(e.g.express.m7g.large) → Express broker.
Key Standard vs Express differences
Standard brokers use customer-managed EBS volumes for storage. You choose
instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage
scaling. Standard brokers have scheduled maintenance windows.
Express brokers provide fully managed, pay-as-you-go storage with no EBS
provisioning. Instance types are prefixed with express.m7g.*. Express brokers
offer up to 3× more throughput per broker than Standard, and have no maintenance
windows. Express enforces a fixed replication factor of 3 and
min.insync.replicas=2 — you cannot create topics with RF=1.
Critical Warnings
- This skill is read-only. Every command in this file and in
references/that mutates cluster state —update-broker-storage,create-configuration,update-cluster-configuration,update-monitoring,put-metric-alarm,reboot-broker, and any partition reassignment — is a recommendation for the operator to run after review. Present these as proposed remediations with expected impact and preconditions; do NOT execute them, and do NOT imply that the agent will run them. - NEVER reboot brokers while
UnderReplicatedPartitions> 0 (Standard only — Express brokers do not emit URP). This risks data loss and extended outages. - NEVER recommend partition reassignment without first checking replication status. Reassignment during URP compounds the problem.
linger.ms=0is the #1 cause of "high CPU" on MSK. ALWAYS check client batch configuration before recommending broker scaling.- EBS throughput ceilings are invisible in Kafka metrics — ALWAYS check EBS
volume metrics (
VolumeReadBytes,VolumeWriteBytes,VolumeQueueLength) when diagnosing Standard broker latency. - Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned EBS throughput for Express clusters.
- Express brokers enforce fixed RF=3 and
min.insync.replicas=2— do NOT attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard brokers.
Quick Diagnostics
These five checks cover the most common MSK issues. Use them before loading a reference file.
CpuUser + CpuSystem> 60%: CheckRequestHandlerAvgIdlePercent(PER_BROKER monitoring level). If < 30%, request threads are saturated. Check clientbatch.sizeandlinger.msbefore recommending scaling.KafkaDataLogsDiskUsed> 85% (Standard only): Recommend to the operator that EBS be expanded viaaws kafka update-broker-storage(do not execute). Identify high-growth topics via per-topicBytesInPerSecto size the increase. Express clusters useStorageUsedmetric instead and storage is fully managed.UnderReplicatedPartitions> 0 (Standard only): Check if a maintenance operation or broker restart is in progress. If URP is decreasing, wait for recovery. Do NOT restart brokers or reassign partitions during URP. Express brokers do not emit this metric — monitorProduceThrottleTime,FetchThrottleTime, and consumer lag instead.Consumer
OffsetLag/MaxOffsetLagincreasing: Determine if broker-side (highProduceTotalTimeMsMean, CPU saturation) or client-side (slow processing, insufficient consumers). Per-partition lag fromPER_TOPIC_PER_PARTITIONmonitoring level helps isolate hot partitions.BytesInPerSecnear throughput ceiling: For Standard, check EBS volume type and calculate:BytesInPerSec × ReplicationFactorvs volume throughput limit. For Express, check against the per-broker sustained performance limits in the MSK quotas.
Which Reference Do You Need?
Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory.
| Customer Intent | Reference |
|---|---|
| High CPU, high produce/fetch latency, slow cluster, TrafficShaping | references/troubleshoot-performance.md |
| Consumer lag increasing, rebalance storms, stuck consumer groups | references/troubleshoot-consumer-lag.md |
| Disk filling up, retention planning, tiered storage, EBS scaling | references/manage-storage.md |
| Setting up monitoring level, dashboards, recommended CloudWatch alarms | references/monitor-and-alarm.md |
| Rolling restart impact, patching, Kafka version upgrades, maintenance resilience | references/maintenance-operations.md |
| Producer / consumer configuration, IAM / SCRAM / TLS auth for clients | references/configure-clients.md |
For sizing questions (broker count, instance type choice, monthly cost), refer the user to the Amazon MSK best practices — right-size your cluster documentation. Do not size from memory.
Operational Review Workflow
Use this workflow when the user asks for a review, audit, health check, or assessment of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report.
Follow the steps in order for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check.
Step 1 — Identify Target Clusters
Ask the user which MSK clusters to review. Accept any of:
- Specific cluster names or ARNs and regions
- "all clusters" in specific regions
- "all MSK clusters in all regions"
If no scope is given, default to all configured account regions. Enumerate
clusters with aws kafka list-clusters-v2 per region.
Step 2 — Determine Broker Type Per Cluster
For each cluster:
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard
brokers (kafka.*) and Express brokers (express.*) require different checks
in the later steps — some metrics only exist on one type.
Step 3 — Collect Cluster Configuration
For each cluster, gather:
aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn> # last 30 days
aws kafka describe-configuration-revision \
--arn <configuration-arn> --revision <revision> # if a custom config is applied
Capture:
- Cluster: state, Kafka version, number of broker nodes, AZ distribution
(
ZoneIds), storage mode (EBS / Tiered), current version. - Broker: instance type, EBS volume size (Standard), provisioned throughput (if any).
- Encryption:
EncryptionInTransit.ClientBroker(TLS / TLS_PLAINTEXT / PLAINTEXT),EncryptionInTransit.InCluster,EncryptionAtRest.DataVolumeKMSKeyId. - Auth:
ClientAuthentication.Sasl.Iam.Enabled,ClientAuthentication.Sasl.Scram.Enabled,ClientAuthentication.Tls.Enabled,ClientAuthentication.Unauthenticated.Enabled. - Public access:
BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type. - Monitoring level:
EnhancedMonitoring(DEFAULT/PER_BROKER/PER_TOPIC_PER_BROKER/PER_TOPIC_PER_PARTITION). - Logging:
LoggingInfo.BrokerLogs(CloudWatch / S3 / Firehose destinations and theirEnabledflags). - Open monitoring:
OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker,NodeExporter.EnabledInBroker. - Recent operations: From
list-cluster-operations-v2, note anySECURITY_PATCHING,BROKER_UPDATE,UPDATE_CLUSTER_CONFIGURATION,UPDATE_STORAGE, orUPDATE_MONITORINGevents in the review window.
Step 4 — Detect Monitoring Level and Gaps
The EnhancedMonitoring value from Step 3 determines which checks are
available. At DEFAULT, most per-broker health metrics are still available
(CPU, disk, network, partitions, connections, memory, TrafficShaping), but the
following checks are not possible without upgrading:
ReplicationBytesInPerSec/ReplicationBytesOutPerSec(inter-broker replication load)RequestHandlerAvgIdlePercent/NetworkProcessorAvgIdlePercent(thread pool saturation)BwInAllowanceExceeded/BwOutAllowanceExceeded(detailed bandwidth breaches)VolumeQueueLength(EBS I/O queue depth — Standard only)VolumeReadBytes/VolumeWriteBytes(EBS throughput utilization — Standard only)- IAM connection metrics (
IAMTooManyConnections)
Record the monitoring level and list any dimensions that will be scored
partially or skipped. Recommend upgrading to PER_BROKER if any dimension is
degraded by the current level.
Step 5 — Collect CloudWatch Metrics (7-Day Historical)
Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for
per-broker metrics; Cluster Name and Consumer Group and Topic for
consumer lag; Cluster Name only for cluster-wide metrics.
Use one cloudwatch.GetMetricData batch per cluster where possible.
Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.
5.1 Cluster-wide (both broker types)
| Metric | Stat | Purpose |
|---|---|---|
ActiveControllerCount |
Sum | Must be exactly 1 |
OfflinePartitionsCount |
Maximum | Must be 0 |
GlobalPartitionCount |
Maximum | Total leader partitions |
GlobalTopicCount |
Maximum | Total topics |
5.2 Per-broker — Standard (kafka.*)
| Metric | Stat | Threshold |
|---|---|---|
CpuUser + CpuSystem |
Average, Maximum | < 60% avg |
KafkaDataLogsDiskUsed |
Average, Maximum | < 70% avg, < 85% max |
PartitionCount |
Maximum | ≤ recommended limit for broker size |
LeaderCount |
Maximum | Compare across brokers; skew < 10% |
UnderReplicatedPartitions |
Maximum | 0 in steady state |
UnderMinIsrPartitionCount |
Maximum | 0 |
BytesInPerSec / BytesOutPerSec |
Average, Maximum | vs baseline bandwidth |
ConnectionCount |
Average, Maximum | Compare across brokers |
HeapMemoryAfterGC |
Maximum | < 60% |
TrafficShaping |
Sum | Must be 0 |
NetworkRxDropped / NetworkTxDropped / NetworkRxErrors / NetworkTxErrors |
Sum | Must be 0 |
ReplicationBytesInPerSec / ReplicationBytesOutPerSec |
Average | Requires PER_BROKER |
RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent |
Average | > 30% (PER_BROKER) |
VolumeQueueLength |
Average, Maximum | Avg < 1 (PER_BROKER) |
VolumeReadBytes + VolumeWriteBytes |
Sum | vs EBS baseline throughput (PER_BROKER) |
5.3 Per-broker — Express (express.*)
| Metric | Stat | Threshold |
|---|---|---|
CpuUser + CpuSystem |
Average, Maximum | < 60% avg |
StorageUsed |
Maximum | Fully managed — flag if trending against per-broker quota |
PartitionCount |
Maximum | ≤ recommended limit for broker size |
LeaderCount |
Maximum | Compare across brokers |
BytesInPerSec / BytesOutPerSec |
Average, Maximum | vs Express per-broker ingress/egress quotas |
ProduceThrottleTime / FetchThrottleTime |
Maximum | Should be 0 |
ClientConnectionCount |
Average, Maximum | vs listener quota (see references/monitor-and-alarm.md) |
Express brokers do NOT emit: UnderReplicatedPartitions,
UnderMinIsrPartitionCount, HeapMemoryAfterGC, TrafficShaping, Volume*
metrics, KafkaDataLogsDiskUsed, ProduceMessageConversionsPerSec,
FetchMessageConversionsPerSec.
5.4 Consumer lag (both broker types)
Per consumer group (identified from the customer or from the broker logs):
| Metric | Stat | Notes |
|---|---|---|
SumOffsetLag per (Consumer Group, Topic) |
Maximum | DEFAULT level |
MaxOffsetLag per (Consumer Group, Topic) |
Maximum | DEFAULT level |
EstimatedMaxTimeLag per (Consumer Group, Topic) |
Maximum | DEFAULT level |
OffsetLag per (Consumer Group, Topic, Partition) |
Maximum | Requires PER_TOPIC_PER_PARTITION |
Step 6 — Alarm Coverage
Collect existing alarms:
aws cloudwatch describe-alarms --namespace AWS/Kafka
Compare against the 13-alarm recommended set (details in
references/monitor-and-alarm.md) and produce a coverage table (present / missing / firing).
Step 7 — Analyze Against Best Practices
Assign a severity to every finding: CRITICAL / HIGH / MEDIUM / LOW / INFO (see Severity Definitions later in this file).
Evaluate across seven dimensions. Some checks are skipped for Express — noted inline.
7.1 Cluster Configuration
- Cluster state is
ACTIVE.MAINTENANCE/UPDATINGis transient; anything else is a finding. - Deployed across 3 AZs (Standard: broker count multiple of AZ count).
- Kafka version within N-2 of the latest supported.
- Enhanced monitoring at
PER_BROKERor higher —DEFAULT→ MEDIUM. - Storage mode: Tiered storage enabled for topics with long retention (Standard).
7.2 Security
- Encryption in transit:
TLS.TLS_PLAINTEXT→ MEDIUM (prod: HIGH).PLAINTEXT→ CRITICAL. - Encryption at rest: enabled with KMS. Customer-managed KMS preferred over AWS-managed for regulated workloads.
- Authentication: at least one of IAM / SCRAM / mTLS enabled.
Unauthenticated.Enabled=truein production → CRITICAL. - Public access:
SERVICE_PROVIDED_EIPSin production → HIGH.
7.3 Logging & Monitoring
- Broker logs enabled to at least one destination (CloudWatch / S3 / Firehose). All disabled → HIGH.
- Open monitoring (Prometheus JMX + Node exporter) — informational.
- Alarm coverage from Step 6: missing alarms → MEDIUM each; missing critical alarms (Active Controller, Offline Partitions, Disk, CPU) → HIGH.
- Any alarm currently in
ALARMstate → HIGH (surface in report header).
7.4 Partition Health
PartitionCountper broker ≤ recommended limit for the broker instance type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks update operations).LeaderCountvariance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH.UnderReplicatedPartitions> 0 sustained (Standard only) → HIGH. Transient during aSECURITY_PATCHING/BROKER_UPDATEoperation from Step 3 → INFO — do NOT flag.UnderMinIsrPartitionCount> 0 (Standard only) → CRITICAL.- Config-level ISR risk: topics with
min.insync.replicas >= replication.factorare a configuration error the CloudWatch metric will not surface. If the user has provided topic configs, flag any such topic as HIGH.
7.5 Compute Health
CpuUser + CpuSystemavg < 60%. 60-70% → MEDIUM. > 70% → HIGH.- CPU variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
HeapMemoryAfterGC< 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH.RequestHandlerAvgIdlePercent> 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.NetworkProcessorAvgIdlePercent> 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.
7.6 Network Health
TrafficShaping= 0 (Standard only). Any non-zero → HIGH.BwInAllowanceExceeded/BwOutAllowanceExceeded= 0 (PER_BROKER). Non-zero → HIGH.- Throughput variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH.
- Total per-broker throughput < 60% of baseline bandwidth (see
references/troubleshoot-performance.mdfor baseline table). 60-70% → MEDIUM. > 70% → HIGH. - Express:
ProduceThrottleTime/FetchThrottleTime> 0 → HIGH.
7.7 Storage Health
Standard only (Express storage is managed — Express clusters only get the
StorageUsed check):
KafkaDataLogsDiskUsed< 70%. 70-85% → HIGH. > 85% → CRITICAL.VolumeQueueLengthavg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH.- EBS auto-scaling configured, or disk headroom > 30% → PASS. No auto-scaling AND disk > 50% → MEDIUM. No auto-scaling AND disk > 70% → HIGH.
- EBS throughput (
VolumeReadBytes + VolumeWriteBytes) < 60% of instance baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH.
Express:
StorageUsedper broker vs Express per-broker storage quota (see MSK Express quotas). Approaching quota → MEDIUM.
Step 8 — Generate the Report
Generate a separate report artifact per cluster reviewed.
Artifact naming: msk-review-<cluster-name>-<YYYY-MM-DD>.md
Example: msk-review-prod-orders-2026-04-29.md
Report structure:
Report Header
# MSK Operational Review — <cluster-name>
Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD>
Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n>
Kafka Version: <version> | Monitoring Level: <level>
Executive Summary
- Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL
- Finding counts by severity
- Top 3 CRITICAL/HIGH items
Configuration Snapshot
| Item | Value | | Cluster state / version | … | | Broker type / instance / count / AZs | … | | Storage | mode, size (Standard), provisioned throughput (if any) | | Encryption | in-transit, in-cluster, at-rest (KMS) | | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags | | Public access | DISABLED / SERVICE_PROVIDED_EIPS | | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION | | Logging | destinations enabled |
Findings by Dimension
For each of the 7 dimensions (7.1-7.7):
| # | Finding | Severity | Current State | Recommendation |
If a dimension was skipped or partial due to monitoring level or broker type, say so explicitly in a note above the table for that dimension.
CloudWatch Metrics (7-Day)
| Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding |
Alarm Coverage
| # | Recommended Alarm | Metric | Threshold | Priority | Status |
Also list alarms currently in ALARM state with timestamps.
Recent Cluster Operations (Last 30 Days)
From list-cluster-operations-v2 in Step 3:
| Operation Type | Start Time | End Time | State |
Call out any operations that would explain transient metric anomalies.
Priority Matrix
| # | Finding | Severity | Dimension | Effort | Impact |
Sorted by severity.
Next Steps
- Immediate (CRITICAL / HIGH — 7 days)
- Short-term (MEDIUM — 30 days)
- Long-term (LOW — 90 days)
Appendix — Reference Links
- Amazon MSK best practices — Standard brokers
- Amazon MSK best practices — Express brokers
- Amazon MSK — Monitoring an MSK cluster
- Amazon MSK — CloudWatch metrics for Provisioned clusters
- Amazon MSK — Right-size your cluster
- Amazon MSK — Service quotas
Common CLI Recipes
Describe cluster:
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
List brokers:
aws kafka list-nodes --cluster-arn <cluster-arn>
Get bootstrap brokers:
aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn>
List recent cluster operations (patching, config updates, storage changes):
aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn>
Expand Standard broker storage — Operator-run (recommend, do not execute):
aws kafka update-broker-storage \
--cluster-arn <cluster-arn> \
--current-version <cluster-version> \
--target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]'
Get a CloudWatch metric (example: CpuUser per broker):
aws cloudwatch get-metric-statistics \
--namespace AWS/Kafka \
--metric-name CpuUser \
--dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \
--start-time <start> --end-time <end> --period 300 --statistics Average
Create a cluster configuration (server.properties) — Operator-run (recommend, do not execute):
The --server-properties argument MUST be a real Kafka properties file with
one key=value per line, separated by actual newline characters — NOT the
literal two-character escape sequence \n. The MSK API accepts the bytes as-is;
if you pass "k1=v1\nk2=v2" as a single string with escaped newlines, MSK
stores ONE invalid property line and the cluster will fail to apply it.
Recommended pattern: write the properties to a local file with real newlines,
then pass it via fileb:// so the CLI uploads the raw bytes verbatim. Verify by
reading the revision back with describe-configuration-revision and
base64-decoding ServerProperties — you should see one property per line.
cat > server.properties <<'EOF'
auto.create.topics.enable=false
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
num.io.threads=32
num.network.threads=16
log.retention.hours=168
EOF
aws kafka create-configuration \
--name <config-name> \
--kafka-versions "3.6.0" \
--server-properties fileb://server.properties
Common Error Reference
| Error | Cause | Fix |
|---|---|---|
aws kafka update-broker-storage returns "storage is optimizing" |
Previous storage expansion still in cool-down (minimum 6 hours) | Wait for optimization to complete. Check cluster state with describe-cluster-v2. |
ClusterState is MAINTENANCE |
Standard brokers undergoing patching. Express brokers stay ACTIVE during maintenance. |
Wait for cluster to return to ACTIVE. Do not perform update operations during MAINTENANCE. |
Consumer receives GROUP_COORDINATOR_NOT_AVAILABLE |
Coordinator broker is temporarily unavailable during rolling restart or overloaded | Retry with backoff. Check if maintenance is in progress via list-cluster-operations-v2. |
NotEnoughReplicasException on produce |
Fewer brokers in ISR than min.insync.replicas (default: 2) |
Check UnderReplicatedPartitions (Standard only). For Express, check ProduceThrottleTime and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower min.insync.replicas to work around this. |
Severity Definitions (for review-style reports)
| Severity | Definition | SLA |
|---|---|---|
| CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours |
| HIGH | Significant gap that could lead to incidents | Fix within 1 week |
| MEDIUM | Notable improvement opportunity | Plan within 30 days |
| LOW | Minor optimization or hardening | Address when convenient |
| INFO | Observation, no action required | N/A |
Additional Resources
Files (tools-for-devops-agent)
-
evals
-
benchmark.json 26.6 KB
Loading… -
evals.json 3.1 KB
Loading… -
eval_queries.json 1 KB
Loading… -
report.json 1.6 KB
Loading… -
trigger_report.json 3.4 KB
Loading…
-
-
references
-
configure-clients.md 6.7 KB
Loading… -
maintenance-operations.md 12.3 KB
Loading… -
manage-storage.md 7.1 KB
Loading… -
monitor-and-alarm.md 13.3 KB
Loading… -
troubleshoot-consumer-lag.md 11.1 KB
Loading… -
troubleshoot-performance.md 9.1 KB
Loading…
-
-
.skilleval.yaml 77 B
Loading… -
CHANGELOG.md 1.9 KB
Loading… -
README.md 8 KB
# MSK Operations — AWS DevOps Agent Skill An Amazon MSK Provisioned operations, troubleshooting, and health-assessment skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html). Covers Standard and Express brokers and both proactive operational reviews and ad-hoc incident response (performance, consumer lag, storage, maintenance, client tuning). > ⚠️ **Non-production disclaimer.** This skill is sample code, not intended for > production use without additional review and testing. Users should validate in > a non-production environment first. ## Purpose Give AWS DevOps Agent the domain knowledge to answer MSK questions accurately without falling back on training data (which routinely conflates Standard and Express broker behavior). The skill activates in two shapes: 1. **Operational review** — assess an MSK cluster's health against best practices and produce a prioritized findings report. 2. **Ad-hoc troubleshooting** — investigate a specific MSK symptom (high CPU, consumer lag, disk full, TrafficShaping, unexpected broker restart, etc.) and recommend the correct next action. ## Key Capabilities - Determine broker type (Standard vs Express) and route to the correct diagnostic path — many CloudWatch metrics and behaviors differ between the two. - Troubleshoot broker performance issues (`CpuUser`, `CpuSystem`, `RequestHandlerAvgIdlePercent`, `NetworkProcessorAvgIdlePercent`, `ProduceTotalTimeMsMean`, `TrafficShaping`). - Diagnose consumer lag using `SumOffsetLag`, `MaxOffsetLag`, and (when `PER_TOPIC_PER_PARTITION` is enabled) per-partition `OffsetLag`. - Manage broker storage — EBS scaling for Standard, `StorageUsed` monitoring for Express, tiered storage, retention planning. - Recommend CloudWatch alarms in the `AWS/Kafka` namespace and validate that the right monitoring level is enabled. - Explain rolling restart, patching, and version upgrade behavior; call out operations that are unsafe during `UnderReplicatedPartitions > 0`. - Advise on Kafka client (producer / consumer) configuration — batch sizing, `linger.ms`, compression, `acks`, `min.insync.replicas` — and on authentication choices (IAM, SCRAM, mTLS). ## Prerequisites ### IAM permissions The AWS DevOps Agent's primary cloud-source role needs read access to MSK and CloudWatch. All calls except `kafka:GetBootstrapBrokers` are covered by `AIDevOpsAgentAccessPolicy`. `kafka:GetBootstrapBrokers` is granted by the opt-in `EnableMskOperations` parameter (default `true`) in [`cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml`](https://github.com/aws/tools-for-devops-agent/blob/main/cloudformation/devops-agent-skill-policies/devops-agent-skill-policies.yaml). The full set of actions the skill uses in practice: ``` kafka:DescribeClusterV2 kafka:ListClustersV2 kafka:ListNodes kafka:GetBootstrapBrokers kafka:ListClusterOperationsV2 kafka:DescribeConfigurationRevision kafka:ListConfigurations cloudwatch:GetMetricData cloudwatch:GetMetricStatistics cloudwatch:ListMetrics cloudwatch:DescribeAlarms logs:DescribeLogGroups logs:FilterLogEvents ``` Write actions (`UpdateBrokerStorage`, `CreateConfiguration`, `PutMetricAlarm`, etc.) are only recommended in the skill output — the operator is expected to run them after review. ### Cluster access - The MSK cluster must live in an account configured as a cloud source in your Agent Space. - No data-plane / Kafka-protocol access is required. Everything the skill uses comes from the AWS control-plane APIs and CloudWatch. ## Limitations - Covers **MSK Provisioned only** — does **not** cover MSK Connect, MSK Serverless, or MSK Replicator. - No Kafka data-plane visibility. The skill cannot read topic contents, run `kafka-consumer-groups.sh`, or otherwise interact with the Kafka protocol. - Some checks require monitoring level `PER_BROKER` or higher (thread pool idle %, `VolumeQueueLength`, `BwInAllowanceExceeded`, IAM connection metrics). At `DEFAULT` level the skill will explicitly note which checks are limited. - Sizing questions (broker count, instance type choice, monthly cost) are redirected to the AWS documentation — the skill does not include a bundled sizing calculator. ## Agent Types This skill is intended for the following agent types (selected in the Operator Web App at upload time): - **Chat tasks** — conversational invocation in Chat ("my MSK cluster is latent", "run an MSK health check on `prod-cluster`", "why did broker 2 restart last night?"). Select **All agents** instead if you want the skill available to all agent types. ## Uploading to AWS DevOps Agent > Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) ### 1. Package the skill From the `skills/` directory in this repo: ```bash cd skills zip -r msk-operations.zip msk-operations/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*' ``` The resulting `msk-operations.zip` contains: ``` msk-operations/ ├── SKILL.md └── references/ ├── troubleshoot-performance.md ├── troubleshoot-consumer-lag.md ├── manage-storage.md ├── monitor-and-alarm.md ├── maintenance-operations.md └── configure-clients.md ``` Constraints (enforced at upload time): - Total zip size ≤ **6 MB**. - `SKILL.md` is required and must include `name` and `description` frontmatter. - A `scripts/` directory is **not** allowed — uploads containing scripts are rejected. ### 2. Upload via the Operator Web App 1. Navigate to the **Skills** page in your Agent Space Operator Web App. 2. Click **Add skill** → **Upload skill**. 3. Drag and drop `msk-operations.zip` (or browse to it). 4. Select agent types: **Chat tasks** (or leave **All agents** to make it available to all agent types). 5. Review the validation results. 6. Click **Upload**. ## How to Use This Skill In DevOps Agent Chat, use natural language. You do NOT need to mention the skill name — the agent activates it based on the MSK / Kafka triggers in the skill description. ### Ad-hoc troubleshooting prompts - *"My MSK cluster `prod-orders` is showing high produce latency in `us-east-1`. What should I check?"* - *"Consumer group `payments-consumer` on cluster `prod-orders` has growing lag. Diagnose it."* - *"UnderReplicatedPartitions is > 0 on broker 2 of `prod-orders`. What's the safe next step?"* - *"Why did broker 3 of `prod-orders` restart last night?"* - *"Storage on `prod-orders` is at 82%. What are my options?"* - *"TrafficShaping is firing on all three brokers of `prod-orders`."* ### Operational-review prompts - *"Run an MSK operational review on cluster `prod-orders`."* - *"Audit `prod-orders` against MSK best practices."* - *"Are the CloudWatch alarms on `prod-orders` complete? Which ones am I missing?"* - *"Health-check every MSK Provisioned cluster in `us-east-1`."* ### Client tuning prompts - *"My Java Kafka producer is getting `NotEnoughReplicasException` when writing to MSK. What producer settings should I check?"* - *"What should `linger.ms` and `batch.size` be for a producer writing 200 MB/s to a Standard broker cluster?"* ## Skill Contents ``` msk-operations/ ├── SKILL.md ├── README.md ├── CHANGELOG.md ├── .skilleval.yaml ├── evals/ │ ├── evals.json │ └── eval_queries.json └── references/ ├── troubleshoot-performance.md ├── troubleshoot-consumer-lag.md ├── manage-storage.md ├── monitor-and-alarm.md ├── maintenance-operations.md └── configure-clients.md ``` `README.md`, `CHANGELOG.md`, `.skilleval.yaml`, and `evals/` are for repo maintenance and are not required at the cluster — the `zip` command above excludes them from the upload artifact. -
SKILL.md 26 KB
--- name: msk-operations description: Amazon MSK Provisioned operations, troubleshooting, and health assessment for Standard and Express brokers. Use whenever the user mentions Amazon MSK, MSK Provisioned, MSK Standard/Express brokers, Apache Kafka on AWS, `kafka.*` / `express.*` instance types, or the `AWS/Kafka` CloudWatch namespace. Covers MSK performance issues (high CPU, produce/fetch latency, TrafficShaping), consumer lag, storage/EBS issues, rolling restarts, Kafka version upgrades, SECURITY_PATCHING, BROKER_UPDATE, CloudWatch alarm design, Kafka client (producer/consumer) tuning, under-replicated partitions, unexpected broker reboots, and full MSK operational reviews / health checks / best-practices audits. Do NOT use for MSK Connect, MSK Serverless, or MSK Replicator. Do NOT use for authoring CloudFormation, CDK, or Terraform templates. Do NOT use for other AWS services (RDS, Aurora, S3, DynamoDB, Kinesis, Lambda, EC2) unless MSK is explicitly named. metadata: author: kjjanaki version: "1.0.5" aws-devops-agent-skills.agent-types: "CHAT" aws-devops-agent-skills.aws-services: "Amazon MSK" aws-devops-agent-skills.technical-domains: "Analytics" --- # Amazon MSK Operations Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots). ## When to Use Activate this skill when the user asks to: - Review, audit, or assess an MSK cluster for best practices, health, or operational readiness. - Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency, consumer lag, broker storage running out, TrafficShaping events, under-replicated partitions, or an unexpected broker restart. - Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch alarms and dashboards, understand the metrics available in the `AWS/Kafka` namespace. - Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security patching, broker instance type change. - Advise on Kafka client (producer / consumer) configuration when the client is connecting to an MSK cluster. Do **not** activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces. ## Broker Type Determination Determine the broker type first — many checks differ between Standard and Express. ``` aws kafka describe-cluster-v2 --cluster-arn <cluster-arn> ``` Check `ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType`: - Starts with `kafka.` (e.g. `kafka.m5.large`, `kafka.m7g.xlarge`) → **Standard broker**. - Starts with `express.` (e.g. `express.m7g.large`) → **Express broker**. ### Key Standard vs Express differences **Standard brokers** use customer-managed EBS volumes for storage. You choose instance types (`kafka.m5.*`, `kafka.m7g.*`), provision EBS, and manage storage scaling. Standard brokers have scheduled maintenance windows. **Express brokers** provide fully managed, pay-as-you-go storage with no EBS provisioning. Instance types are prefixed with `express.m7g.*`. Express brokers offer up to 3× more throughput per broker than Standard, and have no maintenance windows. Express enforces a fixed replication factor of 3 and `min.insync.replicas=2` — you cannot create topics with RF=1. ## Critical Warnings - **This skill is read-only.** Every command in this file and in `references/` that mutates cluster state — `update-broker-storage`, `create-configuration`, `update-cluster-configuration`, `update-monitoring`, `put-metric-alarm`, `reboot-broker`, and any partition reassignment — is a **recommendation for the operator to run after review**. Present these as proposed remediations with expected impact and preconditions; do NOT execute them, and do NOT imply that the agent will run them. - **NEVER reboot brokers while `UnderReplicatedPartitions` > 0** (Standard only — Express brokers do not emit URP). This risks data loss and extended outages. - **NEVER recommend partition reassignment without first checking replication status.** Reassignment during URP compounds the problem. - **`linger.ms=0` is the #1 cause of "high CPU" on MSK.** ALWAYS check client batch configuration before recommending broker scaling. - **EBS throughput ceilings are invisible in Kafka metrics** — ALWAYS check EBS volume metrics (`VolumeReadBytes`, `VolumeWriteBytes`, `VolumeQueueLength`) when diagnosing Standard broker latency. - **Express brokers have NO customer-managed EBS** — do NOT recommend EBS expansion or provisioned EBS throughput for Express clusters. - **Express brokers enforce fixed RF=3 and `min.insync.replicas=2`** — do NOT attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard brokers. ## Quick Diagnostics These five checks cover the most common MSK issues. Use them before loading a reference file. 1. **`CpuUser + CpuSystem` > 60%**: Check `RequestHandlerAvgIdlePercent` (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check client `batch.size` and `linger.ms` before recommending scaling. 2. **`KafkaDataLogsDiskUsed` > 85%** (Standard only): **Recommend to the operator** that EBS be expanded via `aws kafka update-broker-storage` (do not execute). Identify high-growth topics via per-topic `BytesInPerSec` to size the increase. Express clusters use `StorageUsed` metric instead and storage is fully managed. 3. **`UnderReplicatedPartitions` > 0** (Standard only): Check if a maintenance operation or broker restart is in progress. If URP is decreasing, wait for recovery. Do NOT restart brokers or reassign partitions during URP. Express brokers do not emit this metric — monitor `ProduceThrottleTime`, `FetchThrottleTime`, and consumer lag instead. 4. **Consumer `OffsetLag` / `MaxOffsetLag` increasing**: Determine if broker-side (high `ProduceTotalTimeMsMean`, CPU saturation) or client-side (slow processing, insufficient consumers). Per-partition lag from `PER_TOPIC_PER_PARTITION` monitoring level helps isolate hot partitions. 5. **`BytesInPerSec` near throughput ceiling**: For Standard, check EBS volume type and calculate: `BytesInPerSec × ReplicationFactor` vs volume throughput limit. For Express, check against the per-broker sustained performance limits in the MSK quotas. ## Which Reference Do You Need? Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory. | Customer Intent | Reference | |---|---| | High CPU, high produce/fetch latency, slow cluster, TrafficShaping | `references/troubleshoot-performance.md` | | Consumer lag increasing, rebalance storms, stuck consumer groups | `references/troubleshoot-consumer-lag.md` | | Disk filling up, retention planning, tiered storage, EBS scaling | `references/manage-storage.md` | | Setting up monitoring level, dashboards, recommended CloudWatch alarms | `references/monitor-and-alarm.md` | | Rolling restart impact, patching, Kafka version upgrades, maintenance resilience | `references/maintenance-operations.md` | | Producer / consumer configuration, IAM / SCRAM / TLS auth for clients | `references/configure-clients.md` | For sizing questions (broker count, instance type choice, monthly cost), refer the user to the [Amazon MSK best practices — right-size your cluster](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-right-size-cluster) documentation. Do not size from memory. ## Operational Review Workflow Use this workflow when the user asks for a **review, audit, health check, or assessment** of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report. Follow the steps **in order** for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check. ### Step 1 — Identify Target Clusters Ask the user which MSK clusters to review. Accept any of: - Specific cluster names or ARNs and regions - "all clusters" in specific regions - "all MSK clusters in all regions" If no scope is given, default to all configured account regions. Enumerate clusters with `aws kafka list-clusters-v2` per region. ### Step 2 — Determine Broker Type Per Cluster For each cluster: ``` aws kafka describe-cluster-v2 --cluster-arn <cluster-arn> ``` Read `ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType`. Standard brokers (`kafka.*`) and Express brokers (`express.*`) require different checks in the later steps — some metrics only exist on one type. ### Step 3 — Collect Cluster Configuration For each cluster, gather: ``` aws kafka describe-cluster-v2 --cluster-arn <arn> aws kafka list-nodes --cluster-arn <arn> aws kafka get-bootstrap-brokers --cluster-arn <arn> aws kafka list-cluster-operations-v2 --cluster-arn <arn> # last 30 days aws kafka describe-configuration-revision \ --arn <configuration-arn> --revision <revision> # if a custom config is applied ``` Capture: - **Cluster:** state, Kafka version, number of broker nodes, AZ distribution (`ZoneIds`), storage mode (EBS / Tiered), current version. - **Broker:** instance type, EBS volume size (Standard), provisioned throughput (if any). - **Encryption:** `EncryptionInTransit.ClientBroker` (TLS / TLS_PLAINTEXT / PLAINTEXT), `EncryptionInTransit.InCluster`, `EncryptionAtRest.DataVolumeKMSKeyId`. - **Auth:** `ClientAuthentication.Sasl.Iam.Enabled`, `ClientAuthentication.Sasl.Scram.Enabled`, `ClientAuthentication.Tls.Enabled`, `ClientAuthentication.Unauthenticated.Enabled`. - **Public access:** `BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type`. - **Monitoring level:** `EnhancedMonitoring` (`DEFAULT` / `PER_BROKER` / `PER_TOPIC_PER_BROKER` / `PER_TOPIC_PER_PARTITION`). - **Logging:** `LoggingInfo.BrokerLogs` (CloudWatch / S3 / Firehose destinations and their `Enabled` flags). - **Open monitoring:** `OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker`, `NodeExporter.EnabledInBroker`. - **Recent operations:** From `list-cluster-operations-v2`, note any `SECURITY_PATCHING`, `BROKER_UPDATE`, `UPDATE_CLUSTER_CONFIGURATION`, `UPDATE_STORAGE`, or `UPDATE_MONITORING` events in the review window. ### Step 4 — Detect Monitoring Level and Gaps The `EnhancedMonitoring` value from Step 3 determines which checks are available. At `DEFAULT`, most per-broker health metrics are still available (CPU, disk, network, partitions, connections, memory, TrafficShaping), but the following checks are **not possible without upgrading**: - `ReplicationBytesInPerSec` / `ReplicationBytesOutPerSec` (inter-broker replication load) - `RequestHandlerAvgIdlePercent` / `NetworkProcessorAvgIdlePercent` (thread pool saturation) - `BwInAllowanceExceeded` / `BwOutAllowanceExceeded` (detailed bandwidth breaches) - `VolumeQueueLength` (EBS I/O queue depth — Standard only) - `VolumeReadBytes` / `VolumeWriteBytes` (EBS throughput utilization — Standard only) - IAM connection metrics (`IAMTooManyConnections`) Record the monitoring level and list any dimensions that will be scored partially or skipped. Recommend upgrading to `PER_BROKER` if any dimension is degraded by the current level. ### Step 5 — Collect CloudWatch Metrics (7-Day Historical) Namespace: `AWS/Kafka`. Dimensions: `Cluster Name` and `Broker ID` for per-broker metrics; `Cluster Name` and `Consumer Group` and `Topic` for consumer lag; `Cluster Name` only for cluster-wide metrics. Use one `cloudwatch.GetMetricData` batch per cluster where possible. `Period: 3600` (1 hour). `StartTime`: 7 days ago. `EndTime`: now. #### 5.1 Cluster-wide (both broker types) | Metric | Stat | Purpose | |---|---|---| | `ActiveControllerCount` | Sum | Must be exactly 1 | | `OfflinePartitionsCount` | Maximum | Must be 0 | | `GlobalPartitionCount` | Maximum | Total leader partitions | | `GlobalTopicCount` | Maximum | Total topics | #### 5.2 Per-broker — Standard (`kafka.*`) | Metric | Stat | Threshold | |---|---|---| | `CpuUser` + `CpuSystem` | Average, Maximum | < 60% avg | | `KafkaDataLogsDiskUsed` | Average, Maximum | < 70% avg, < 85% max | | `PartitionCount` | Maximum | ≤ recommended limit for broker size | | `LeaderCount` | Maximum | Compare across brokers; skew < 10% | | `UnderReplicatedPartitions` | Maximum | 0 in steady state | | `UnderMinIsrPartitionCount` | Maximum | 0 | | `BytesInPerSec` / `BytesOutPerSec` | Average, Maximum | vs baseline bandwidth | | `ConnectionCount` | Average, Maximum | Compare across brokers | | `HeapMemoryAfterGC` | Maximum | < 60% | | `TrafficShaping` | Sum | Must be 0 | | `NetworkRxDropped` / `NetworkTxDropped` / `NetworkRxErrors` / `NetworkTxErrors` | Sum | Must be 0 | | `ReplicationBytesInPerSec` / `ReplicationBytesOutPerSec` | Average | Requires PER_BROKER | | `RequestHandlerAvgIdlePercent` / `NetworkProcessorAvgIdlePercent` | Average | > 30% (PER_BROKER) | | `VolumeQueueLength` | Average, Maximum | Avg < 1 (PER_BROKER) | | `VolumeReadBytes` + `VolumeWriteBytes` | Sum | vs EBS baseline throughput (PER_BROKER) | #### 5.3 Per-broker — Express (`express.*`) | Metric | Stat | Threshold | |---|---|---| | `CpuUser` + `CpuSystem` | Average, Maximum | < 60% avg | | `StorageUsed` | Maximum | Fully managed — flag if trending against per-broker quota | | `PartitionCount` | Maximum | ≤ recommended limit for broker size | | `LeaderCount` | Maximum | Compare across brokers | | `BytesInPerSec` / `BytesOutPerSec` | Average, Maximum | vs Express per-broker ingress/egress quotas | | `ProduceThrottleTime` / `FetchThrottleTime` | Maximum | Should be 0 | | `ClientConnectionCount` | Average, Maximum | vs listener quota (see `references/monitor-and-alarm.md`) | **Express brokers do NOT emit:** `UnderReplicatedPartitions`, `UnderMinIsrPartitionCount`, `HeapMemoryAfterGC`, `TrafficShaping`, `Volume*` metrics, `KafkaDataLogsDiskUsed`, `ProduceMessageConversionsPerSec`, `FetchMessageConversionsPerSec`. #### 5.4 Consumer lag (both broker types) Per consumer group (identified from the customer or from the broker logs): | Metric | Stat | Notes | |---|---|---| | `SumOffsetLag` per (Consumer Group, Topic) | Maximum | DEFAULT level | | `MaxOffsetLag` per (Consumer Group, Topic) | Maximum | DEFAULT level | | `EstimatedMaxTimeLag` per (Consumer Group, Topic) | Maximum | DEFAULT level | | `OffsetLag` per (Consumer Group, Topic, Partition) | Maximum | Requires PER_TOPIC_PER_PARTITION | ### Step 6 — Alarm Coverage Collect existing alarms: ``` aws cloudwatch describe-alarms --namespace AWS/Kafka ``` Compare against the 13-alarm recommended set (details in `references/monitor-and-alarm.md`) and produce a coverage table (present / missing / firing). ### Step 7 — Analyze Against Best Practices Assign a severity to every finding: **CRITICAL / HIGH / MEDIUM / LOW / INFO** (see Severity Definitions later in this file). Evaluate across seven dimensions. Some checks are skipped for Express — noted inline. #### 7.1 Cluster Configuration - Cluster state is `ACTIVE`. `MAINTENANCE` / `UPDATING` is transient; anything else is a finding. - Deployed across **3 AZs** (Standard: broker count multiple of AZ count). - Kafka version within N-2 of the latest supported. - Enhanced monitoring at `PER_BROKER` or higher — `DEFAULT` → MEDIUM. - Storage mode: Tiered storage enabled for topics with long retention (Standard). #### 7.2 Security - Encryption in transit: `TLS`. `TLS_PLAINTEXT` → MEDIUM (prod: HIGH). `PLAINTEXT` → CRITICAL. - Encryption at rest: enabled with KMS. Customer-managed KMS preferred over AWS-managed for regulated workloads. - Authentication: at least one of IAM / SCRAM / mTLS enabled. `Unauthenticated.Enabled=true` in production → CRITICAL. - Public access: `SERVICE_PROVIDED_EIPS` in production → HIGH. #### 7.3 Logging & Monitoring - Broker logs enabled to at least one destination (CloudWatch / S3 / Firehose). All disabled → HIGH. - Open monitoring (Prometheus JMX + Node exporter) — informational. - Alarm coverage from Step 6: missing alarms → MEDIUM each; missing critical alarms (Active Controller, Offline Partitions, Disk, CPU) → HIGH. - Any alarm currently in `ALARM` state → HIGH (surface in report header). #### 7.4 Partition Health - `PartitionCount` per broker ≤ recommended limit for the broker instance type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks update operations). - `LeaderCount` variance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH. - `UnderReplicatedPartitions` > 0 sustained (Standard only) → HIGH. Transient during a `SECURITY_PATCHING` / `BROKER_UPDATE` operation from Step 3 → INFO — do NOT flag. - `UnderMinIsrPartitionCount` > 0 (Standard only) → CRITICAL. - **Config-level ISR risk:** topics with `min.insync.replicas >= replication.factor` are a configuration error the CloudWatch metric will not surface. If the user has provided topic configs, flag any such topic as HIGH. #### 7.5 Compute Health - `CpuUser + CpuSystem` avg < 60%. 60-70% → MEDIUM. > 70% → HIGH. - CPU variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH. - `HeapMemoryAfterGC` < 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH. - `RequestHandlerAvgIdlePercent` > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH. - `NetworkProcessorAvgIdlePercent` > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH. #### 7.6 Network Health - `TrafficShaping` = 0 (Standard only). Any non-zero → HIGH. - `BwInAllowanceExceeded` / `BwOutAllowanceExceeded` = 0 (PER_BROKER). Non-zero → HIGH. - Throughput variance across brokers < 20%. 20-50% → MEDIUM. > 50% → HIGH. - Total per-broker throughput < 60% of baseline bandwidth (see `references/troubleshoot-performance.md` for baseline table). 60-70% → MEDIUM. > 70% → HIGH. - Express: `ProduceThrottleTime` / `FetchThrottleTime` > 0 → HIGH. #### 7.7 Storage Health **Standard only** (Express storage is managed — Express clusters only get the `StorageUsed` check): - `KafkaDataLogsDiskUsed` < 70%. 70-85% → HIGH. > 85% → CRITICAL. - `VolumeQueueLength` avg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH. - EBS auto-scaling configured, or disk headroom > 30% → PASS. No auto-scaling AND disk > 50% → MEDIUM. No auto-scaling AND disk > 70% → HIGH. - EBS throughput (`VolumeReadBytes + VolumeWriteBytes`) < 60% of instance baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH. **Express:** - `StorageUsed` per broker vs Express per-broker storage quota (see MSK Express quotas). Approaching quota → MEDIUM. ### Step 8 — Generate the Report Generate a **separate report artifact per cluster reviewed**. Artifact naming: `msk-review-<cluster-name>-<YYYY-MM-DD>.md` Example: `msk-review-prod-orders-2026-04-29.md` Report structure: #### Report Header ``` # MSK Operational Review — <cluster-name> Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD> Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n> Kafka Version: <version> | Monitoring Level: <level> ``` #### Executive Summary - Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL - Finding counts by severity - Top 3 CRITICAL/HIGH items #### Configuration Snapshot | Item | Value | | Cluster state / version | … | | Broker type / instance / count / AZs | … | | Storage | mode, size (Standard), provisioned throughput (if any) | | Encryption | in-transit, in-cluster, at-rest (KMS) | | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags | | Public access | DISABLED / SERVICE_PROVIDED_EIPS | | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION | | Logging | destinations enabled | #### Findings by Dimension For each of the 7 dimensions (7.1-7.7): | # | Finding | Severity | Current State | Recommendation | If a dimension was skipped or partial due to monitoring level or broker type, say so explicitly in a note above the table for that dimension. #### CloudWatch Metrics (7-Day) | Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding | #### Alarm Coverage | # | Recommended Alarm | Metric | Threshold | Priority | Status | Also list alarms currently in `ALARM` state with timestamps. #### Recent Cluster Operations (Last 30 Days) From `list-cluster-operations-v2` in Step 3: | Operation Type | Start Time | End Time | State | Call out any operations that would explain transient metric anomalies. #### Priority Matrix | # | Finding | Severity | Dimension | Effort | Impact | Sorted by severity. #### Next Steps - Immediate (CRITICAL / HIGH — 7 days) - Short-term (MEDIUM — 30 days) - Long-term (LOW — 90 days) #### Appendix — Reference Links - [Amazon MSK best practices — Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html) - [Amazon MSK best practices — Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-express.html) - [Amazon MSK — Monitoring an MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/monitoring.html) - [Amazon MSK — CloudWatch metrics for Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html) - [Amazon MSK — Right-size your cluster](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-right-size-cluster) - [Amazon MSK — Service quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html) ## Common CLI Recipes **Describe cluster:** ``` aws kafka describe-cluster-v2 --cluster-arn <cluster-arn> ``` **List brokers:** ``` aws kafka list-nodes --cluster-arn <cluster-arn> ``` **Get bootstrap brokers:** ``` aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn> ``` **List recent cluster operations (patching, config updates, storage changes):** ``` aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn> ``` **Expand Standard broker storage** — *Operator-run (recommend, do not execute):* ``` aws kafka update-broker-storage \ --cluster-arn <cluster-arn> \ --current-version <cluster-version> \ --target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]' ``` **Get a CloudWatch metric (example: `CpuUser` per broker):** ``` aws cloudwatch get-metric-statistics \ --namespace AWS/Kafka \ --metric-name CpuUser \ --dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \ --start-time <start> --end-time <end> --period 300 --statistics Average ``` **Create a cluster configuration (`server.properties`)** — *Operator-run (recommend, do not execute):* The `--server-properties` argument MUST be a real Kafka properties file with one `key=value` per line, separated by actual newline characters — NOT the literal two-character escape sequence `\n`. The MSK API accepts the bytes as-is; if you pass `"k1=v1\nk2=v2"` as a single string with escaped newlines, MSK stores ONE invalid property line and the cluster will fail to apply it. Recommended pattern: write the properties to a local file with real newlines, then pass it via `fileb://` so the CLI uploads the raw bytes verbatim. Verify by reading the revision back with `describe-configuration-revision` and base64-decoding `ServerProperties` — you should see one property per line. ``` cat > server.properties <<'EOF' auto.create.topics.enable=false default.replication.factor=3 min.insync.replicas=2 unclean.leader.election.enable=false num.io.threads=32 num.network.threads=16 log.retention.hours=168 EOF aws kafka create-configuration \ --name <config-name> \ --kafka-versions "3.6.0" \ --server-properties fileb://server.properties ``` ## Common Error Reference | Error | Cause | Fix | |---|---|---| | `aws kafka update-broker-storage` returns "storage is optimizing" | Previous storage expansion still in cool-down (minimum 6 hours) | Wait for optimization to complete. Check cluster state with `describe-cluster-v2`. | | `ClusterState` is `MAINTENANCE` | Standard brokers undergoing patching. Express brokers stay `ACTIVE` during maintenance. | Wait for cluster to return to `ACTIVE`. Do not perform update operations during `MAINTENANCE`. | | Consumer receives `GROUP_COORDINATOR_NOT_AVAILABLE` | Coordinator broker is temporarily unavailable during rolling restart or overloaded | Retry with backoff. Check if maintenance is in progress via `list-cluster-operations-v2`. | | `NotEnoughReplicasException` on produce | Fewer brokers in ISR than `min.insync.replicas` (default: 2) | Check `UnderReplicatedPartitions` (Standard only). For Express, check `ProduceThrottleTime` and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower `min.insync.replicas` to work around this. | ## Severity Definitions (for review-style reports) | Severity | Definition | SLA | |---|---|---| | CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours | | HIGH | Significant gap that could lead to incidents | Fix within 1 week | | MEDIUM | Notable improvement opportunity | Plan within 30 days | | LOW | Minor optimization or hardening | Address when convenient | | INFO | Observation, no action required | N/A | ## Additional Resources - [Amazon MSK best practices — Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html) - [Amazon MSK best practices — Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-express.html) - [Amazon MSK — Apache Kafka client best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-kafka-client.html) - [Amazon MSK — Monitoring an MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/monitoring.html) - [Amazon MSK — CloudWatch metrics for Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html) - [Amazon MSK — Service quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html) - [Amazon MSK — Custom MSK configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration.html)
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.