{"slug":"msk-operations","title":"msk-operations","summary":"Amazon MSK Provisioned operations, troubleshooting, and health","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-17T16:54:18.700875Z","repo":{"url":"https://github.com/aws/tools-for-devops-agent","stars":82,"forks":62,"license":"Apache-2.0","updatedAt":"2026-09-25T14:34:10Z"},"bodyHtml":"<h1>MSK Operations — AWS DevOps Agent Skill</h1>\n<p>An Amazon MSK Provisioned operations, troubleshooting, and health-assessment\nskill for <a href=\"https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html\">AWS DevOps Agent</a>.\nCovers Standard and Express brokers and both proactive operational reviews and\nad-hoc incident response (performance, consumer lag, storage, maintenance,\nclient tuning).</p>\n<blockquote>\n<p>⚠️ <strong>Non-production disclaimer.</strong> This skill is sample code, not intended for\nproduction use without additional review and testing. Users should validate in\na non-production environment first.</p>\n</blockquote>\n<h2>Purpose</h2>\n<p>Give AWS DevOps Agent the domain knowledge to answer MSK questions accurately\nwithout falling back on training data (which routinely conflates Standard and\nExpress broker behavior). The skill activates in two shapes:</p>\n<ol>\n<li><strong>Operational review</strong> — assess an MSK cluster's health against best\npractices and produce a prioritized findings report.</li>\n<li><strong>Ad-hoc troubleshooting</strong> — investigate a specific MSK symptom (high CPU,\nconsumer lag, disk full, TrafficShaping, unexpected broker restart, etc.)\nand recommend the correct next action.</li>\n</ol>\n<h2>Key Capabilities</h2>\n<ul>\n<li>Determine broker type (Standard vs Express) and route to the correct diagnostic\npath — many CloudWatch metrics and behaviors differ between the two.</li>\n<li>Troubleshoot broker performance issues (<code>CpuUser</code>, <code>CpuSystem</code>,\n<code>RequestHandlerAvgIdlePercent</code>, <code>NetworkProcessorAvgIdlePercent</code>,\n<code>ProduceTotalTimeMsMean</code>, <code>TrafficShaping</code>).</li>\n<li>Diagnose consumer lag using <code>SumOffsetLag</code>, <code>MaxOffsetLag</code>, and (when\n<code>PER_TOPIC_PER_PARTITION</code> is enabled) per-partition <code>OffsetLag</code>.</li>\n<li>Manage broker storage — EBS scaling for Standard, <code>StorageUsed</code> monitoring for\nExpress, tiered storage, retention planning.</li>\n<li>Recommend CloudWatch alarms in the <code>AWS/Kafka</code> namespace and validate that\nthe right monitoring level is enabled.</li>\n<li>Explain rolling restart, patching, and version upgrade behavior; call out\noperations that are unsafe during <code>UnderReplicatedPartitions &gt; 0</code>.</li>\n<li>Advise on Kafka client (producer / consumer) configuration — batch sizing,\n<code>linger.ms</code>, compression, <code>acks</code>, <code>min.insync.replicas</code> — and on\nauthentication choices (IAM, SCRAM, mTLS).</li>\n</ul>\n<h2>Prerequisites</h2>\n<h3>IAM permissions</h3>\n<p>The AWS DevOps Agent's primary cloud-source role needs read access to MSK and\nCloudWatch. All calls except <code>kafka:GetBootstrapBrokers</code> are covered by\n<code>AIDevOpsAgentAccessPolicy</code>. <code>kafka:GetBootstrapBrokers</code> is granted by the\nopt-in <code>EnableMskOperations</code> parameter (default <code>true</code>) in\n<a href=\"https://github.com/aws/tools-for-devops-agent/blob/main/cloudformation/devops-agent-skill-policies.yaml\"><code>cloudformation/devops-agent-skill-policies.yaml</code></a>.\nThe full set of actions the skill uses in practice:</p>\n<pre><code>kafka:DescribeClusterV2\nkafka:ListClustersV2\nkafka:ListNodes\nkafka:GetBootstrapBrokers\nkafka:ListClusterOperationsV2\nkafka:DescribeConfigurationRevision\nkafka:ListConfigurations\ncloudwatch:GetMetricData\ncloudwatch:GetMetricStatistics\ncloudwatch:ListMetrics\ncloudwatch:DescribeAlarms\nlogs:DescribeLogGroups\nlogs:FilterLogEvents\n</code></pre>\n<p>Write actions (<code>UpdateBrokerStorage</code>, <code>CreateConfiguration</code>,\n<code>PutMetricAlarm</code>, etc.) are only recommended in the skill output — the operator\nis expected to run them after review.</p>\n<h3>Cluster access</h3>\n<ul>\n<li>The MSK cluster must live in an account configured as a cloud source in your\nAgent Space.</li>\n<li>No data-plane / Kafka-protocol access is required. Everything the skill uses\ncomes from the AWS control-plane APIs and CloudWatch.</li>\n</ul>\n<h2>Limitations</h2>\n<ul>\n<li>Covers <strong>MSK Provisioned only</strong> — does <strong>not</strong> cover MSK Connect, MSK\nServerless, or MSK Replicator.</li>\n<li>No Kafka data-plane visibility. The skill cannot read topic contents, run\n<code>kafka-consumer-groups.sh</code>, or otherwise interact with the Kafka protocol.</li>\n<li>Some checks require monitoring level <code>PER_BROKER</code> or higher (thread pool idle\n%, <code>VolumeQueueLength</code>, <code>BwInAllowanceExceeded</code>, IAM connection metrics). At\n<code>DEFAULT</code> level the skill will explicitly note which checks are limited.</li>\n<li>Sizing questions (broker count, instance type choice, monthly cost) are\nredirected to the AWS documentation — the skill does not include a bundled\nsizing calculator.</li>\n</ul>\n<h2>Agent Types</h2>\n<p>This skill is intended for the following agent types (selected in the Operator\nWeb App at upload time):</p>\n<ul>\n<li><strong>Chat tasks</strong> — conversational invocation in Chat (\"my MSK cluster is\nlatent\", \"run an MSK health check on <code>prod-cluster</code>\", \"why did broker 2\nrestart last night?\").</li>\n<li><strong>Evaluation</strong> — proactive best-practices recommendations.</li>\n</ul>\n<p>Select <strong>Generic</strong> instead if you want the skill available to all agent types.</p>\n<h2>Uploading to AWS DevOps Agent</h2>\n<blockquote>\n<p>Reference: <a href=\"https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill\">Uploading a skill</a></p>\n</blockquote>\n<h3>1. Package the skill</h3>\n<p>From the <code>skills/</code> directory in this repo:</p>\n<pre><code>cd skills\nzip -r msk-operations.zip msk-operations/ -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' '*.xml' '*.csv' '*.tsv' '*.html' '*.htm' '*.png' '*.jpg' '*.jpeg' '*.gif' '*.svg' '*.webp' '*.pdf' -x '*/.claude/*' '*/scripts/*' '*/README.md' '*/.skilleval.yaml' '*/.skilleval.yml' '*/CHANGELOG.md' '*/evals/*'\n</code></pre>\n<p>The resulting <code>msk-operations.zip</code> contains:</p>\n<pre><code>msk-operations/\n├── SKILL.md\n└── references/\n    ├── troubleshoot-performance.md\n    ├── troubleshoot-consumer-lag.md\n    ├── manage-storage.md\n    ├── monitor-and-alarm.md\n    ├── maintenance-operations.md\n    └── configure-clients.md\n</code></pre>\n<p>Constraints (enforced at upload time):</p>\n<ul>\n<li>Total zip size ≤ <strong>6 MB</strong>.</li>\n<li><code>SKILL.md</code> is required and must include <code>name</code> and <code>description</code> frontmatter.</li>\n<li>A <code>scripts/</code> directory is <strong>not</strong> allowed — uploads containing scripts are rejected.</li>\n</ul>\n<h3>2. Upload via the Operator Web App</h3>\n<ol>\n<li>Navigate to the <strong>Skills</strong> page in your Agent Space Operator Web App.</li>\n<li>Click <strong>Add skill</strong> → <strong>Upload skill</strong>.</li>\n<li>Drag and drop <code>msk-operations.zip</code> (or browse to it).</li>\n<li>Select agent types: <strong>Chat tasks</strong> and <strong>Evaluation</strong> (or leave <strong>Generic</strong>\nto make it available to all agent types).</li>\n<li>Review the validation results.</li>\n<li>Click <strong>Upload</strong>.</li>\n</ol>\n<h2>How to Use This Skill</h2>\n<p>In DevOps Agent Chat, use natural language. You do NOT need to mention the\nskill name — the agent activates it based on the MSK / Kafka triggers in the\nskill description.</p>\n<h3>Ad-hoc troubleshooting prompts</h3>\n<ul>\n<li><em>\"My MSK cluster <code>prod-orders</code> is showing high produce latency in <code>us-east-1</code>.\nWhat should I check?\"</em></li>\n<li><em>\"Consumer group <code>payments-consumer</code> on cluster <code>prod-orders</code> has growing\nlag. Diagnose it.\"</em></li>\n<li><em>\"UnderReplicatedPartitions is &gt; 0 on broker 2 of <code>prod-orders</code>. What's the\nsafe next step?\"</em></li>\n<li><em>\"Why did broker 3 of <code>prod-orders</code> restart last night?\"</em></li>\n<li><em>\"Storage on <code>prod-orders</code> is at 82%. What are my options?\"</em></li>\n<li><em>\"TrafficShaping is firing on all three brokers of <code>prod-orders</code>.\"</em></li>\n</ul>\n<h3>Operational-review prompts</h3>\n<ul>\n<li><em>\"Run an MSK operational review on cluster <code>prod-orders</code>.\"</em></li>\n<li><em>\"Audit <code>prod-orders</code> against MSK best practices.\"</em></li>\n<li><em>\"Are the CloudWatch alarms on <code>prod-orders</code> complete? Which ones am I\nmissing?\"</em></li>\n<li><em>\"Health-check every MSK Provisioned cluster in <code>us-east-1</code>.\"</em></li>\n</ul>\n<h3>Client tuning prompts</h3>\n<ul>\n<li><em>\"My Java Kafka producer is getting <code>NotEnoughReplicasException</code> when writing\nto MSK. What producer settings should I check?\"</em></li>\n<li><em>\"What should <code>linger.ms</code> and <code>batch.size</code> be for a producer writing 200 MB/s\nto a Standard broker cluster?\"</em></li>\n</ul>\n<h2>Skill Contents</h2>\n<pre><code>msk-operations/\n├── SKILL.md\n├── README.md\n├── CHANGELOG.md\n├── .skilleval.yaml\n├── evals/\n│   ├── evals.json\n│   └── eval_queries.json\n└── references/\n    ├── troubleshoot-performance.md\n    ├── troubleshoot-consumer-lag.md\n    ├── manage-storage.md\n    ├── monitor-and-alarm.md\n    ├── maintenance-operations.md\n    └── configure-clients.md\n</code></pre>\n<p><code>README.md</code>, <code>CHANGELOG.md</code>, <code>.skilleval.yaml</code>, and <code>evals/</code> are for repo\nmaintenance and are not required at the cluster — the <code>zip</code> command above\nexcludes them from the upload artifact.</p>\n","files":[{"path":"CHANGELOG.md","sizeBytes":1460,"isText":true},{"path":"evals/benchmark.json","sizeBytes":27255,"isText":true},{"path":"evals/eval_queries.json","sizeBytes":1059,"isText":true},{"path":"evals/evals.json","sizeBytes":3178,"isText":true},{"path":"evals/report.json","sizeBytes":1604,"isText":true},{"path":"evals/trigger_report.json","sizeBytes":3454,"isText":true},{"path":"README.md","sizeBytes":8194,"isText":true},{"path":"references/configure-clients.md","sizeBytes":6873,"isText":true},{"path":"references/maintenance-operations.md","sizeBytes":12558,"isText":true},{"path":"references/manage-storage.md","sizeBytes":7250,"isText":true},{"path":"references/monitor-and-alarm.md","sizeBytes":13630,"isText":true},{"path":"references/troubleshoot-consumer-lag.md","sizeBytes":11390,"isText":true},{"path":"references/troubleshoot-performance.md","sizeBytes":9304,"isText":true},{"path":".skilleval.yaml","sizeBytes":77,"isText":true},{"path":"SKILL.md","sizeBytes":26682,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-17T16:54:53.193899Z","sha256":"9AD042D753B1B4BDF4C897ABAA032CFAECB873E465A7AF3A10108656BF974C85","sizeBytes":47977},"review":null,"source":{"repositoryUrl":"https://github.com/aws/tools-for-devops-agent","path":"skills/msk-operations","license":"Apache-2.0","commit":"a9ca636abac7bde16132ce9508586143753db97a","subtreeSha":"1C73E190F4C9B4FF2408460401D8159E729DCA719B1F5AEA5330C3821496C755","lastSyncedAt":"2026-09-25T23:11:37.941909Z"},"reviewedAt":"2026-09-17T16:56:15.15097Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/msk-operations"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart"},{"target":"git","command":"git clone https://github.com/aws/tools-for-devops-agent.git"}]}