Claude Skill

eks-upgrade-readiness

Use this skill when a user asks to assess, plan, or validate an

LLM Mart · 0 points · 3 views 0 listing impressions 0 install-command copies
Download aws-tools-for-devops-agent-skills_eks-upgrade-readiness-d8f96e1.zip · 58 KB

Install

skills CLI npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/eks-upgrade-readiness
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart
Git git clone https://github.com/aws/tools-for-devops-agent.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.

README

EKS Upgrade Readiness Skill

A skill for AWS DevOps Agent that performs read-only upgrade readiness assessments for Amazon EKS clusters, aligned with the AWS EKS Best Practices Guide — Cluster Upgrades.

Purpose

EKS upgrades can fail or cause downtime when deprecated APIs, incompatible addons, version-skewed node groups, or misconfigured PDBs are not caught beforehand. This skill systematically checks every dimension the Best Practices Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE verdict with a prioritized remediation plan.

Key Capabilities

  • EKS Upgrade Insights (primary signal) — UPGRADE_READINESS findings from the ListInsights/DescribeInsight APIs
  • Infrastructure prerequisites — subnet IP availability (VPC CNI mode-aware), IAM role, KMS key, service quotas
  • API deprecation analysis — maps removed APIs to replacements, scans Helm stored manifests (deployed revision), vendor-aware CRD checks
  • Addon compatibility (live API) — validates managed addons via DescribeAddonVersions, detects self-managed addons via deployment/Helm scan
  • Full data plane inventory — managed node groups, self-managed ASGs, Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles, kubelet version map
  • AL2→AL2023 migration — launch template analysis, custom AMI detection, bootstrap differences, cgroup v2, IMDSv2
  • Upgrade ordering — pre-upgrade alignment (Karpenter/CA/webhooks before CP) vs post-upgrade sequence
  • PDB and topology spread — detects drain blockers and availability risks
  • StatefulSet safety — grace period, PVC retention policy, single-replica risks
  • Pre-upgrade health baseline — node Ready status, pending CSRs, system pod health, DNS/metrics baseline
  • Capacity planning — surge calculation, ODCR/FDCR guidance, blue-green alternative, autoscaler pause
  • GitOps/IaC detection — routes remediation through owning tool (Terraform/CDK/ArgoCD/Flux/eksctl)
  • Post-upgrade validation — DNS, metrics, scheduling, LB health, IRSA smoke tests
  • Client/CI tooling skew — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level)
  • Pod Identity awareness — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance
  • Remediation playbook — all mutations separated, require operator approval
  • Structured upgrade plan — ordered execution with rollback gates
  • Machine-readable output — optional JSON verdict (per-gate status, confidence, evidence) alongside the markdown report, for CI/CD gating

Prerequisites

IAM Permissions

The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling:

eks:DescribeCluster
eks:ListClusters
eks:ListInsights
eks:DescribeInsight
eks:ListAddons
eks:DescribeAddon
eks:DescribeAddonVersions
eks:ListNodegroups
eks:DescribeNodegroup
eks:ListFargateProfiles
eks:DescribeFargateProfile
eks:ListUpdates
eks:DescribeUpdate
ec2:DescribeSubnets
ec2:DescribeInstances
ec2:DescribeLaunchTemplateVersions
ec2:DescribeImages
ec2:DescribeCapacityReservations
iam:GetRole
autoscaling:DescribeAutoScalingGroups
autoscaling:DescribeLaunchConfigurations
servicequotas:GetServiceQuota

Kubernetes RBAC (if kubectl access available)

A ClusterRole with read-only access to nodes, pods, deployments, statefulsets, daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources, and ENIConfigs. See the "Required Permissions" section in SKILL.md for the full ClusterRole manifest. kubectl access is optional — the assessment still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks.

AWS Resources

  • One or more Amazon EKS clusters (any supported version)
  • Control plane logging enabled (recommended for post-upgrade debugging)

Limitations

  • EKS clusters only. Does not cover EKS Anywhere, Outposts, or Local Zones.
  • Read-only by design. The skill produces recommendations; it never executes mutating APIs. All mutations are in the Remediation Playbook (Step 14).
  • UNKNOWN ≠ PASS. Missing data or access denial produces UNKNOWN, never PASS. The overall verdict cannot be READY while any gate is UNKNOWN.
  • Addon version data may lag. Static reference table is fallback only — always prefer live describe-addon-versions API.
  • Pagination required. Large clusters with many node groups or addons require exhausting API pagination tokens.

Agent Types

  • Chat tasks — ask for upgrade readiness assessments

Uploading to AWS DevOps Agent

Option A: Import from GitHub (recommended)

If you have a GitHub connection configured in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then point to skills/eks-upgrade-readiness. See Importing a skill from a repository for full instructions.

Note: You cannot connect the aws-samples GitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup.

Option B: Upload as a zip file

  1. Zip the eks-upgrade-readiness/ directory (only including allowed extensions):
cd skills
zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \
  -i '*.md' '*.json' '*.yaml' '*.yml' \
  -x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*'
  1. In the AWS DevOps Agent web app, navigate to the Skills page.
  2. Click Add skill → Upload skill.
  3. Drag and drop the eks-upgrade-readiness.zip file (max 6 MB).
  4. Select the agent type: Chat tasks.
  5. Click Upload.

Option C: Upload via the Asset API

Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to the CHAT agent type. See Managing a skill end-to-end for the full API workflow.

How to Use This Skill

Describe the task in natural language — you do not need to name the skill.

Example Prompts

"Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?"
"Check upgrade readiness for all my EKS clusters"
"What deprecated APIs would break if I upgrade to Kubernetes 1.32?"
"Plan the upgrade of my cluster from 1.30 to 1.31 including node groups"
"Are my addons compatible with EKS 1.31?"
"Will my PDBs block a node group upgrade?"
"I need to upgrade a 50-node cluster — what capacity do I need?"
"Compare in-place vs blue-green strategy for my 200-node cluster"
"My cluster is managed by Terraform — how should I do the upgrade?"
"We use AL2 with custom bootstrap scripts — what breaks going to 1.33?"
"We just upgraded to 1.31 — what should we validate?"
"We use Pod Identity and are planning a blue-green migration — what identity work is needed?"
"Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline"

Modes

Mode Trigger Behavior
Full assessment "upgrade readiness", "ready to upgrade" All 17 steps, scored report
Targeted check "deprecated APIs", "addon compatibility", "PDB check" One dimension, focused
Planning "upgrade plan", "upgrade runbook" Execution order with rollback gates
Comparison "blue-green vs in-place" Strategy recommendation
Validation "post-upgrade check", "validate upgrade" Smoke tests (Step 15)

Skill Structure

eks-upgrade-readiness/
├── SKILL.md                # Main skill instructions (17-step workflow)
├── README.md               # This file
├── CHANGELOG.md            # Version history
├── .skilleval.yaml         # Agent Skill Eval config
├── evals/
│   ├── evals.json          # 24 functional evaluation scenarios
│   └── eval_queries.json   # Trigger tests (positive and negative)
└── references/
    ├── safety-invariants.md         # Hard safety rules, knowledge hierarchy, operation classification
    ├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity
    ├── pre-flight-checks.yaml       # Blocking vs warning checks, timeouts, soak periods, rollback conditions
    ├── api-deprecations.md          # K8s API removal schedule by version
    ├── addon-version-matrix.md      # EKS addon compatibility (static fallback)
    ├── capacity-planning.md         # FDCR/ODCR surge capacity guidance
    ├── upgrade-troubleshooting.md   # Tools, feature removals, blue-green
    ├── karpenter-checks.md          # Full 14-check Karpenter registry (KARP-01 to KARP-14)
    ├── pre-drain-safety.md          # DRAIN-01 to DRAIN-06 detection and remediation
    ├── al2-al2023-migration.md      # AL2→AL2023 migration assessment details
    └── data-plane-inventory.md      # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands

Safety

This skill operates in read-only mode:

  • No cluster modifications — upgrade actions are recommendations only
  • No update-*, delete-*, or create-* API calls
  • All mutations isolated in Step 14 Remediation Playbook (operator approval)
  • All findings include evidence and specific remediation steps
  • The operator reviews the report and decides whether to proceed
  • UNKNOWN verdicts prevent false confidence (never marks missing data as PASS)

Non-production disclaimer

⚠️ This skill is sample code, not intended for production use without additional review and testing. Validate in a non-production environment first. Compatibility data and version matrices are point-in-time references — always verify with aws eks describe-addon-versions for the latest data.

Skill manifest

EKS Upgrade Readiness

Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade validation aligned with the EKS Best Practices Guide.

When to Use

Activate this skill when the user asks to:

  • Check if an EKS cluster is ready to upgrade
  • Plan an EKS version upgrade (control plane, node groups, or both)
  • Identify deprecated Kubernetes APIs before upgrading
  • Validate addon compatibility with a target version
  • Assess node group upgrade strategy and capacity requirements
  • Review Pod Disruption Budgets or topology spread for upgrade safety
  • Understand EKS end-of-support, extended support, or auto-upgrade implications
  • Evaluate Karpenter Drift or node expiry upgrade behavior
  • Compare in-place vs blue-green upgrade strategies
  • Create an upgrade runbook or checklist
  • Detect GitOps/IaC version ownership before upgrading

Safety First

Before doing anything, load references/safety-invariants.md. It defines the knowledge hierarchy, hard rules, operation classification, and uncertainty handling. Keep it in context for the entire assessment.

Critical Warnings

  • This skill is read-only. All commands are describe*, list*, get*. The agent does NOT execute mutating APIs. Mutations are in Step 14 and require explicit operator approval.
  • One minor version at a time. EKS control plane upgrades proceed one minor version per operation (e.g., 1.30 → 1.31).
  • Version skew policy. Before planning an upgrade, no kubelet may be newer than the current control plane. For the target version, kubelet may be no more than N-3 on 1.28+ (N-2 below 1.28).
  • Addons must be upgraded AFTER the control plane (exceptions in Step 8).
  • Auto-upgrade policy. Clusters past the 26-month lifecycle will be auto-upgraded. Proactive upgrade avoids disruption.
  • Control plane rollback (July 2026+). 7-day rollback window after upgrade. Conditional, not guaranteed — skill checks eligibility.
  • UNKNOWN ≠ PASS. Any gate that cannot be assessed MUST be UNKNOWN, never PASS. Overall verdict cannot be READY while any gate is UNKNOWN.

Evidence Completeness

Uses references/required-check-registry.yaml to track checks performed, skipped, or blocked. EC = checks_performed / total_applicable × 100%. EC < 50% produces a mandatory warning.

Grading and Confidence

Level Meaning When to Use
HIGH (90%+) Confirmed from authoritative source EKS Insights API, direct kubectl query, AWS API response
MEDIUM (60-89%) Inferred from available data Partial kubectl access, version matching heuristics
LOW (30-59%) Limited data, possible gaps No kubectl, no logging enabled, partial API access
UNKNOWN Cannot determine Tool unavailable, no data, access denied

False-positive guards:

  • Empty query result ≠ PASS (mark UNKNOWN)
  • No kubectl ≠ N/A for everything (AWS APIs still work)
  • EKS Insights PASSING ≠ skip other checks (covers a subset only)
  • Addon "compatible" ≠ "recommended"
  • Pagination not exhausted → confidence LOW

Verdict rules (evaluate applicable gates only; N/A gates are excluded):

  1. NOT READY: one or more applicable gates are FAIL. A known blocker wins over uncertainty because proceeding is unsafe.
  2. CANNOT DETERMINE: no gate is FAIL, but one or more applicable gates are UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data).
  3. READY WITH WARNINGS: all applicable gates are assessed, none FAIL or UNKNOWN, and one or more are WARN.
  4. READY: every applicable gate is PASS.

Format: [PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>

Cost Awareness

  • EKS Insights API (Step 3) is free — always use first.
  • CloudWatch Logs Insights cost ~$0.0076/GB scanned. Default to 60-min windows.
  • Extended support costs $0.60/cluster/hour — upgrading saves money.
  • Surge nodes incur temporary EC2 cost during overlap period.

Required Permissions

AWS IAM — see README.md "Prerequisites → IAM Permissions" for the full read-only action list (eks:Describe*, eks:List*, ec2:Describe*, autoscaling:Describe*, iam:GetRole, servicequotas:GetServiceQuota).

Kubernetes RBAC (only if kubectl access is available — the assessment still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB checks). Read-only ClusterRole covering every kubectl get/describe used in this skill:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks-upgrade-readiness-readonly
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - pods
      - configmaps
      - secrets
      - events
      - persistentvolumeclaims
      - certificatesigningrequests
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources:
      - validatingwebhookconfigurations
      - mutatingwebhookconfigurations
    verbs: ["get", "list", "watch"]
  - apiGroups: ["karpenter.sh"]
    resources: ["nodepools", "nodeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["karpenter.k8s.aws"]
    resources: ["ec2nodeclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["crd.k8s.amazonaws.com"]
    resources: ["eniconfigs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses", "csinodes"]
    verbs: ["get", "list", "watch"]

Bind with a ClusterRoleBinding to the identity the agent assumes (e.g. via IRSA/Pod Identity or an EKS access entry). secrets read access is required only for the Helm stored-manifest scan (Step 4) — omit that rule and accept UNKNOWN on Helm checks if a customer's security policy disallows it.


Step 1: Gather Cluster Context

aws eks describe-cluster --name <cluster-name> --region <region>

Extract: cluster.version, platformVersion, status (must be ACTIVE), kubernetesNetworkConfig, logging.clusterLogging (audit log must be enabled), resourcesVpcConfig.subnetIds, tags (IaC ownership detection).

Determine target version: ask user or default to current + 1 minor. Confirm target is in standard support via the EKS release calendar.

Step 2: Verify Infrastructure Prerequisites

Check these — failures are BLOCKERs:

  1. Subnet IP availability — need ≥5 IPs per cluster subnet. Mode-aware: standard IPv4, prefix delegation, custom networking, IPv6, SGP. Use aws ec2 describe-subnets with cluster subnet IDs.
  2. EKS IAM role — verify role exists with eks.amazonaws.com trust.
  3. KMS key (if encryption enabled) — verify cluster role has key access.
  4. Service quota headroom — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000) must have room for surge nodes. Use aws service-quotas get-service-quota.

VPC CNI mode and capacity-input detection:

kubectl get ds aws-node -n kube-system -o json | jq '
  .spec.template.spec.containers[0].env[]
  | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET"))
  | {name, value}'

VPC CNI Surge-Capacity Gate

Do not treat "mode detected" as capacity validated. First calculate the managed-node-group surge using references/capacity-planning.md, distribute it by the node group's AZ placement, then verify the relevant subnet/ENI resource for the selected mode. Record the inputs and calculations as evidence; a missing mode-specific input is UNKNOWN, not PASS.

Mode Required assessment before a node surge Pass condition
Standard IPv4 Inspect WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET on aws-node; use node status.allocatable.pods and current pod count to calculate the additional secondary-IP demand for every surge node. Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand.
Prefix delegation Confirm ENABLE_PREFIX_DELEGATION=true; each IPv4 prefix consumes a /28 (16 addresses). Calculate required additional prefixes as ceil(additional_pod_ips / 16) per affected subnet/AZ. floor(availableIpAddressCount / 16) covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target.
Custom networking Confirm AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, enumerate ENIConfig resources, and map each node AZ to its spec.subnet and security groups. The ENIConfig pod subnet, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ.
Security Groups for Pods Confirm ENABLE_POD_ENI=true, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute.
IPv6 Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6.
# Custom networking: inspect every AZ-to-pod-subnet mapping
kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}'

# Security Groups for Pods: inspect trunk/branch interfaces after mode detection
aws ec2 describe-network-interfaces \
  --filters Name=interface-type,Values=trunk,branch \
  --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}'

Client and CI Tooling Skew (WARN-level, not a blocker)

Operator/CI tooling that is too far behind the target version causes confusing failures during and after the upgrade. Check installed versions where available:

kubectl version --client -o json   # client minor version
eksctl version                      # if eksctl-managed
helm version --short                # if Helm-managed workloads
Tool Skew Rule Risk if Violated
kubectl Must be within ±1 minor of the target kube-apiserver version (upstream Kubernetes version skew policy) Unrecognized fields, API calls silently rejected or misinterpreted
eksctl Must support the target EKS version (check release notes for the version that added support) eksctl upgrade commands fail or use stale defaults
Helm 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated Chart operations may fail independent of the cluster upgrade
Terraform AWS provider Must be new enough to support any target-version-specific attributes in use (e.g. upgrade_policy, compute_config for Auto Mode) — check the provider changelog for the attribute terraform apply fails validation or silently ignores the new attribute

Do not hardcode exact version floors here — they shift every EKS release. Report the installed version, the rule, and a WARN if it cannot be confirmed current; never treat "tool not detected" as PASS.

Step 3: Check EKS Upgrade Insights

Primary authoritative signal. Always query first.

aws eks list-insights --cluster-name <cluster> \
  --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}'
aws eks describe-insight --cluster-name <cluster> --id <insight-id>
Status Gate Action
ERROR FAIL Must fix before upgrade
WARNING WARN Recommended fix
PASSING PASS No action
UNKNOWN UNKNOWN EKS could not evaluate the check; investigate and refresh
None returned UNKNOWN Continue other checks

Freshness gate: For every returned summary, capture lastRefreshTime and lastTransitionTime, then call DescribeInsight to collect the status, affected resources, and recommendation. Treat data as stale when lastRefreshTime is more than 24 hours old at assessment time, or predates a known relevant workload/addon change. A stale, missing, inaccessible, or unpaginated insight set is UNKNOWN; never reuse it as PASS. The assessment must not call StartInsightsRefresh because this skill is read-only. Instead, ask the operator to refresh insights through an approved workflow and rerun the assessment after the refresh completes.

Critical: Insights does NOT cover Helm stored manifests, CRD deprecations, StatefulSets, Karpenter, service quotas, PDBs, or capacity planning.

Step 4: API Deprecation Analysis

Check version-specific removal gates relevant to user's target:

  • ≥1.33: AL2 AMI unavailable (Critical)
  • ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed
  • ≥1.25: Dockershim and PodSecurityPolicy removed
  • ≥1.23: In-tree EBS provisioner deprecated

Helm stored manifests — the #1 missed blocker. Scan latest deployed release secrets for deprecated apiVersion lines:

kubectl get secrets -A -l owner=helm,status=deployed
# Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest'

Third-party CRD deprecations — check Istio, cert-manager, Karpenter, Flux, Argo, Prometheus Operator versions against known deprecation timelines.

Tools: kubent, pluto detect-all-in-cluster, helm mapkubeapis --dry-run. See references/api-deprecations.md for full removal schedule.

Step 5: Addon Compatibility Check

Managed addons: Use live API to build compatibility matrix:

aws eks list-addons --cluster-name <cluster>
aws eks describe-addon --cluster-name <cluster> --addon-name <name>
aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target>

Self-managed addons: First compare ListAddons with the actual kube-system workloads. Explicitly detect the three core addons: aws-node (VPC CNI), coredns, and kube-proxy. If a core component is absent from the managed-addon inventory but present in-cluster, mark it self-managed/custom and inspect its image, args, and configuration before assessing target support:

kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get deployment coredns -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get configmap coredns aws-node -o yaml

For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI, validate custom environment/config-map values and the mode-specific capacity gate in Step 2. For kube-proxy, validate its deployed mode and version against its upstream support policy. Then scan other self-managed components (aws-load-balancer, external-dns, metrics-server, cluster-autoscaler, cert-manager, ingress-nginx, argocd, flux); extract image tags and validate against their K8s support matrices.

Pod Identity Agent (eks-pod-identity-agent): Check like any other managed addon via DescribeAddon / DescribeAddonVersions — its version gates which association features are available (e.g. multiple associations per pod, target IAM role sessions). If not installed but kubectl get pods -A -o json shows service accounts with eks.amazonaws.com/role-arn annotations instead, the cluster is on IRSA, not Pod Identity — note this for blue-green planning (see references/upgrade-troubleshooting.md → Identity Migration Considerations).

Upgrade order: Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks. Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed.

See references/addon-version-matrix.md for static fallback reference.

Step 6: Full Data Plane Inventory

Inventory ALL node populations. See references/data-plane-inventory.md for complete detection commands.

  • MNG: aws eks list-nodegroups + describe-nodegroup for each
  • Self-managed ASGs: Find by cluster tag in describe-auto-scaling-groups
  • Karpenter: NodePools, EC2NodeClasses, controller version and health
  • Auto Mode: Check cluster.computeConfig.enabled
  • Fargate: aws eks list-fargate-profiles + describe each
  • Kubelet versions: kubectl get nodes — confirm within skew window

Version skew requires two independent predicates:

  1. Current-state upper bound: no kubelet may be newer than the current control plane (kubelet_minor <= current_control_plane_minor). A node already newer than the current API server is an invalid state and must be corrected before planning the upgrade.
  2. Target lower bound: for target 1.X, kubelet must be at least 1.(X-3) when X>=28, or 1.(X-2) when X<28.

Any node violating either predicate is a FAIL.

Step 7: AL2 → AL2023 Migration Assessment

If AL2 detected and target ≥1.33: CRITICAL blocker (EKS releases AL2 AMIs only through 1.32). If AL2 is detected with a target <1.33: WARNING — upstream Amazon Linux 2 reaches end of life on June 30, 2026.

Assess: bootstrap method (bootstrap.sh vs nodeadm), custom AMIs, user data compatibility (yum→dnf, kubelet-extra-args→NodeConfig), cgroup v2 workload compatibility, IMDSv2 readiness.

See references/al2-al2023-migration.md for full detection commands and migration strategy.

Step 8: Upgrade Ordering and Pre-Upgrade Alignment

Pre-CP: Karpenter (if needed), Cluster Autoscaler (must match target), admission webhooks with failurePolicy: Fail, custom controllers using deprecated APIs.

Post-CP: Standard addon and node group upgrade order (Step 5).

Webhook check:

kubectl get validatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'
kubectl get mutatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'

Step 9: PDB, Topology Spread, and Workload Safety

PDB blockers: maxUnavailable: 0, minAvailable == replicas, orphaned PDBs:

kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)'

Pre-drain safety (DRAIN-01 to DRAIN-06): Bare pods, emptyDir data loss, custom finalizers, EBS AZ-pinning, webhook deadlock, CoreDNS SPOF. See references/pre-drain-safety.md for full detection commands.

TopologySpreadConstraints: Flag multi-replica deployments without topology spread.

StatefulSet safety: Check terminationGracePeriodSeconds != 0, PVC retention policy, single-replica without PDB, update strategy.

Scaled-to-zero workloads: Detect and flag for separate validation.

Step 10: Pre-Upgrade Cluster Health Baseline

Confirm healthy steady state before upgrade. Failures compound on unhealthy clusters.

  • Node health: All nodes Ready, no MemoryPressure/DiskPressure/PIDPressure
  • Pending CSRs: Indicate node registration issues
  • Crash-looping system pods: Check kube-system, monitoring, ingress namespaces
  • Metrics and DNS baseline: Verify metrics-server and CoreDNS responding

Record baselines for post-upgrade comparison.

Step 11: Fargate Considerations

Fargate pods upgrade when redeployed after CP upgrade. All Fargate pods must be restarted post-upgrade. Restart command is in Step 14 (mutation, requires approval).

Step 12: Management Plane and IaC Ownership Detection

Detect management method to route remediation correctly:

Detection Management Plane Mutation Routing
ACK CRD + Cluster CR ACK Patch ACK Cluster CR
ACK CR with kro.run/owned KRO over ACK Patch kro instance
Tags: terraform:* Terraform Update .tf, terraform apply
Tags: aws:cloudformation:* CloudFormation Update template, stack update
Tags: aws:cdk:* CDK Update construct, cdk deploy
Tags: eksctl.cluster.k8s.io/* eksctl Update config, eksctl upgrade
Labels: argocd.argoproj.io/* ArgoCD Update Git source, sync
Labels: kustomize.toolkit.fluxcd.io/* Flux Update Git source, reconcile
Tags: pulumi:* Pulumi Update program, pulumi up
None found unknown Block mutations until confirmed

Route ALL remediation through the owning tool — never suggest direct AWS CLI when IaC is detected (causes drift).

Step 13: Autoscaler Pause During Node Rotation

During upgrades, autoscalers can interfere with rolling replacement. Check current Karpenter consolidation config and Cluster Autoscaler scale-down state. Recommend pausing both before node rotation and re-enabling after completion.

Pause commands are in Step 14 (mutations, require operator approval).

Step 14: Remediation Playbook (Operator Approval Required)

⚠️ ALL commands in this section are MUTATIONS. The agent MUST NOT execute these — present as a playbook for operator review.

  • 14.1 Helm stored manifest fix: helm mapkubeapis + helm upgrade
  • 14.2 Addon conflict resolution: first capture DescribeAddon output and configurationValues; use --resolve-conflicts PRESERVE to retain reviewed custom configuration, or OVERWRITE only after approving replacement with EKS defaults and recording rollback steps. OVERWRITE can discard custom configuration.
  • 14.3 Fargate pod restart: kubectl rollout restart across namespaces
  • 14.4 PDB temporary adjustment: kubectl patch pdb (revert after upgrade)
  • 14.5 Karpenter pause: kubectl annotate nodepools --all "karpenter.sh/do-not-disrupt=true"
  • 14.6 CA scale-down pause: patch CA config scale-down-enabled=false
  • 14.7 Node group upgrade: MNG via update-nodegroup-version, Karpenter via EC2NodeClass patch (drift), self-managed via launch template update

Step 15: Post-Upgrade Functional Validation

Present as validation checklist for operator:

  • DNS resolution (nslookup kubernetes.default)
  • Metrics server (kubectl top nodes/pods)
  • Pod scheduling (run test pod)
  • Load balancer health (target group check)
  • IRSA / Pod Identity (sts get-caller-identity from pod)
  • CoreDNS and kube-proxy pods running
  • Compare against Step 10 baseline (node count, no new CrashLoopBackOff)

Step 16: Generate Upgrade Plan and Report

Execution Order:

  1. Pre-upgrade alignment (Karpenter/CA/webhooks)
  2. Pause autoscalers
  3. Control plane upgrade (15-40 min)
  4. Wait for ACTIVE status
  5. kube-proxy → vpc-cni → coredns → other managed addons
  6. Self-managed addons
  7. Node groups (one at a time, validate between)
  8. Karpenter nodes (drift-based)
  9. Self-managed nodes (launch template update)
  10. Fargate pods (restart)
  11. Re-enable autoscalers
  12. Post-upgrade validation

Rollback Matrix:

Component Reversibility Method
Control plane CONDITIONAL (7-day window) aws eks update-cluster-version --kubernetes-version <N-1>
Addons FULL Downgrade to previous version
MNG PARTIAL Can halt; completed nodes stay
Karpenter nodes FULL Revert EC2NodeClass
Self-managed FULL Revert launch template
Fargate FULL Redeploy previous config

Rollback eligibility has two phases:

  • Pre-upgrade (advisory only): confirm the planned upgrade is one minor, document the 7-day window and component rollback order, but do not claim the future cluster will be eligible. Rollback readiness insights do not exist until after an eligible upgrade completes.
  • Post-upgrade (authoritative): while the cluster is ACTIVE and still inside the 7-day window, run aws eks list-insights --cluster-name <cluster> --filter '{"categories":["ROLLBACK_READINESS"]}', paginate, then describe-insight for each entry. ERROR blocks a normal rollback; UNKNOWN means EKS could not evaluate readiness and also blocks a normal rollback. Only PASSING insights support an eligible rollback.

This assessment reports the result but never performs update-cluster-version or a forced rollback.

Step 17: Report Format

## EKS Upgrade Readiness Report
**Cluster:** <name> (<region>)
**Current Version:** <current>
**Target Version:** <target>
**Assessment Date:** <date>
**Management Plane:** <detected>
**Evidence Completeness:** <X>% (<performed>/<applicable>)
**Overall Readiness:** READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE

### Pre-Upgrade Health Baseline
- [PASS/FAIL] (confidence: HIGH) All nodes Ready
- [PASS/FAIL] (confidence: HIGH) No pending CSRs
- [PASS/FAIL] (confidence: HIGH) No crash-looping system pods
- [PASS/FAIL] (confidence: HIGH) DNS resolution working
- [PASS/FAIL] (confidence: HIGH) Metrics server responding

### Infrastructure Prerequisites
- [PASS/FAIL] (confidence: HIGH) Subnet IP availability (mode: <type>)
- [PASS/FAIL] (confidence: HIGH) EKS IAM role valid
- [PASS/FAIL/N/A] (confidence: HIGH) KMS key access
- [PASS/FAIL] (confidence: HIGH) EC2 vCPU quota headroom
- [PASS/FAIL] (confidence: HIGH) EBS volume quota headroom

### EKS Upgrade Insights
- [PASS/FAIL/UNKNOWN] (confidence: HIGH) <summary>

### Data Plane Inventory
- Managed Node Groups: <count> (versions: <list>)
- Self-Managed ASGs: <count> (versions: <list>)
- Karpenter NodePools: <count> (version: <ver>)
- Fargate Profiles: <count>
- Total Nodes: <count>

### Blockers (must fix)
1. [FAIL] (confidence: HIGH) <description> — <remediation>

### Warnings (recommended)
1. [WARN] (confidence: MEDIUM) <description> — <recommendation>

### Passing Checks
1. [PASS] (confidence: HIGH) <description>

### Unknown / Not Assessed
1. [UNKNOWN] <gate> — <reason>

### Upgrade Plan
<execution order from Step 16>

### Rollback Window
- Rollback eligibility: ELIGIBLE / NOT ELIGIBLE / CHECK AFTER UPGRADE
- Window: 7 days from CP upgrade completion
- Note: Add-ons and node groups must be rolled back BEFORE CP

### Pre-Drain Risks
- Bare pods (DRAIN-01): <count>
- EmptyDir data loss (DRAIN-02): <count>
- EBS AZ-pinning (DRAIN-04): <count>
- Webhook deadlock (DRAIN-05): <assessment>
- CoreDNS SPOF (DRAIN-06): <status>

### Estimated Timeline
- Control plane: ~30 min
- Addons: ~5 min each
- Node groups: ~<X> min per group
- Total: ~<Y> min

Machine-Readable Output

When the operator asks for a structured result (CI/CD gating, scripted polling, dashboards), emit this JSON alongside — never instead of — the markdown report. Every gate in the markdown report must have a matching entry; the JSON is a serialization of the same evidence, not a summary.

{
  "cluster": "<name>",
  "region": "<region>",
  "assessmentTimestamp": "<ISO-8601>",
  "currentVersion": "<current>",
  "targetVersion": "<target>",
  "overallVerdict": "READY | READY_WITH_WARNINGS | NOT_READY | CANNOT_DETERMINE",
  "evidenceCompletenessPct": 0,
  "gates": [
    {
      "id": "<check-id from required-check-registry.yaml, e.g. NODE-04, ADDON-02, INFRA-01>",
      "name": "<human-readable check name>",
      "status": "PASS | FAIL | WARN | UNKNOWN | N_A",
      "confidence": "HIGH | MEDIUM | LOW",
      "evidence": "<short evidence string, same as markdown bullet>",
      "remediation": "<remediation text, or null if PASS>",
      "checkedAt": "<ISO-8601>"
    }
  ],
  "rollback": {
    "eligible": true,
    "windowExpiresAt": "<ISO-8601 or null>"
  }
}

gates[].id maps 1:1 to the IDs in references/required-check-registry.yaml (prefixes: PF- pre-flight, INFRA- infrastructure, NODE- node assessment, ADDON- addon assessment, WKLD- workload assessment, KARP- Karpenter, DRAIN- pre-drain safety, ROLL- rollback), so a CI pipeline can gate on specific check categories (e.g. fail only on NODE-* or ADDON-* FAILs, warn-only on others) instead of just the overall verdict. overallVerdict follows the same rules as the markdown report — it is never READY while any gate is UNKNOWN.

References

See references/ directory for:

  • safety-invariants.md — Hard safety rules, knowledge hierarchy, operation classification
  • required-check-registry.yaml — All 60+ checks with IDs, categories, and severity
  • pre-flight-checks.yaml — Blocking vs warning checks, timeouts, soak periods, rollback conditions
  • api-deprecations.md — Full K8s API removal schedule by version
  • addon-version-matrix.md — EKS addon compatibility per version (static fallback)
  • capacity-planning.md — FDCR/ODCR and surge capacity guidance
  • upgrade-troubleshooting.md — Common failures, feature removals, and tools
  • karpenter-checks.md — Full 14-check Karpenter registry (KARP-01 to KARP-14)
  • pre-drain-safety.md — DRAIN-01 to DRAIN-06 detection and remediation
  • al2-al2023-migration.md — AL2→AL2023 migration assessment details
  • data-plane-inventory.md — MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
Files (tools-for-devops-agent)
  • evals
    • evals.json 16.4 KB
      Loading…
    • eval_queries.json 3.5 KB
      Loading…
  • references
    • addon-version-matrix.md 5.4 KB
      Loading…
    • al2-al2023-migration.md 6 KB
      Loading…
    • api-deprecations.md 3.6 KB
      Loading…
    • capacity-planning.md 5.9 KB
      Loading…
    • data-plane-inventory.md 6.9 KB
      Loading…
    • karpenter-checks.md 5.5 KB
      Loading…
    • pre-drain-safety.md 5.9 KB
      Loading…
    • pre-flight-checks.yaml 8 KB
      Loading…
    • required-check-registry.yaml 16 KB
      Loading…
    • safety-invariants.md 4.1 KB
      Loading…
    • upgrade-troubleshooting.md 11.5 KB
      Loading…
  • .skilleval.yaml 77 B
    Loading…
  • CHANGELOG.md 9.3 KB
    Loading…
  • README.md 10.3 KB
    # EKS Upgrade Readiness Skill
    
    A skill for AWS DevOps Agent that performs **read-only** upgrade readiness
    assessments for Amazon EKS clusters, aligned with the
    [AWS EKS Best Practices Guide — Cluster Upgrades](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html).
    
    ## Purpose
    
    EKS upgrades can fail or cause downtime when deprecated APIs, incompatible
    addons, version-skewed node groups, or misconfigured PDBs are not caught
    beforehand. This skill systematically checks every dimension the Best Practices
    Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS /
    CANNOT DETERMINE verdict with a prioritized remediation plan.
    
    ## Key Capabilities
    
    - **EKS Upgrade Insights (primary signal)** — UPGRADE_READINESS findings from
      the ListInsights/DescribeInsight APIs
    - **Infrastructure prerequisites** — subnet IP availability (VPC CNI mode-aware),
      IAM role, KMS key, service quotas
    - **API deprecation analysis** — maps removed APIs to replacements, scans Helm
      stored manifests (deployed revision), vendor-aware CRD checks
    - **Addon compatibility (live API)** — validates managed addons via
      DescribeAddonVersions, detects self-managed addons via deployment/Helm scan
    - **Full data plane inventory** — managed node groups, self-managed ASGs,
      Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles,
      kubelet version map
    - **AL2→AL2023 migration** — launch template analysis, custom AMI detection,
      bootstrap differences, cgroup v2, IMDSv2
    - **Upgrade ordering** — pre-upgrade alignment (Karpenter/CA/webhooks before CP)
      vs post-upgrade sequence
    - **PDB and topology spread** — detects drain blockers and availability risks
    - **StatefulSet safety** — grace period, PVC retention policy, single-replica risks
    - **Pre-upgrade health baseline** — node Ready status, pending CSRs, system pod
      health, DNS/metrics baseline
    - **Capacity planning** — surge calculation, ODCR/FDCR guidance, blue-green
      alternative, autoscaler pause
    - **GitOps/IaC detection** — routes remediation through owning tool
      (Terraform/CDK/ArgoCD/Flux/eksctl)
    - **Post-upgrade validation** — DNS, metrics, scheduling, LB health, IRSA smoke tests
    - **Client/CI tooling skew** — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level)
    - **Pod Identity awareness** — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance
    - **Remediation playbook** — all mutations separated, require operator approval
    - **Structured upgrade plan** — ordered execution with rollback gates
    - **Machine-readable output** — optional JSON verdict (per-gate status, confidence,
      evidence) alongside the markdown report, for CI/CD gating
    
    ## Prerequisites
    
    ### IAM Permissions
    
    The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling:
    
    ```
    eks:DescribeCluster
    eks:ListClusters
    eks:ListInsights
    eks:DescribeInsight
    eks:ListAddons
    eks:DescribeAddon
    eks:DescribeAddonVersions
    eks:ListNodegroups
    eks:DescribeNodegroup
    eks:ListFargateProfiles
    eks:DescribeFargateProfile
    eks:ListUpdates
    eks:DescribeUpdate
    ec2:DescribeSubnets
    ec2:DescribeInstances
    ec2:DescribeLaunchTemplateVersions
    ec2:DescribeImages
    ec2:DescribeCapacityReservations
    iam:GetRole
    autoscaling:DescribeAutoScalingGroups
    autoscaling:DescribeLaunchConfigurations
    servicequotas:GetServiceQuota
    ```
    
    ### Kubernetes RBAC (if kubectl access available)
    
    A `ClusterRole` with read-only access to nodes, pods, deployments, statefulsets,
    daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources,
    and ENIConfigs. See the "Required Permissions" section in SKILL.md for the
    full `ClusterRole` manifest. `kubectl` access is optional — the assessment
    still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks.
    
    ### AWS Resources
    
    - One or more Amazon EKS clusters (any supported version)
    - Control plane logging enabled (recommended for post-upgrade debugging)
    
    ## Limitations
    
    - **EKS clusters only.** Does not cover EKS Anywhere, Outposts, or Local Zones.
    - **Read-only by design.** The skill produces recommendations; it never executes
      mutating APIs. All mutations are in the Remediation Playbook (Step 14).
    - **UNKNOWN ≠ PASS.** Missing data or access denial produces UNKNOWN, never
      PASS. The overall verdict cannot be READY while any gate is UNKNOWN.
    - **Addon version data may lag.** Static reference table is fallback only —
      always prefer live `describe-addon-versions` API.
    - **Pagination required.** Large clusters with many node groups or addons
      require exhausting API pagination tokens.
    
    ## Agent Types
    
    - **Chat tasks** — ask for upgrade readiness assessments
    
    ## Uploading to AWS DevOps Agent
    
    **Option A: Import from GitHub (recommended)**
    
    If you have a [GitHub connection configured](https://docs.aws.amazon.com/devopsagent/latest/userguide/connecting-to-cicd-pipelines-connecting-github.html) in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then
    point to `skills/eks-upgrade-readiness`. See [Importing a skill from a repository](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#creating-skills) for full instructions.
    
    > **Note:** You cannot connect the `aws-samples` GitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup.
    
    **Option B: Upload as a zip file**
    
    1. Zip the `eks-upgrade-readiness/` directory (only including allowed extensions):
    
    ```bash
    cd skills
    zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \
      -i '*.md' '*.json' '*.yaml' '*.yml' \
      -x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*'
    ```
    
    2. In the AWS DevOps Agent web app, navigate to the **Skills** page.
    3. Click **Add skill** → **Upload skill**.
    4. Drag and drop the `eks-upgrade-readiness.zip` file (max 6 MB).
    5. Select the agent type: **Chat tasks**.
    6. Click **Upload**.
    
    **Option C: Upload via the Asset API**
    
    Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to the `CHAT` agent type. See [Managing a skill end-to-end](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-managing-assets.html#managing-a-skill-end-to-end) for the full API workflow.
    
    ## How to Use This Skill
    
    Describe the task in natural language — you do not need to name the skill.
    
    ### Example Prompts
    
    ```
    "Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?"
    "Check upgrade readiness for all my EKS clusters"
    "What deprecated APIs would break if I upgrade to Kubernetes 1.32?"
    "Plan the upgrade of my cluster from 1.30 to 1.31 including node groups"
    "Are my addons compatible with EKS 1.31?"
    "Will my PDBs block a node group upgrade?"
    "I need to upgrade a 50-node cluster — what capacity do I need?"
    "Compare in-place vs blue-green strategy for my 200-node cluster"
    "My cluster is managed by Terraform — how should I do the upgrade?"
    "We use AL2 with custom bootstrap scripts — what breaks going to 1.33?"
    "We just upgraded to 1.31 — what should we validate?"
    "We use Pod Identity and are planning a blue-green migration — what identity work is needed?"
    "Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline"
    ```
    
    ### Modes
    
    | Mode | Trigger | Behavior |
    |------|---------|----------|
    | Full assessment | "upgrade readiness", "ready to upgrade" | All 17 steps, scored report |
    | Targeted check | "deprecated APIs", "addon compatibility", "PDB check" | One dimension, focused |
    | Planning | "upgrade plan", "upgrade runbook" | Execution order with rollback gates |
    | Comparison | "blue-green vs in-place" | Strategy recommendation |
    | Validation | "post-upgrade check", "validate upgrade" | Smoke tests (Step 15) |
    
    ## Skill Structure
    
    ```
    eks-upgrade-readiness/
    ├── SKILL.md                # Main skill instructions (17-step workflow)
    ├── README.md               # This file
    ├── CHANGELOG.md            # Version history
    ├── .skilleval.yaml         # Agent Skill Eval config
    ├── evals/
    │   ├── evals.json          # 24 functional evaluation scenarios
    │   └── eval_queries.json   # Trigger tests (positive and negative)
    └── references/
        ├── safety-invariants.md         # Hard safety rules, knowledge hierarchy, operation classification
        ├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity
        ├── pre-flight-checks.yaml       # Blocking vs warning checks, timeouts, soak periods, rollback conditions
        ├── api-deprecations.md          # K8s API removal schedule by version
        ├── addon-version-matrix.md      # EKS addon compatibility (static fallback)
        ├── capacity-planning.md         # FDCR/ODCR surge capacity guidance
        ├── upgrade-troubleshooting.md   # Tools, feature removals, blue-green
        ├── karpenter-checks.md          # Full 14-check Karpenter registry (KARP-01 to KARP-14)
        ├── pre-drain-safety.md          # DRAIN-01 to DRAIN-06 detection and remediation
        ├── al2-al2023-migration.md      # AL2→AL2023 migration assessment details
        └── data-plane-inventory.md      # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
    ```
    
    ## Safety
    
    This skill operates in **read-only** mode:
    
    - No cluster modifications — upgrade actions are recommendations only
    - No `update-*`, `delete-*`, or `create-*` API calls
    - All mutations isolated in Step 14 Remediation Playbook (operator approval)
    - All findings include evidence and specific remediation steps
    - The operator reviews the report and decides whether to proceed
    - UNKNOWN verdicts prevent false confidence (never marks missing data as PASS)
    
    ## Non-production disclaimer
    
    > ⚠️ This skill is sample code, not intended for production use without
    > additional review and testing. Validate in a non-production environment first.
    > Compatibility data and version matrices are point-in-time references — always
    > verify with `aws eks describe-addon-versions` for the latest data.
    
  • SKILL.md 30.3 KB
    ---
    name: eks-upgrade-readiness
    description: Use this skill when a user asks to assess, plan, or validate an
      Amazon EKS cluster upgrade. Activate on requests mentioning "EKS upgrade",
      "Kubernetes version upgrade", "upgrade readiness", "upgrade plan",
      "pre-upgrade check", "version skew", "deprecated API", "addon compatibility",
      "node group upgrade", "control plane upgrade", "EKS end of support",
      "EKS extended support", "Karpenter drift", "kubelet version skew", or
      "blue-green cluster migration". It runs a pre-upgrade assessment per the
      AWS EKS Best Practices Guide, covering infrastructure prerequisites, EKS
      Upgrade Insights, API deprecations, addon compatibility, full data plane
      inventory (managed/self-managed nodes, Karpenter, Auto Mode, Fargate),
      AL2 to AL2023 migration, PDB/topology and StatefulSet safety, capacity
      planning, and pre/post-upgrade validation, then produces a scored readiness
      verdict with prioritized remediation. Do NOT use for ECS, general EKS
      troubleshooting unrelated to version upgrades, or EKS Anywhere/Outpost
      clusters.
    metadata:
      author: LearningNewbie
      version: "2.0.2"
      aws-devops-agent-skills.agent-types: "CHAT"
      aws-devops-agent-skills.aws-services: "Amazon EKS"
      aws-devops-agent-skills.technical-domains: "Containers"
    ---
    
    # EKS Upgrade Readiness
    
    Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade
    validation aligned with the [EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html).
    
    ## When to Use
    
    Activate this skill when the user asks to:
    - Check if an EKS cluster is ready to upgrade
    - Plan an EKS version upgrade (control plane, node groups, or both)
    - Identify deprecated Kubernetes APIs before upgrading
    - Validate addon compatibility with a target version
    - Assess node group upgrade strategy and capacity requirements
    - Review Pod Disruption Budgets or topology spread for upgrade safety
    - Understand EKS end-of-support, extended support, or auto-upgrade implications
    - Evaluate Karpenter Drift or node expiry upgrade behavior
    - Compare in-place vs blue-green upgrade strategies
    - Create an upgrade runbook or checklist
    - Detect GitOps/IaC version ownership before upgrading
    
    ## Safety First
    
    **Before doing anything, load `references/safety-invariants.md`.** It defines
    the knowledge hierarchy, hard rules, operation classification, and uncertainty
    handling. Keep it in context for the entire assessment.
    
    ## Critical Warnings
    
    - **This skill is read-only.** All commands are `describe*`, `list*`, `get*`.
      The agent does NOT execute mutating APIs. Mutations are in Step 14 and
      require explicit operator approval.
    - **One minor version at a time.** EKS control plane upgrades proceed one
      minor version per operation (e.g., 1.30 → 1.31).
    - **Version skew policy.** Before planning an upgrade, no kubelet may be newer
      than the **current** control plane. For the target version, kubelet may be no
      more than N-3 on 1.28+ (N-2 below 1.28).
    - **Addons must be upgraded AFTER the control plane** (exceptions in Step 8).
    - **Auto-upgrade policy.** Clusters past the 26-month lifecycle will be
      auto-upgraded. Proactive upgrade avoids disruption.
    - **Control plane rollback (July 2026+).** 7-day rollback window after upgrade.
      Conditional, not guaranteed — skill checks eligibility.
    - **UNKNOWN ≠ PASS.** Any gate that cannot be assessed MUST be UNKNOWN, never
      PASS. Overall verdict cannot be READY while any gate is UNKNOWN.
    
    ## Evidence Completeness
    
    Uses `references/required-check-registry.yaml` to track checks performed,
    skipped, or blocked. EC = checks_performed / total_applicable × 100%.
    EC < 50% produces a mandatory warning.
    
    ## Grading and Confidence
    
    | Level | Meaning | When to Use |
    |-------|---------|-------------|
    | HIGH (90%+) | Confirmed from authoritative source | EKS Insights API, direct kubectl query, AWS API response |
    | MEDIUM (60-89%) | Inferred from available data | Partial kubectl access, version matching heuristics |
    | LOW (30-59%) | Limited data, possible gaps | No kubectl, no logging enabled, partial API access |
    | UNKNOWN | Cannot determine | Tool unavailable, no data, access denied |
    
    **False-positive guards:**
    - Empty query result ≠ PASS (mark UNKNOWN)
    - No kubectl ≠ N/A for everything (AWS APIs still work)
    - EKS Insights PASSING ≠ skip other checks (covers a subset only)
    - Addon "compatible" ≠ "recommended"
    - Pagination not exhausted → confidence LOW
    
    **Verdict rules (evaluate applicable gates only; `N/A` gates are excluded):**
    1. **NOT READY**: one or more applicable gates are FAIL. A known blocker wins
       over uncertainty because proceeding is unsafe.
    2. **CANNOT DETERMINE**: no gate is FAIL, but one or more applicable gates are
       UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data).
    3. **READY WITH WARNINGS**: all applicable gates are assessed, none FAIL or
       UNKNOWN, and one or more are WARN.
    4. **READY**: every applicable gate is PASS.
    
    Format: `[PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>`
    
    ## Cost Awareness
    
    - **EKS Insights API** (Step 3) is free — always use first.
    - **CloudWatch Logs Insights** cost ~$0.0076/GB scanned. Default to 60-min windows.
    - **Extended support** costs $0.60/cluster/hour — upgrading saves money.
    - **Surge nodes** incur temporary EC2 cost during overlap period.
    
    ## Required Permissions
    
    **AWS IAM** — see README.md "Prerequisites → IAM Permissions" for the full
    read-only action list (`eks:Describe*`, `eks:List*`, `ec2:Describe*`,
    `autoscaling:Describe*`, `iam:GetRole`, `servicequotas:GetServiceQuota`).
    
    **Kubernetes RBAC** (only if `kubectl` access is available — the assessment
    still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB
    checks). Read-only `ClusterRole` covering every `kubectl get`/`describe` used
    in this skill:
    
    ```yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: eks-upgrade-readiness-readonly
    rules:
      - apiGroups: [""]
        resources:
          - nodes
          - pods
          - configmaps
          - secrets
          - events
          - persistentvolumeclaims
          - certificatesigningrequests
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apps"]
        resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["policy"]
        resources: ["poddisruptionbudgets"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["customresourcedefinitions"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["admissionregistration.k8s.io"]
        resources:
          - validatingwebhookconfigurations
          - mutatingwebhookconfigurations
        verbs: ["get", "list", "watch"]
      - apiGroups: ["karpenter.sh"]
        resources: ["nodepools", "nodeclaims"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["karpenter.k8s.aws"]
        resources: ["ec2nodeclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["crd.k8s.amazonaws.com"]
        resources: ["eniconfigs"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses", "csinodes"]
        verbs: ["get", "list", "watch"]
    ```
    
    Bind with a `ClusterRoleBinding` to the identity the agent assumes (e.g. via
    IRSA/Pod Identity or an EKS access entry). `secrets` read access is required
    only for the Helm stored-manifest scan (Step 4) — omit that rule and accept
    UNKNOWN on Helm checks if a customer's security policy disallows it.
    
    ---
    
    ## Step 1: Gather Cluster Context
    
    ```bash
    aws eks describe-cluster --name <cluster-name> --region <region>
    ```
    
    Extract: `cluster.version`, `platformVersion`, `status` (must be ACTIVE),
    `kubernetesNetworkConfig`, `logging.clusterLogging` (audit log must be enabled),
    `resourcesVpcConfig.subnetIds`, `tags` (IaC ownership detection).
    
    Determine **target version**: ask user or default to current + 1 minor.
    Confirm target is in standard support via the EKS release calendar.
    
    ## Step 2: Verify Infrastructure Prerequisites
    
    Check these — failures are **BLOCKERs**:
    
    1. **Subnet IP availability** — need ≥5 IPs per cluster subnet. Mode-aware:
       standard IPv4, prefix delegation, custom networking, IPv6, SGP.
       Use `aws ec2 describe-subnets` with cluster subnet IDs.
    2. **EKS IAM role** — verify role exists with `eks.amazonaws.com` trust.
    3. **KMS key** (if encryption enabled) — verify cluster role has key access.
    4. **Service quota headroom** — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000)
       must have room for surge nodes. Use `aws service-quotas get-service-quota`.
    
    VPC CNI mode and capacity-input detection:
    ```bash
    kubectl get ds aws-node -n kube-system -o json | jq '
      .spec.template.spec.containers[0].env[]
      | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET"))
      | {name, value}'
    ```
    
    ### VPC CNI Surge-Capacity Gate
    
    Do not treat "mode detected" as capacity validated. First calculate the
    managed-node-group surge using `references/capacity-planning.md`, distribute it
    by the node group's AZ placement, then verify the relevant subnet/ENI resource
    for the selected mode. Record the inputs and calculations as evidence; a missing
    mode-specific input is `UNKNOWN`, not PASS.
    
    | Mode | Required assessment before a node surge | Pass condition |
    |------|------------------------------------------|----------------|
    | Standard IPv4 | Inspect `WARM_IP_TARGET`, `MINIMUM_IP_TARGET`, and `WARM_ENI_TARGET` on `aws-node`; use node `status.allocatable.pods` and current pod count to calculate the additional secondary-IP demand for every surge node. | Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand. |
    | Prefix delegation | Confirm `ENABLE_PREFIX_DELEGATION=true`; each IPv4 prefix consumes a `/28` (16 addresses). Calculate required additional prefixes as `ceil(additional_pod_ips / 16)` per affected subnet/AZ. | `floor(availableIpAddressCount / 16)` covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target. |
    | Custom networking | Confirm `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`, enumerate `ENIConfig` resources, and map each node AZ to its `spec.subnet` and security groups. | The **ENIConfig pod subnet**, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ. |
    | Security Groups for Pods | Confirm `ENABLE_POD_ENI=true`, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. | Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute. |
    | IPv6 | Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. | Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6. |
    
    ```bash
    # Custom networking: inspect every AZ-to-pod-subnet mapping
    kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}'
    
    # Security Groups for Pods: inspect trunk/branch interfaces after mode detection
    aws ec2 describe-network-interfaces \
      --filters Name=interface-type,Values=trunk,branch \
      --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}'
    ```
    
    ### Client and CI Tooling Skew (WARN-level, not a blocker)
    
    Operator/CI tooling that is too far behind the target version causes confusing
    failures during and after the upgrade. Check installed versions where available:
    
    ```bash
    kubectl version --client -o json   # client minor version
    eksctl version                      # if eksctl-managed
    helm version --short                # if Helm-managed workloads
    ```
    
    | Tool | Skew Rule | Risk if Violated |
    |------|-----------|-------------------|
    | kubectl | Must be within ±1 minor of the target `kube-apiserver` version (upstream [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/#kubectl)) | Unrecognized fields, API calls silently rejected or misinterpreted |
    | eksctl | Must support the target EKS version (check release notes for the version that added support) | `eksctl upgrade` commands fail or use stale defaults |
    | Helm | 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated | Chart operations may fail independent of the cluster upgrade |
    | Terraform AWS provider | Must be new enough to support any target-version-specific attributes in use (e.g. `upgrade_policy`, `compute_config` for Auto Mode) — check the [provider changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md) for the attribute | `terraform apply` fails validation or silently ignores the new attribute |
    
    Do not hardcode exact version floors here — they shift every EKS release.
    Report the installed version, the rule, and a WARN if it cannot be confirmed
    current; never treat "tool not detected" as PASS.
    
    ## Step 3: Check EKS Upgrade Insights
    
    **Primary authoritative signal.** Always query first.
    
    ```bash
    aws eks list-insights --cluster-name <cluster> \
      --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}'
    aws eks describe-insight --cluster-name <cluster> --id <insight-id>
    ```
    
    | Status | Gate | Action |
    |--------|------|--------|
    | ERROR | FAIL | Must fix before upgrade |
    | WARNING | WARN | Recommended fix |
    | PASSING | PASS | No action |
    | UNKNOWN | UNKNOWN | EKS could not evaluate the check; investigate and refresh |
    | None returned | UNKNOWN | Continue other checks |
    
    **Freshness gate:** For every returned summary, capture `lastRefreshTime` and
    `lastTransitionTime`, then call `DescribeInsight` to collect the status,
    affected resources, and recommendation. Treat data as **stale** when
    `lastRefreshTime` is more than 24 hours old at assessment time, or predates a
    known relevant workload/addon change. A stale, missing, inaccessible, or
    unpaginated insight set is `UNKNOWN`; never reuse it as PASS. The assessment
    must not call `StartInsightsRefresh` because this skill is read-only. Instead,
    ask the operator to refresh insights through an approved workflow and rerun the
    assessment after the refresh completes.
    
    **Critical:** Insights does NOT cover Helm stored manifests, CRD deprecations,
    StatefulSets, Karpenter, service quotas, PDBs, or capacity planning.
    
    ## Step 4: API Deprecation Analysis
    
    Check version-specific removal gates relevant to user's target:
    - ≥1.33: AL2 AMI unavailable (Critical)
    - ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed
    - ≥1.25: Dockershim and PodSecurityPolicy removed
    - ≥1.23: In-tree EBS provisioner deprecated
    
    **Helm stored manifests** — the #1 missed blocker. Scan latest deployed
    release secrets for deprecated `apiVersion` lines:
    ```bash
    kubectl get secrets -A -l owner=helm,status=deployed
    # Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest'
    ```
    
    **Third-party CRD deprecations** — check Istio, cert-manager, Karpenter,
    Flux, Argo, Prometheus Operator versions against known deprecation timelines.
    
    Tools: `kubent`, `pluto detect-all-in-cluster`, `helm mapkubeapis --dry-run`.
    See `references/api-deprecations.md` for full removal schedule.
    
    ## Step 5: Addon Compatibility Check
    
    **Managed addons:** Use live API to build compatibility matrix:
    ```bash
    aws eks list-addons --cluster-name <cluster>
    aws eks describe-addon --cluster-name <cluster> --addon-name <name>
    aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target>
    ```
    
    **Self-managed addons:** First compare `ListAddons` with the actual
    `kube-system` workloads. Explicitly detect the three core addons: `aws-node`
    (VPC CNI), `coredns`, and `kube-proxy`. If a core component is absent from the
    managed-addon inventory but present in-cluster, mark it self-managed/custom and
    inspect its image, args, and configuration before assessing target support:
    
    ```bash
    kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \
      jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
    kubectl -n kube-system get deployment coredns -o json | \
      jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
    kubectl -n kube-system get configmap coredns aws-node -o yaml
    ```
    
    For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI,
    validate custom environment/config-map values and the mode-specific capacity
    gate in Step 2. For kube-proxy, validate its deployed mode and version against
    its upstream support policy. Then scan other self-managed components
    (aws-load-balancer, external-dns, metrics-server, cluster-autoscaler,
    cert-manager, ingress-nginx, argocd, flux); extract image tags and validate
    against their K8s support matrices.
    
    **Pod Identity Agent (`eks-pod-identity-agent`):** Check like any other managed
    addon via `DescribeAddon` / `DescribeAddonVersions` — its version gates which
    association features are available (e.g. multiple associations per pod,
    target IAM role sessions). If not installed but `kubectl get pods -A -o json`
    shows service accounts with `eks.amazonaws.com/role-arn` annotations instead,
    the cluster is on IRSA, not Pod Identity — note this for blue-green planning
    (see `references/upgrade-troubleshooting.md` → Identity Migration Considerations).
    
    **Upgrade order:** Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks.
    Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed.
    
    See `references/addon-version-matrix.md` for static fallback reference.
    
    ## Step 6: Full Data Plane Inventory
    
    Inventory ALL node populations. See `references/data-plane-inventory.md` for
    complete detection commands.
    
    - **MNG:** `aws eks list-nodegroups` + `describe-nodegroup` for each
    - **Self-managed ASGs:** Find by cluster tag in `describe-auto-scaling-groups`
    - **Karpenter:** NodePools, EC2NodeClasses, controller version and health
    - **Auto Mode:** Check `cluster.computeConfig.enabled`
    - **Fargate:** `aws eks list-fargate-profiles` + describe each
    - **Kubelet versions:** `kubectl get nodes` — confirm within skew window
    
    Version skew requires two independent predicates:
    
    1. **Current-state upper bound:** no kubelet may be newer than the **current**
       control plane (`kubelet_minor <= current_control_plane_minor`). A node
       already newer than the current API server is an invalid state and must be
       corrected before planning the upgrade.
    2. **Target lower bound:** for target 1.X, kubelet must be at least 1.(X-3)
       when X>=28, or 1.(X-2) when X<28.
    
    Any node violating either predicate is a **FAIL**.
    
    ## Step 7: AL2 → AL2023 Migration Assessment
    
    If AL2 detected and target ≥1.33: **CRITICAL** blocker (EKS releases AL2
    AMIs only through 1.32). If AL2 is detected with a target <1.33: **WARNING** —
    upstream Amazon Linux 2 reaches end of life on June 30, 2026.
    
    Assess: bootstrap method (bootstrap.sh vs nodeadm), custom AMIs, user data
    compatibility (yum→dnf, kubelet-extra-args→NodeConfig), cgroup v2 workload
    compatibility, IMDSv2 readiness.
    
    See `references/al2-al2023-migration.md` for full detection commands and
    migration strategy.
    
    ## Step 8: Upgrade Ordering and Pre-Upgrade Alignment
    
    **Pre-CP:** Karpenter (if needed), Cluster Autoscaler (must match target),
    admission webhooks with `failurePolicy: Fail`, custom controllers using
    deprecated APIs.
    
    **Post-CP:** Standard addon and node group upgrade order (Step 5).
    
    Webhook check:
    ```bash
    kubectl get validatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'
    kubectl get mutatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'
    ```
    
    ## Step 9: PDB, Topology Spread, and Workload Safety
    
    **PDB blockers:** `maxUnavailable: 0`, `minAvailable` == replicas, orphaned PDBs:
    ```bash
    kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)'
    ```
    
    **Pre-drain safety (DRAIN-01 to DRAIN-06):** Bare pods, emptyDir data loss,
    custom finalizers, EBS AZ-pinning, webhook deadlock, CoreDNS SPOF.
    See `references/pre-drain-safety.md` for full detection commands.
    
    **TopologySpreadConstraints:** Flag multi-replica deployments without topology spread.
    
    **StatefulSet safety:** Check `terminationGracePeriodSeconds != 0`, PVC retention
    policy, single-replica without PDB, update strategy.
    
    **Scaled-to-zero workloads:** Detect and flag for separate validation.
    
    ## Step 10: Pre-Upgrade Cluster Health Baseline
    
    Confirm healthy steady state before upgrade. Failures compound on unhealthy clusters.
    
    - **Node health:** All nodes Ready, no MemoryPressure/DiskPressure/PIDPressure
    - **Pending CSRs:** Indicate node registration issues
    - **Crash-looping system pods:** Check kube-system, monitoring, ingress namespaces
    - **Metrics and DNS baseline:** Verify metrics-server and CoreDNS responding
    
    Record baselines for post-upgrade comparison.
    
    ## Step 11: Fargate Considerations
    
    Fargate pods upgrade when redeployed after CP upgrade. All Fargate pods must be
    restarted post-upgrade. Restart command is in Step 14 (mutation, requires approval).
    
    ## Step 12: Management Plane and IaC Ownership Detection
    
    Detect management method to route remediation correctly:
    
    | Detection | Management Plane | Mutation Routing |
    |-----------|-----------------|-----------------|
    | ACK CRD + Cluster CR | ACK | Patch ACK Cluster CR |
    | ACK CR with `kro.run/owned` | KRO over ACK | Patch kro instance |
    | Tags: `terraform:*` | Terraform | Update .tf, `terraform apply` |
    | Tags: `aws:cloudformation:*` | CloudFormation | Update template, stack update |
    | Tags: `aws:cdk:*` | CDK | Update construct, `cdk deploy` |
    | Tags: `eksctl.cluster.k8s.io/*` | eksctl | Update config, `eksctl upgrade` |
    | Labels: `argocd.argoproj.io/*` | ArgoCD | Update Git source, sync |
    | Labels: `kustomize.toolkit.fluxcd.io/*` | Flux | Update Git source, reconcile |
    | Tags: `pulumi:*` | Pulumi | Update program, `pulumi up` |
    | None found | unknown | Block mutations until confirmed |
    
    Route ALL remediation through the owning tool — never suggest direct AWS CLI
    when IaC is detected (causes drift).
    
    ## Step 13: Autoscaler Pause During Node Rotation
    
    During upgrades, autoscalers can interfere with rolling replacement. Check
    current Karpenter consolidation config and Cluster Autoscaler scale-down state.
    Recommend pausing both before node rotation and re-enabling after completion.
    
    Pause commands are in Step 14 (mutations, require operator approval).
    
    ## Step 14: Remediation Playbook (Operator Approval Required)
    
    > ⚠️ **ALL commands in this section are MUTATIONS.** The agent MUST NOT execute
    > these — present as a playbook for operator review.
    
    - **14.1** Helm stored manifest fix: `helm mapkubeapis` + `helm upgrade`
    - **14.2** Addon conflict resolution: first capture `DescribeAddon` output and
      `configurationValues`; use `--resolve-conflicts PRESERVE` to retain reviewed
      custom configuration, or `OVERWRITE` only after approving replacement with
      EKS defaults and recording rollback steps. `OVERWRITE` can discard custom
      configuration.
    - **14.3** Fargate pod restart: `kubectl rollout restart` across namespaces
    - **14.4** PDB temporary adjustment: `kubectl patch pdb` (revert after upgrade)
    - **14.5** Karpenter pause: `kubectl annotate nodepools --all "karpenter.sh/do-not-disrupt=true"`
    - **14.6** CA scale-down pause: patch CA config `scale-down-enabled=false`
    - **14.7** Node group upgrade: MNG via `update-nodegroup-version`, Karpenter via
      EC2NodeClass patch (drift), self-managed via launch template update
    
    ## Step 15: Post-Upgrade Functional Validation
    
    Present as validation checklist for operator:
    - DNS resolution (nslookup kubernetes.default)
    - Metrics server (kubectl top nodes/pods)
    - Pod scheduling (run test pod)
    - Load balancer health (target group check)
    - IRSA / Pod Identity (sts get-caller-identity from pod)
    - CoreDNS and kube-proxy pods running
    - Compare against Step 10 baseline (node count, no new CrashLoopBackOff)
    
    ## Step 16: Generate Upgrade Plan and Report
    
    **Execution Order:**
    1. Pre-upgrade alignment (Karpenter/CA/webhooks)
    2. Pause autoscalers
    3. Control plane upgrade (15-40 min)
    4. Wait for ACTIVE status
    5. kube-proxy → vpc-cni → coredns → other managed addons
    6. Self-managed addons
    7. Node groups (one at a time, validate between)
    8. Karpenter nodes (drift-based)
    9. Self-managed nodes (launch template update)
    10. Fargate pods (restart)
    11. Re-enable autoscalers
    12. Post-upgrade validation
    
    **Rollback Matrix:**
    
    | Component | Reversibility | Method |
    |-----------|--------------|--------|
    | Control plane | CONDITIONAL (7-day window) | `aws eks update-cluster-version --kubernetes-version <N-1>` |
    | Addons | FULL | Downgrade to previous version |
    | MNG | PARTIAL | Can halt; completed nodes stay |
    | Karpenter nodes | FULL | Revert EC2NodeClass |
    | Self-managed | FULL | Revert launch template |
    | Fargate | FULL | Redeploy previous config |
    
    **Rollback eligibility has two phases:**
    
    - **Pre-upgrade (advisory only):** confirm the planned upgrade is one minor,
      document the 7-day window and component rollback order, but do not claim the
      future cluster will be eligible. Rollback readiness insights do not exist
      until after an eligible upgrade completes.
    - **Post-upgrade (authoritative):** while the cluster is ACTIVE and still
      inside the 7-day window, run
      `aws eks list-insights --cluster-name <cluster> --filter '{"categories":["ROLLBACK_READINESS"]}'`,
      paginate, then `describe-insight` for each entry. `ERROR` blocks a normal
      rollback; `UNKNOWN` means EKS could not evaluate readiness and also blocks a
      normal rollback. Only `PASSING` insights support an eligible rollback.
    
    This assessment reports the result but never performs `update-cluster-version`
    or a forced rollback.
    
    ## Step 17: Report Format
    
    ```
    ## EKS Upgrade Readiness Report
    **Cluster:** <name> (<region>)
    **Current Version:** <current>
    **Target Version:** <target>
    **Assessment Date:** <date>
    **Management Plane:** <detected>
    **Evidence Completeness:** <X>% (<performed>/<applicable>)
    **Overall Readiness:** READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE
    
    ### Pre-Upgrade Health Baseline
    - [PASS/FAIL] (confidence: HIGH) All nodes Ready
    - [PASS/FAIL] (confidence: HIGH) No pending CSRs
    - [PASS/FAIL] (confidence: HIGH) No crash-looping system pods
    - [PASS/FAIL] (confidence: HIGH) DNS resolution working
    - [PASS/FAIL] (confidence: HIGH) Metrics server responding
    
    ### Infrastructure Prerequisites
    - [PASS/FAIL] (confidence: HIGH) Subnet IP availability (mode: <type>)
    - [PASS/FAIL] (confidence: HIGH) EKS IAM role valid
    - [PASS/FAIL/N/A] (confidence: HIGH) KMS key access
    - [PASS/FAIL] (confidence: HIGH) EC2 vCPU quota headroom
    - [PASS/FAIL] (confidence: HIGH) EBS volume quota headroom
    
    ### EKS Upgrade Insights
    - [PASS/FAIL/UNKNOWN] (confidence: HIGH) <summary>
    
    ### Data Plane Inventory
    - Managed Node Groups: <count> (versions: <list>)
    - Self-Managed ASGs: <count> (versions: <list>)
    - Karpenter NodePools: <count> (version: <ver>)
    - Fargate Profiles: <count>
    - Total Nodes: <count>
    
    ### Blockers (must fix)
    1. [FAIL] (confidence: HIGH) <description> — <remediation>
    
    ### Warnings (recommended)
    1. [WARN] (confidence: MEDIUM) <description> — <recommendation>
    
    ### Passing Checks
    1. [PASS] (confidence: HIGH) <description>
    
    ### Unknown / Not Assessed
    1. [UNKNOWN] <gate> — <reason>
    
    ### Upgrade Plan
    <execution order from Step 16>
    
    ### Rollback Window
    - Rollback eligibility: ELIGIBLE / NOT ELIGIBLE / CHECK AFTER UPGRADE
    - Window: 7 days from CP upgrade completion
    - Note: Add-ons and node groups must be rolled back BEFORE CP
    
    ### Pre-Drain Risks
    - Bare pods (DRAIN-01): <count>
    - EmptyDir data loss (DRAIN-02): <count>
    - EBS AZ-pinning (DRAIN-04): <count>
    - Webhook deadlock (DRAIN-05): <assessment>
    - CoreDNS SPOF (DRAIN-06): <status>
    
    ### Estimated Timeline
    - Control plane: ~30 min
    - Addons: ~5 min each
    - Node groups: ~<X> min per group
    - Total: ~<Y> min
    ```
    
    ### Machine-Readable Output
    
    When the operator asks for a structured result (CI/CD gating, scripted
    polling, dashboards), emit this JSON alongside — never instead of — the
    markdown report. Every gate in the markdown report must have a matching
    entry; the JSON is a serialization of the same evidence, not a summary.
    
    ```json
    {
      "cluster": "<name>",
      "region": "<region>",
      "assessmentTimestamp": "<ISO-8601>",
      "currentVersion": "<current>",
      "targetVersion": "<target>",
      "overallVerdict": "READY | READY_WITH_WARNINGS | NOT_READY | CANNOT_DETERMINE",
      "evidenceCompletenessPct": 0,
      "gates": [
        {
          "id": "<check-id from required-check-registry.yaml, e.g. NODE-04, ADDON-02, INFRA-01>",
          "name": "<human-readable check name>",
          "status": "PASS | FAIL | WARN | UNKNOWN | N_A",
          "confidence": "HIGH | MEDIUM | LOW",
          "evidence": "<short evidence string, same as markdown bullet>",
          "remediation": "<remediation text, or null if PASS>",
          "checkedAt": "<ISO-8601>"
        }
      ],
      "rollback": {
        "eligible": true,
        "windowExpiresAt": "<ISO-8601 or null>"
      }
    }
    ```
    
    `gates[].id` maps 1:1 to the IDs in `references/required-check-registry.yaml`
    (prefixes: `PF-` pre-flight, `INFRA-` infrastructure, `NODE-` node assessment,
    `ADDON-` addon assessment, `WKLD-` workload assessment, `KARP-` Karpenter,
    `DRAIN-` pre-drain safety, `ROLL-` rollback), so a CI pipeline can gate on
    specific check categories (e.g. fail only on `NODE-*` or `ADDON-*` FAILs,
    warn-only on others) instead of just the overall verdict. `overallVerdict`
    follows the same rules as the markdown report — it is never `READY` while
    any gate is `UNKNOWN`.
    
    ## References
    
    See `references/` directory for:
    - `safety-invariants.md` — Hard safety rules, knowledge hierarchy, operation classification
    - `required-check-registry.yaml` — All 60+ checks with IDs, categories, and severity
    - `pre-flight-checks.yaml` — Blocking vs warning checks, timeouts, soak periods, rollback conditions
    - `api-deprecations.md` — Full K8s API removal schedule by version
    - `addon-version-matrix.md` — EKS addon compatibility per version (static fallback)
    - `capacity-planning.md` — FDCR/ODCR and surge capacity guidance
    - `upgrade-troubleshooting.md` — Common failures, feature removals, and tools
    - `karpenter-checks.md` — Full 14-check Karpenter registry (KARP-01 to KARP-14)
    - `pre-drain-safety.md` — DRAIN-01 to DRAIN-06 detection and remediation
    - `al2-al2023-migration.md` — AL2→AL2023 migration assessment details
    - `data-plane-inventory.md` — MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.