eks-upgrade-readiness
Use this skill when a user asks to assess, plan, or validate an
Install
npx skills add https://github.com/aws/tools-for-devops-agent/tree/main/skills/eks-upgrade-readiness
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aws-tools-for-devops-agent@llmmart
git clone https://github.com/aws/tools-for-devops-agent.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aws/tools-for-devops-agent collection as a plugin from our marketplace. Git is the plain clone.
README
EKS Upgrade Readiness Skill
A skill for AWS DevOps Agent that performs read-only upgrade readiness assessments for Amazon EKS clusters, aligned with the AWS EKS Best Practices Guide — Cluster Upgrades.
Purpose
EKS upgrades can fail or cause downtime when deprecated APIs, incompatible addons, version-skewed node groups, or misconfigured PDBs are not caught beforehand. This skill systematically checks every dimension the Best Practices Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE verdict with a prioritized remediation plan.
Key Capabilities
- EKS Upgrade Insights (primary signal) — UPGRADE_READINESS findings from the ListInsights/DescribeInsight APIs
- Infrastructure prerequisites — subnet IP availability (VPC CNI mode-aware), IAM role, KMS key, service quotas
- API deprecation analysis — maps removed APIs to replacements, scans Helm stored manifests (deployed revision), vendor-aware CRD checks
- Addon compatibility (live API) — validates managed addons via DescribeAddonVersions, detects self-managed addons via deployment/Helm scan
- Full data plane inventory — managed node groups, self-managed ASGs, Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles, kubelet version map
- AL2→AL2023 migration — launch template analysis, custom AMI detection, bootstrap differences, cgroup v2, IMDSv2
- Upgrade ordering — pre-upgrade alignment (Karpenter/CA/webhooks before CP) vs post-upgrade sequence
- PDB and topology spread — detects drain blockers and availability risks
- StatefulSet safety — grace period, PVC retention policy, single-replica risks
- Pre-upgrade health baseline — node Ready status, pending CSRs, system pod health, DNS/metrics baseline
- Capacity planning — surge calculation, ODCR/FDCR guidance, blue-green alternative, autoscaler pause
- GitOps/IaC detection — routes remediation through owning tool (Terraform/CDK/ArgoCD/Flux/eksctl)
- Post-upgrade validation — DNS, metrics, scheduling, LB health, IRSA smoke tests
- Client/CI tooling skew — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level)
- Pod Identity awareness — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance
- Remediation playbook — all mutations separated, require operator approval
- Structured upgrade plan — ordered execution with rollback gates
- Machine-readable output — optional JSON verdict (per-gate status, confidence, evidence) alongside the markdown report, for CI/CD gating
Prerequisites
IAM Permissions
The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling:
eks:DescribeCluster
eks:ListClusters
eks:ListInsights
eks:DescribeInsight
eks:ListAddons
eks:DescribeAddon
eks:DescribeAddonVersions
eks:ListNodegroups
eks:DescribeNodegroup
eks:ListFargateProfiles
eks:DescribeFargateProfile
eks:ListUpdates
eks:DescribeUpdate
ec2:DescribeSubnets
ec2:DescribeInstances
ec2:DescribeLaunchTemplateVersions
ec2:DescribeImages
ec2:DescribeCapacityReservations
iam:GetRole
autoscaling:DescribeAutoScalingGroups
autoscaling:DescribeLaunchConfigurations
servicequotas:GetServiceQuota
Kubernetes RBAC (if kubectl access available)
A ClusterRole with read-only access to nodes, pods, deployments, statefulsets,
daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources,
and ENIConfigs. See the "Required Permissions" section in SKILL.md for the
full ClusterRole manifest. kubectl access is optional — the assessment
still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks.
AWS Resources
- One or more Amazon EKS clusters (any supported version)
- Control plane logging enabled (recommended for post-upgrade debugging)
Limitations
- EKS clusters only. Does not cover EKS Anywhere, Outposts, or Local Zones.
- Read-only by design. The skill produces recommendations; it never executes mutating APIs. All mutations are in the Remediation Playbook (Step 14).
- UNKNOWN ≠ PASS. Missing data or access denial produces UNKNOWN, never PASS. The overall verdict cannot be READY while any gate is UNKNOWN.
- Addon version data may lag. Static reference table is fallback only —
always prefer live
describe-addon-versionsAPI. - Pagination required. Large clusters with many node groups or addons require exhausting API pagination tokens.
Agent Types
- Chat tasks — ask for upgrade readiness assessments
Uploading to AWS DevOps Agent
Option A: Import from GitHub (recommended)
If you have a GitHub connection configured in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then
point to skills/eks-upgrade-readiness. See Importing a skill from a repository for full instructions.
Note: You cannot connect the
aws-samplesGitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup.
Option B: Upload as a zip file
- Zip the
eks-upgrade-readiness/directory (only including allowed extensions):
cd skills
zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \
-i '*.md' '*.json' '*.yaml' '*.yml' \
-x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*'
- In the AWS DevOps Agent web app, navigate to the Skills page.
- Click Add skill → Upload skill.
- Drag and drop the
eks-upgrade-readiness.zipfile (max 6 MB). - Select the agent type: Chat tasks.
- Click Upload.
Option C: Upload via the Asset API
Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to the CHAT agent type. See Managing a skill end-to-end for the full API workflow.
How to Use This Skill
Describe the task in natural language — you do not need to name the skill.
Example Prompts
"Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?"
"Check upgrade readiness for all my EKS clusters"
"What deprecated APIs would break if I upgrade to Kubernetes 1.32?"
"Plan the upgrade of my cluster from 1.30 to 1.31 including node groups"
"Are my addons compatible with EKS 1.31?"
"Will my PDBs block a node group upgrade?"
"I need to upgrade a 50-node cluster — what capacity do I need?"
"Compare in-place vs blue-green strategy for my 200-node cluster"
"My cluster is managed by Terraform — how should I do the upgrade?"
"We use AL2 with custom bootstrap scripts — what breaks going to 1.33?"
"We just upgraded to 1.31 — what should we validate?"
"We use Pod Identity and are planning a blue-green migration — what identity work is needed?"
"Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline"
Modes
| Mode | Trigger | Behavior |
|---|---|---|
| Full assessment | "upgrade readiness", "ready to upgrade" | All 17 steps, scored report |
| Targeted check | "deprecated APIs", "addon compatibility", "PDB check" | One dimension, focused |
| Planning | "upgrade plan", "upgrade runbook" | Execution order with rollback gates |
| Comparison | "blue-green vs in-place" | Strategy recommendation |
| Validation | "post-upgrade check", "validate upgrade" | Smoke tests (Step 15) |
Skill Structure
eks-upgrade-readiness/
├── SKILL.md # Main skill instructions (17-step workflow)
├── README.md # This file
├── CHANGELOG.md # Version history
├── .skilleval.yaml # Agent Skill Eval config
├── evals/
│ ├── evals.json # 24 functional evaluation scenarios
│ └── eval_queries.json # Trigger tests (positive and negative)
└── references/
├── safety-invariants.md # Hard safety rules, knowledge hierarchy, operation classification
├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity
├── pre-flight-checks.yaml # Blocking vs warning checks, timeouts, soak periods, rollback conditions
├── api-deprecations.md # K8s API removal schedule by version
├── addon-version-matrix.md # EKS addon compatibility (static fallback)
├── capacity-planning.md # FDCR/ODCR surge capacity guidance
├── upgrade-troubleshooting.md # Tools, feature removals, blue-green
├── karpenter-checks.md # Full 14-check Karpenter registry (KARP-01 to KARP-14)
├── pre-drain-safety.md # DRAIN-01 to DRAIN-06 detection and remediation
├── al2-al2023-migration.md # AL2→AL2023 migration assessment details
└── data-plane-inventory.md # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
Safety
This skill operates in read-only mode:
- No cluster modifications — upgrade actions are recommendations only
- No
update-*,delete-*, orcreate-*API calls - All mutations isolated in Step 14 Remediation Playbook (operator approval)
- All findings include evidence and specific remediation steps
- The operator reviews the report and decides whether to proceed
- UNKNOWN verdicts prevent false confidence (never marks missing data as PASS)
Non-production disclaimer
⚠️ This skill is sample code, not intended for production use without additional review and testing. Validate in a non-production environment first. Compatibility data and version matrices are point-in-time references — always verify with
aws eks describe-addon-versionsfor the latest data.
Skill manifest
EKS Upgrade Readiness
Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade validation aligned with the EKS Best Practices Guide.
When to Use
Activate this skill when the user asks to:
- Check if an EKS cluster is ready to upgrade
- Plan an EKS version upgrade (control plane, node groups, or both)
- Identify deprecated Kubernetes APIs before upgrading
- Validate addon compatibility with a target version
- Assess node group upgrade strategy and capacity requirements
- Review Pod Disruption Budgets or topology spread for upgrade safety
- Understand EKS end-of-support, extended support, or auto-upgrade implications
- Evaluate Karpenter Drift or node expiry upgrade behavior
- Compare in-place vs blue-green upgrade strategies
- Create an upgrade runbook or checklist
- Detect GitOps/IaC version ownership before upgrading
Safety First
Before doing anything, load references/safety-invariants.md. It defines
the knowledge hierarchy, hard rules, operation classification, and uncertainty
handling. Keep it in context for the entire assessment.
Critical Warnings
- This skill is read-only. All commands are
describe*,list*,get*. The agent does NOT execute mutating APIs. Mutations are in Step 14 and require explicit operator approval. - One minor version at a time. EKS control plane upgrades proceed one minor version per operation (e.g., 1.30 → 1.31).
- Version skew policy. Before planning an upgrade, no kubelet may be newer than the current control plane. For the target version, kubelet may be no more than N-3 on 1.28+ (N-2 below 1.28).
- Addons must be upgraded AFTER the control plane (exceptions in Step 8).
- Auto-upgrade policy. Clusters past the 26-month lifecycle will be auto-upgraded. Proactive upgrade avoids disruption.
- Control plane rollback (July 2026+). 7-day rollback window after upgrade. Conditional, not guaranteed — skill checks eligibility.
- UNKNOWN ≠ PASS. Any gate that cannot be assessed MUST be UNKNOWN, never PASS. Overall verdict cannot be READY while any gate is UNKNOWN.
Evidence Completeness
Uses references/required-check-registry.yaml to track checks performed,
skipped, or blocked. EC = checks_performed / total_applicable × 100%.
EC < 50% produces a mandatory warning.
Grading and Confidence
| Level | Meaning | When to Use |
|---|---|---|
| HIGH (90%+) | Confirmed from authoritative source | EKS Insights API, direct kubectl query, AWS API response |
| MEDIUM (60-89%) | Inferred from available data | Partial kubectl access, version matching heuristics |
| LOW (30-59%) | Limited data, possible gaps | No kubectl, no logging enabled, partial API access |
| UNKNOWN | Cannot determine | Tool unavailable, no data, access denied |
False-positive guards:
- Empty query result ≠ PASS (mark UNKNOWN)
- No kubectl ≠ N/A for everything (AWS APIs still work)
- EKS Insights PASSING ≠ skip other checks (covers a subset only)
- Addon "compatible" ≠ "recommended"
- Pagination not exhausted → confidence LOW
Verdict rules (evaluate applicable gates only; N/A gates are excluded):
- NOT READY: one or more applicable gates are FAIL. A known blocker wins over uncertainty because proceeding is unsafe.
- CANNOT DETERMINE: no gate is FAIL, but one or more applicable gates are UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data).
- READY WITH WARNINGS: all applicable gates are assessed, none FAIL or UNKNOWN, and one or more are WARN.
- READY: every applicable gate is PASS.
Format: [PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>
Cost Awareness
- EKS Insights API (Step 3) is free — always use first.
- CloudWatch Logs Insights cost ~$0.0076/GB scanned. Default to 60-min windows.
- Extended support costs $0.60/cluster/hour — upgrading saves money.
- Surge nodes incur temporary EC2 cost during overlap period.
Required Permissions
AWS IAM — see README.md "Prerequisites → IAM Permissions" for the full
read-only action list (eks:Describe*, eks:List*, ec2:Describe*,
autoscaling:Describe*, iam:GetRole, servicequotas:GetServiceQuota).
Kubernetes RBAC (only if kubectl access is available — the assessment
still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB
checks). Read-only ClusterRole covering every kubectl get/describe used
in this skill:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: eks-upgrade-readiness-readonly
rules:
- apiGroups: [""]
resources:
- nodes
- pods
- configmaps
- secrets
- events
- persistentvolumeclaims
- certificatesigningrequests
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch"]
- apiGroups: ["admissionregistration.k8s.io"]
resources:
- validatingwebhookconfigurations
- mutatingwebhookconfigurations
verbs: ["get", "list", "watch"]
- apiGroups: ["karpenter.sh"]
resources: ["nodepools", "nodeclaims"]
verbs: ["get", "list", "watch"]
- apiGroups: ["karpenter.k8s.aws"]
resources: ["ec2nodeclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["crd.k8s.amazonaws.com"]
resources: ["eniconfigs"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses", "csinodes"]
verbs: ["get", "list", "watch"]
Bind with a ClusterRoleBinding to the identity the agent assumes (e.g. via
IRSA/Pod Identity or an EKS access entry). secrets read access is required
only for the Helm stored-manifest scan (Step 4) — omit that rule and accept
UNKNOWN on Helm checks if a customer's security policy disallows it.
Step 1: Gather Cluster Context
aws eks describe-cluster --name <cluster-name> --region <region>
Extract: cluster.version, platformVersion, status (must be ACTIVE),
kubernetesNetworkConfig, logging.clusterLogging (audit log must be enabled),
resourcesVpcConfig.subnetIds, tags (IaC ownership detection).
Determine target version: ask user or default to current + 1 minor. Confirm target is in standard support via the EKS release calendar.
Step 2: Verify Infrastructure Prerequisites
Check these — failures are BLOCKERs:
- Subnet IP availability — need ≥5 IPs per cluster subnet. Mode-aware:
standard IPv4, prefix delegation, custom networking, IPv6, SGP.
Use
aws ec2 describe-subnetswith cluster subnet IDs. - EKS IAM role — verify role exists with
eks.amazonaws.comtrust. - KMS key (if encryption enabled) — verify cluster role has key access.
- Service quota headroom — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000)
must have room for surge nodes. Use
aws service-quotas get-service-quota.
VPC CNI mode and capacity-input detection:
kubectl get ds aws-node -n kube-system -o json | jq '
.spec.template.spec.containers[0].env[]
| select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET"))
| {name, value}'
VPC CNI Surge-Capacity Gate
Do not treat "mode detected" as capacity validated. First calculate the
managed-node-group surge using references/capacity-planning.md, distribute it
by the node group's AZ placement, then verify the relevant subnet/ENI resource
for the selected mode. Record the inputs and calculations as evidence; a missing
mode-specific input is UNKNOWN, not PASS.
| Mode | Required assessment before a node surge | Pass condition |
|---|---|---|
| Standard IPv4 | Inspect WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET on aws-node; use node status.allocatable.pods and current pod count to calculate the additional secondary-IP demand for every surge node. |
Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand. |
| Prefix delegation | Confirm ENABLE_PREFIX_DELEGATION=true; each IPv4 prefix consumes a /28 (16 addresses). Calculate required additional prefixes as ceil(additional_pod_ips / 16) per affected subnet/AZ. |
floor(availableIpAddressCount / 16) covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target. |
| Custom networking | Confirm AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, enumerate ENIConfig resources, and map each node AZ to its spec.subnet and security groups. |
The ENIConfig pod subnet, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ. |
| Security Groups for Pods | Confirm ENABLE_POD_ENI=true, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. |
Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute. |
| IPv6 | Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. | Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6. |
# Custom networking: inspect every AZ-to-pod-subnet mapping
kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}'
# Security Groups for Pods: inspect trunk/branch interfaces after mode detection
aws ec2 describe-network-interfaces \
--filters Name=interface-type,Values=trunk,branch \
--query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}'
Client and CI Tooling Skew (WARN-level, not a blocker)
Operator/CI tooling that is too far behind the target version causes confusing failures during and after the upgrade. Check installed versions where available:
kubectl version --client -o json # client minor version
eksctl version # if eksctl-managed
helm version --short # if Helm-managed workloads
| Tool | Skew Rule | Risk if Violated |
|---|---|---|
| kubectl | Must be within ±1 minor of the target kube-apiserver version (upstream Kubernetes version skew policy) |
Unrecognized fields, API calls silently rejected or misinterpreted |
| eksctl | Must support the target EKS version (check release notes for the version that added support) | eksctl upgrade commands fail or use stale defaults |
| Helm | 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated | Chart operations may fail independent of the cluster upgrade |
| Terraform AWS provider | Must be new enough to support any target-version-specific attributes in use (e.g. upgrade_policy, compute_config for Auto Mode) — check the provider changelog for the attribute |
terraform apply fails validation or silently ignores the new attribute |
Do not hardcode exact version floors here — they shift every EKS release. Report the installed version, the rule, and a WARN if it cannot be confirmed current; never treat "tool not detected" as PASS.
Step 3: Check EKS Upgrade Insights
Primary authoritative signal. Always query first.
aws eks list-insights --cluster-name <cluster> \
--filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}'
aws eks describe-insight --cluster-name <cluster> --id <insight-id>
| Status | Gate | Action |
|---|---|---|
| ERROR | FAIL | Must fix before upgrade |
| WARNING | WARN | Recommended fix |
| PASSING | PASS | No action |
| UNKNOWN | UNKNOWN | EKS could not evaluate the check; investigate and refresh |
| None returned | UNKNOWN | Continue other checks |
Freshness gate: For every returned summary, capture lastRefreshTime and
lastTransitionTime, then call DescribeInsight to collect the status,
affected resources, and recommendation. Treat data as stale when
lastRefreshTime is more than 24 hours old at assessment time, or predates a
known relevant workload/addon change. A stale, missing, inaccessible, or
unpaginated insight set is UNKNOWN; never reuse it as PASS. The assessment
must not call StartInsightsRefresh because this skill is read-only. Instead,
ask the operator to refresh insights through an approved workflow and rerun the
assessment after the refresh completes.
Critical: Insights does NOT cover Helm stored manifests, CRD deprecations, StatefulSets, Karpenter, service quotas, PDBs, or capacity planning.
Step 4: API Deprecation Analysis
Check version-specific removal gates relevant to user's target:
- ≥1.33: AL2 AMI unavailable (Critical)
- ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed
- ≥1.25: Dockershim and PodSecurityPolicy removed
- ≥1.23: In-tree EBS provisioner deprecated
Helm stored manifests — the #1 missed blocker. Scan latest deployed
release secrets for deprecated apiVersion lines:
kubectl get secrets -A -l owner=helm,status=deployed
# Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest'
Third-party CRD deprecations — check Istio, cert-manager, Karpenter, Flux, Argo, Prometheus Operator versions against known deprecation timelines.
Tools: kubent, pluto detect-all-in-cluster, helm mapkubeapis --dry-run.
See references/api-deprecations.md for full removal schedule.
Step 5: Addon Compatibility Check
Managed addons: Use live API to build compatibility matrix:
aws eks list-addons --cluster-name <cluster>
aws eks describe-addon --cluster-name <cluster> --addon-name <name>
aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target>
Self-managed addons: First compare ListAddons with the actual
kube-system workloads. Explicitly detect the three core addons: aws-node
(VPC CNI), coredns, and kube-proxy. If a core component is absent from the
managed-addon inventory but present in-cluster, mark it self-managed/custom and
inspect its image, args, and configuration before assessing target support:
kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \
jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get deployment coredns -o json | \
jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get configmap coredns aws-node -o yaml
For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI, validate custom environment/config-map values and the mode-specific capacity gate in Step 2. For kube-proxy, validate its deployed mode and version against its upstream support policy. Then scan other self-managed components (aws-load-balancer, external-dns, metrics-server, cluster-autoscaler, cert-manager, ingress-nginx, argocd, flux); extract image tags and validate against their K8s support matrices.
Pod Identity Agent (eks-pod-identity-agent): Check like any other managed
addon via DescribeAddon / DescribeAddonVersions — its version gates which
association features are available (e.g. multiple associations per pod,
target IAM role sessions). If not installed but kubectl get pods -A -o json
shows service accounts with eks.amazonaws.com/role-arn annotations instead,
the cluster is on IRSA, not Pod Identity — note this for blue-green planning
(see references/upgrade-troubleshooting.md → Identity Migration Considerations).
Upgrade order: Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks. Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed.
See references/addon-version-matrix.md for static fallback reference.
Step 6: Full Data Plane Inventory
Inventory ALL node populations. See references/data-plane-inventory.md for
complete detection commands.
- MNG:
aws eks list-nodegroups+describe-nodegroupfor each - Self-managed ASGs: Find by cluster tag in
describe-auto-scaling-groups - Karpenter: NodePools, EC2NodeClasses, controller version and health
- Auto Mode: Check
cluster.computeConfig.enabled - Fargate:
aws eks list-fargate-profiles+ describe each - Kubelet versions:
kubectl get nodes— confirm within skew window
Version skew requires two independent predicates:
- Current-state upper bound: no kubelet may be newer than the current
control plane (
kubelet_minor <= current_control_plane_minor). A node already newer than the current API server is an invalid state and must be corrected before planning the upgrade. - Target lower bound: for target 1.X, kubelet must be at least 1.(X-3) when X>=28, or 1.(X-2) when X<28.
Any node violating either predicate is a FAIL.
Step 7: AL2 → AL2023 Migration Assessment
If AL2 detected and target ≥1.33: CRITICAL blocker (EKS releases AL2 AMIs only through 1.32). If AL2 is detected with a target <1.33: WARNING — upstream Amazon Linux 2 reaches end of life on June 30, 2026.
Assess: bootstrap method (bootstrap.sh vs nodeadm), custom AMIs, user data compatibility (yum→dnf, kubelet-extra-args→NodeConfig), cgroup v2 workload compatibility, IMDSv2 readiness.
See references/al2-al2023-migration.md for full detection commands and
migration strategy.
Step 8: Upgrade Ordering and Pre-Upgrade Alignment
Pre-CP: Karpenter (if needed), Cluster Autoscaler (must match target),
admission webhooks with failurePolicy: Fail, custom controllers using
deprecated APIs.
Post-CP: Standard addon and node group upgrade order (Step 5).
Webhook check:
kubectl get validatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'
kubectl get mutatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")'
Step 9: PDB, Topology Spread, and Workload Safety
PDB blockers: maxUnavailable: 0, minAvailable == replicas, orphaned PDBs:
kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)'
Pre-drain safety (DRAIN-01 to DRAIN-06): Bare pods, emptyDir data loss,
custom finalizers, EBS AZ-pinning, webhook deadlock, CoreDNS SPOF.
See references/pre-drain-safety.md for full detection commands.
TopologySpreadConstraints: Flag multi-replica deployments without topology spread.
StatefulSet safety: Check terminationGracePeriodSeconds != 0, PVC retention
policy, single-replica without PDB, update strategy.
Scaled-to-zero workloads: Detect and flag for separate validation.
Step 10: Pre-Upgrade Cluster Health Baseline
Confirm healthy steady state before upgrade. Failures compound on unhealthy clusters.
- Node health: All nodes Ready, no MemoryPressure/DiskPressure/PIDPressure
- Pending CSRs: Indicate node registration issues
- Crash-looping system pods: Check kube-system, monitoring, ingress namespaces
- Metrics and DNS baseline: Verify metrics-server and CoreDNS responding
Record baselines for post-upgrade comparison.
Step 11: Fargate Considerations
Fargate pods upgrade when redeployed after CP upgrade. All Fargate pods must be restarted post-upgrade. Restart command is in Step 14 (mutation, requires approval).
Step 12: Management Plane and IaC Ownership Detection
Detect management method to route remediation correctly:
| Detection | Management Plane | Mutation Routing |
|---|---|---|
| ACK CRD + Cluster CR | ACK | Patch ACK Cluster CR |
ACK CR with kro.run/owned |
KRO over ACK | Patch kro instance |
Tags: terraform:* |
Terraform | Update .tf, terraform apply |
Tags: aws:cloudformation:* |
CloudFormation | Update template, stack update |
Tags: aws:cdk:* |
CDK | Update construct, cdk deploy |
Tags: eksctl.cluster.k8s.io/* |
eksctl | Update config, eksctl upgrade |
Labels: argocd.argoproj.io/* |
ArgoCD | Update Git source, sync |
Labels: kustomize.toolkit.fluxcd.io/* |
Flux | Update Git source, reconcile |
Tags: pulumi:* |
Pulumi | Update program, pulumi up |
| None found | unknown | Block mutations until confirmed |
Route ALL remediation through the owning tool — never suggest direct AWS CLI when IaC is detected (causes drift).
Step 13: Autoscaler Pause During Node Rotation
During upgrades, autoscalers can interfere with rolling replacement. Check current Karpenter consolidation config and Cluster Autoscaler scale-down state. Recommend pausing both before node rotation and re-enabling after completion.
Pause commands are in Step 14 (mutations, require operator approval).
Step 14: Remediation Playbook (Operator Approval Required)
⚠️ ALL commands in this section are MUTATIONS. The agent MUST NOT execute these — present as a playbook for operator review.
- 14.1 Helm stored manifest fix:
helm mapkubeapis+helm upgrade - 14.2 Addon conflict resolution: first capture
DescribeAddonoutput andconfigurationValues; use--resolve-conflicts PRESERVEto retain reviewed custom configuration, orOVERWRITEonly after approving replacement with EKS defaults and recording rollback steps.OVERWRITEcan discard custom configuration. - 14.3 Fargate pod restart:
kubectl rollout restartacross namespaces - 14.4 PDB temporary adjustment:
kubectl patch pdb(revert after upgrade) - 14.5 Karpenter pause:
kubectl annotate nodepools --all "karpenter.sh/do-not-disrupt=true" - 14.6 CA scale-down pause: patch CA config
scale-down-enabled=false - 14.7 Node group upgrade: MNG via
update-nodegroup-version, Karpenter via EC2NodeClass patch (drift), self-managed via launch template update
Step 15: Post-Upgrade Functional Validation
Present as validation checklist for operator:
- DNS resolution (nslookup kubernetes.default)
- Metrics server (kubectl top nodes/pods)
- Pod scheduling (run test pod)
- Load balancer health (target group check)
- IRSA / Pod Identity (sts get-caller-identity from pod)
- CoreDNS and kube-proxy pods running
- Compare against Step 10 baseline (node count, no new CrashLoopBackOff)
Step 16: Generate Upgrade Plan and Report
Execution Order:
- Pre-upgrade alignment (Karpenter/CA/webhooks)
- Pause autoscalers
- Control plane upgrade (15-40 min)
- Wait for ACTIVE status
- kube-proxy → vpc-cni → coredns → other managed addons
- Self-managed addons
- Node groups (one at a time, validate between)
- Karpenter nodes (drift-based)
- Self-managed nodes (launch template update)
- Fargate pods (restart)
- Re-enable autoscalers
- Post-upgrade validation
Rollback Matrix:
| Component | Reversibility | Method |
|---|---|---|
| Control plane | CONDITIONAL (7-day window) | aws eks update-cluster-version --kubernetes-version <N-1> |
| Addons | FULL | Downgrade to previous version |
| MNG | PARTIAL | Can halt; completed nodes stay |
| Karpenter nodes | FULL | Revert EC2NodeClass |
| Self-managed | FULL | Revert launch template |
| Fargate | FULL | Redeploy previous config |
Rollback eligibility has two phases:
- Pre-upgrade (advisory only): confirm the planned upgrade is one minor, document the 7-day window and component rollback order, but do not claim the future cluster will be eligible. Rollback readiness insights do not exist until after an eligible upgrade completes.
- Post-upgrade (authoritative): while the cluster is ACTIVE and still
inside the 7-day window, run
aws eks list-insights --cluster-name <cluster> --filter '{"categories":["ROLLBACK_READINESS"]}', paginate, thendescribe-insightfor each entry.ERRORblocks a normal rollback;UNKNOWNmeans EKS could not evaluate readiness and also blocks a normal rollback. OnlyPASSINGinsights support an eligible rollback.
This assessment reports the result but never performs update-cluster-version
or a forced rollback.
Step 17: Report Format
## EKS Upgrade Readiness Report
**Cluster:** <name> (<region>)
**Current Version:** <current>
**Target Version:** <target>
**Assessment Date:** <date>
**Management Plane:** <detected>
**Evidence Completeness:** <X>% (<performed>/<applicable>)
**Overall Readiness:** READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE
### Pre-Upgrade Health Baseline
- [PASS/FAIL] (confidence: HIGH) All nodes Ready
- [PASS/FAIL] (confidence: HIGH) No pending CSRs
- [PASS/FAIL] (confidence: HIGH) No crash-looping system pods
- [PASS/FAIL] (confidence: HIGH) DNS resolution working
- [PASS/FAIL] (confidence: HIGH) Metrics server responding
### Infrastructure Prerequisites
- [PASS/FAIL] (confidence: HIGH) Subnet IP availability (mode: <type>)
- [PASS/FAIL] (confidence: HIGH) EKS IAM role valid
- [PASS/FAIL/N/A] (confidence: HIGH) KMS key access
- [PASS/FAIL] (confidence: HIGH) EC2 vCPU quota headroom
- [PASS/FAIL] (confidence: HIGH) EBS volume quota headroom
### EKS Upgrade Insights
- [PASS/FAIL/UNKNOWN] (confidence: HIGH) <summary>
### Data Plane Inventory
- Managed Node Groups: <count> (versions: <list>)
- Self-Managed ASGs: <count> (versions: <list>)
- Karpenter NodePools: <count> (version: <ver>)
- Fargate Profiles: <count>
- Total Nodes: <count>
### Blockers (must fix)
1. [FAIL] (confidence: HIGH) <description> — <remediation>
### Warnings (recommended)
1. [WARN] (confidence: MEDIUM) <description> — <recommendation>
### Passing Checks
1. [PASS] (confidence: HIGH) <description>
### Unknown / Not Assessed
1. [UNKNOWN] <gate> — <reason>
### Upgrade Plan
<execution order from Step 16>
### Rollback Window
- Rollback eligibility: ELIGIBLE / NOT ELIGIBLE / CHECK AFTER UPGRADE
- Window: 7 days from CP upgrade completion
- Note: Add-ons and node groups must be rolled back BEFORE CP
### Pre-Drain Risks
- Bare pods (DRAIN-01): <count>
- EmptyDir data loss (DRAIN-02): <count>
- EBS AZ-pinning (DRAIN-04): <count>
- Webhook deadlock (DRAIN-05): <assessment>
- CoreDNS SPOF (DRAIN-06): <status>
### Estimated Timeline
- Control plane: ~30 min
- Addons: ~5 min each
- Node groups: ~<X> min per group
- Total: ~<Y> min
Machine-Readable Output
When the operator asks for a structured result (CI/CD gating, scripted polling, dashboards), emit this JSON alongside — never instead of — the markdown report. Every gate in the markdown report must have a matching entry; the JSON is a serialization of the same evidence, not a summary.
{
"cluster": "<name>",
"region": "<region>",
"assessmentTimestamp": "<ISO-8601>",
"currentVersion": "<current>",
"targetVersion": "<target>",
"overallVerdict": "READY | READY_WITH_WARNINGS | NOT_READY | CANNOT_DETERMINE",
"evidenceCompletenessPct": 0,
"gates": [
{
"id": "<check-id from required-check-registry.yaml, e.g. NODE-04, ADDON-02, INFRA-01>",
"name": "<human-readable check name>",
"status": "PASS | FAIL | WARN | UNKNOWN | N_A",
"confidence": "HIGH | MEDIUM | LOW",
"evidence": "<short evidence string, same as markdown bullet>",
"remediation": "<remediation text, or null if PASS>",
"checkedAt": "<ISO-8601>"
}
],
"rollback": {
"eligible": true,
"windowExpiresAt": "<ISO-8601 or null>"
}
}
gates[].id maps 1:1 to the IDs in references/required-check-registry.yaml
(prefixes: PF- pre-flight, INFRA- infrastructure, NODE- node assessment,
ADDON- addon assessment, WKLD- workload assessment, KARP- Karpenter,
DRAIN- pre-drain safety, ROLL- rollback), so a CI pipeline can gate on
specific check categories (e.g. fail only on NODE-* or ADDON-* FAILs,
warn-only on others) instead of just the overall verdict. overallVerdict
follows the same rules as the markdown report — it is never READY while
any gate is UNKNOWN.
References
See references/ directory for:
safety-invariants.md— Hard safety rules, knowledge hierarchy, operation classificationrequired-check-registry.yaml— All 60+ checks with IDs, categories, and severitypre-flight-checks.yaml— Blocking vs warning checks, timeouts, soak periods, rollback conditionsapi-deprecations.md— Full K8s API removal schedule by versionaddon-version-matrix.md— EKS addon compatibility per version (static fallback)capacity-planning.md— FDCR/ODCR and surge capacity guidanceupgrade-troubleshooting.md— Common failures, feature removals, and toolskarpenter-checks.md— Full 14-check Karpenter registry (KARP-01 to KARP-14)pre-drain-safety.md— DRAIN-01 to DRAIN-06 detection and remediational2-al2023-migration.md— AL2→AL2023 migration assessment detailsdata-plane-inventory.md— MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
Files (tools-for-devops-agent)
-
evals
-
evals.json 16.4 KB
Loading… -
eval_queries.json 3.5 KB
Loading…
-
-
references
-
addon-version-matrix.md 5.4 KB
Loading… -
al2-al2023-migration.md 6 KB
Loading… -
api-deprecations.md 3.6 KB
Loading… -
capacity-planning.md 5.9 KB
Loading… -
data-plane-inventory.md 6.9 KB
Loading… -
karpenter-checks.md 5.5 KB
Loading… -
pre-drain-safety.md 5.9 KB
Loading… -
pre-flight-checks.yaml 8 KB
Loading… -
required-check-registry.yaml 16 KB
Loading… -
safety-invariants.md 4.1 KB
Loading… -
upgrade-troubleshooting.md 11.5 KB
Loading…
-
-
.skilleval.yaml 77 B
Loading… -
CHANGELOG.md 9.3 KB
Loading… -
README.md 10.3 KB
# EKS Upgrade Readiness Skill A skill for AWS DevOps Agent that performs **read-only** upgrade readiness assessments for Amazon EKS clusters, aligned with the [AWS EKS Best Practices Guide — Cluster Upgrades](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html). ## Purpose EKS upgrades can fail or cause downtime when deprecated APIs, incompatible addons, version-skewed node groups, or misconfigured PDBs are not caught beforehand. This skill systematically checks every dimension the Best Practices Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE verdict with a prioritized remediation plan. ## Key Capabilities - **EKS Upgrade Insights (primary signal)** — UPGRADE_READINESS findings from the ListInsights/DescribeInsight APIs - **Infrastructure prerequisites** — subnet IP availability (VPC CNI mode-aware), IAM role, KMS key, service quotas - **API deprecation analysis** — maps removed APIs to replacements, scans Helm stored manifests (deployed revision), vendor-aware CRD checks - **Addon compatibility (live API)** — validates managed addons via DescribeAddonVersions, detects self-managed addons via deployment/Helm scan - **Full data plane inventory** — managed node groups, self-managed ASGs, Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles, kubelet version map - **AL2→AL2023 migration** — launch template analysis, custom AMI detection, bootstrap differences, cgroup v2, IMDSv2 - **Upgrade ordering** — pre-upgrade alignment (Karpenter/CA/webhooks before CP) vs post-upgrade sequence - **PDB and topology spread** — detects drain blockers and availability risks - **StatefulSet safety** — grace period, PVC retention policy, single-replica risks - **Pre-upgrade health baseline** — node Ready status, pending CSRs, system pod health, DNS/metrics baseline - **Capacity planning** — surge calculation, ODCR/FDCR guidance, blue-green alternative, autoscaler pause - **GitOps/IaC detection** — routes remediation through owning tool (Terraform/CDK/ArgoCD/Flux/eksctl) - **Post-upgrade validation** — DNS, metrics, scheduling, LB health, IRSA smoke tests - **Client/CI tooling skew** — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level) - **Pod Identity awareness** — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance - **Remediation playbook** — all mutations separated, require operator approval - **Structured upgrade plan** — ordered execution with rollback gates - **Machine-readable output** — optional JSON verdict (per-gate status, confidence, evidence) alongside the markdown report, for CI/CD gating ## Prerequisites ### IAM Permissions The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling: ``` eks:DescribeCluster eks:ListClusters eks:ListInsights eks:DescribeInsight eks:ListAddons eks:DescribeAddon eks:DescribeAddonVersions eks:ListNodegroups eks:DescribeNodegroup eks:ListFargateProfiles eks:DescribeFargateProfile eks:ListUpdates eks:DescribeUpdate ec2:DescribeSubnets ec2:DescribeInstances ec2:DescribeLaunchTemplateVersions ec2:DescribeImages ec2:DescribeCapacityReservations iam:GetRole autoscaling:DescribeAutoScalingGroups autoscaling:DescribeLaunchConfigurations servicequotas:GetServiceQuota ``` ### Kubernetes RBAC (if kubectl access available) A `ClusterRole` with read-only access to nodes, pods, deployments, statefulsets, daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources, and ENIConfigs. See the "Required Permissions" section in SKILL.md for the full `ClusterRole` manifest. `kubectl` access is optional — the assessment still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks. ### AWS Resources - One or more Amazon EKS clusters (any supported version) - Control plane logging enabled (recommended for post-upgrade debugging) ## Limitations - **EKS clusters only.** Does not cover EKS Anywhere, Outposts, or Local Zones. - **Read-only by design.** The skill produces recommendations; it never executes mutating APIs. All mutations are in the Remediation Playbook (Step 14). - **UNKNOWN ≠ PASS.** Missing data or access denial produces UNKNOWN, never PASS. The overall verdict cannot be READY while any gate is UNKNOWN. - **Addon version data may lag.** Static reference table is fallback only — always prefer live `describe-addon-versions` API. - **Pagination required.** Large clusters with many node groups or addons require exhausting API pagination tokens. ## Agent Types - **Chat tasks** — ask for upgrade readiness assessments ## Uploading to AWS DevOps Agent **Option A: Import from GitHub (recommended)** If you have a [GitHub connection configured](https://docs.aws.amazon.com/devopsagent/latest/userguide/connecting-to-cicd-pipelines-connecting-github.html) in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then point to `skills/eks-upgrade-readiness`. See [Importing a skill from a repository](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#creating-skills) for full instructions. > **Note:** You cannot connect the `aws-samples` GitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup. **Option B: Upload as a zip file** 1. Zip the `eks-upgrade-readiness/` directory (only including allowed extensions): ```bash cd skills zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \ -i '*.md' '*.json' '*.yaml' '*.yml' \ -x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*' ``` 2. In the AWS DevOps Agent web app, navigate to the **Skills** page. 3. Click **Add skill** → **Upload skill**. 4. Drag and drop the `eks-upgrade-readiness.zip` file (max 6 MB). 5. Select the agent type: **Chat tasks**. 6. Click **Upload**. **Option C: Upload via the Asset API** Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to the `CHAT` agent type. See [Managing a skill end-to-end](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-managing-assets.html#managing-a-skill-end-to-end) for the full API workflow. ## How to Use This Skill Describe the task in natural language — you do not need to name the skill. ### Example Prompts ``` "Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?" "Check upgrade readiness for all my EKS clusters" "What deprecated APIs would break if I upgrade to Kubernetes 1.32?" "Plan the upgrade of my cluster from 1.30 to 1.31 including node groups" "Are my addons compatible with EKS 1.31?" "Will my PDBs block a node group upgrade?" "I need to upgrade a 50-node cluster — what capacity do I need?" "Compare in-place vs blue-green strategy for my 200-node cluster" "My cluster is managed by Terraform — how should I do the upgrade?" "We use AL2 with custom bootstrap scripts — what breaks going to 1.33?" "We just upgraded to 1.31 — what should we validate?" "We use Pod Identity and are planning a blue-green migration — what identity work is needed?" "Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline" ``` ### Modes | Mode | Trigger | Behavior | |------|---------|----------| | Full assessment | "upgrade readiness", "ready to upgrade" | All 17 steps, scored report | | Targeted check | "deprecated APIs", "addon compatibility", "PDB check" | One dimension, focused | | Planning | "upgrade plan", "upgrade runbook" | Execution order with rollback gates | | Comparison | "blue-green vs in-place" | Strategy recommendation | | Validation | "post-upgrade check", "validate upgrade" | Smoke tests (Step 15) | ## Skill Structure ``` eks-upgrade-readiness/ ├── SKILL.md # Main skill instructions (17-step workflow) ├── README.md # This file ├── CHANGELOG.md # Version history ├── .skilleval.yaml # Agent Skill Eval config ├── evals/ │ ├── evals.json # 24 functional evaluation scenarios │ └── eval_queries.json # Trigger tests (positive and negative) └── references/ ├── safety-invariants.md # Hard safety rules, knowledge hierarchy, operation classification ├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity ├── pre-flight-checks.yaml # Blocking vs warning checks, timeouts, soak periods, rollback conditions ├── api-deprecations.md # K8s API removal schedule by version ├── addon-version-matrix.md # EKS addon compatibility (static fallback) ├── capacity-planning.md # FDCR/ODCR surge capacity guidance ├── upgrade-troubleshooting.md # Tools, feature removals, blue-green ├── karpenter-checks.md # Full 14-check Karpenter registry (KARP-01 to KARP-14) ├── pre-drain-safety.md # DRAIN-01 to DRAIN-06 detection and remediation ├── al2-al2023-migration.md # AL2→AL2023 migration assessment details └── data-plane-inventory.md # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands ``` ## Safety This skill operates in **read-only** mode: - No cluster modifications — upgrade actions are recommendations only - No `update-*`, `delete-*`, or `create-*` API calls - All mutations isolated in Step 14 Remediation Playbook (operator approval) - All findings include evidence and specific remediation steps - The operator reviews the report and decides whether to proceed - UNKNOWN verdicts prevent false confidence (never marks missing data as PASS) ## Non-production disclaimer > ⚠️ This skill is sample code, not intended for production use without > additional review and testing. Validate in a non-production environment first. > Compatibility data and version matrices are point-in-time references — always > verify with `aws eks describe-addon-versions` for the latest data. -
SKILL.md 30.3 KB
--- name: eks-upgrade-readiness description: Use this skill when a user asks to assess, plan, or validate an Amazon EKS cluster upgrade. Activate on requests mentioning "EKS upgrade", "Kubernetes version upgrade", "upgrade readiness", "upgrade plan", "pre-upgrade check", "version skew", "deprecated API", "addon compatibility", "node group upgrade", "control plane upgrade", "EKS end of support", "EKS extended support", "Karpenter drift", "kubelet version skew", or "blue-green cluster migration". It runs a pre-upgrade assessment per the AWS EKS Best Practices Guide, covering infrastructure prerequisites, EKS Upgrade Insights, API deprecations, addon compatibility, full data plane inventory (managed/self-managed nodes, Karpenter, Auto Mode, Fargate), AL2 to AL2023 migration, PDB/topology and StatefulSet safety, capacity planning, and pre/post-upgrade validation, then produces a scored readiness verdict with prioritized remediation. Do NOT use for ECS, general EKS troubleshooting unrelated to version upgrades, or EKS Anywhere/Outpost clusters. metadata: author: LearningNewbie version: "2.0.2" aws-devops-agent-skills.agent-types: "CHAT" aws-devops-agent-skills.aws-services: "Amazon EKS" aws-devops-agent-skills.technical-domains: "Containers" --- # EKS Upgrade Readiness Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade validation aligned with the [EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html). ## When to Use Activate this skill when the user asks to: - Check if an EKS cluster is ready to upgrade - Plan an EKS version upgrade (control plane, node groups, or both) - Identify deprecated Kubernetes APIs before upgrading - Validate addon compatibility with a target version - Assess node group upgrade strategy and capacity requirements - Review Pod Disruption Budgets or topology spread for upgrade safety - Understand EKS end-of-support, extended support, or auto-upgrade implications - Evaluate Karpenter Drift or node expiry upgrade behavior - Compare in-place vs blue-green upgrade strategies - Create an upgrade runbook or checklist - Detect GitOps/IaC version ownership before upgrading ## Safety First **Before doing anything, load `references/safety-invariants.md`.** It defines the knowledge hierarchy, hard rules, operation classification, and uncertainty handling. Keep it in context for the entire assessment. ## Critical Warnings - **This skill is read-only.** All commands are `describe*`, `list*`, `get*`. The agent does NOT execute mutating APIs. Mutations are in Step 14 and require explicit operator approval. - **One minor version at a time.** EKS control plane upgrades proceed one minor version per operation (e.g., 1.30 → 1.31). - **Version skew policy.** Before planning an upgrade, no kubelet may be newer than the **current** control plane. For the target version, kubelet may be no more than N-3 on 1.28+ (N-2 below 1.28). - **Addons must be upgraded AFTER the control plane** (exceptions in Step 8). - **Auto-upgrade policy.** Clusters past the 26-month lifecycle will be auto-upgraded. Proactive upgrade avoids disruption. - **Control plane rollback (July 2026+).** 7-day rollback window after upgrade. Conditional, not guaranteed — skill checks eligibility. - **UNKNOWN ≠ PASS.** Any gate that cannot be assessed MUST be UNKNOWN, never PASS. Overall verdict cannot be READY while any gate is UNKNOWN. ## Evidence Completeness Uses `references/required-check-registry.yaml` to track checks performed, skipped, or blocked. EC = checks_performed / total_applicable × 100%. EC < 50% produces a mandatory warning. ## Grading and Confidence | Level | Meaning | When to Use | |-------|---------|-------------| | HIGH (90%+) | Confirmed from authoritative source | EKS Insights API, direct kubectl query, AWS API response | | MEDIUM (60-89%) | Inferred from available data | Partial kubectl access, version matching heuristics | | LOW (30-59%) | Limited data, possible gaps | No kubectl, no logging enabled, partial API access | | UNKNOWN | Cannot determine | Tool unavailable, no data, access denied | **False-positive guards:** - Empty query result ≠ PASS (mark UNKNOWN) - No kubectl ≠ N/A for everything (AWS APIs still work) - EKS Insights PASSING ≠ skip other checks (covers a subset only) - Addon "compatible" ≠ "recommended" - Pagination not exhausted → confidence LOW **Verdict rules (evaluate applicable gates only; `N/A` gates are excluded):** 1. **NOT READY**: one or more applicable gates are FAIL. A known blocker wins over uncertainty because proceeding is unsafe. 2. **CANNOT DETERMINE**: no gate is FAIL, but one or more applicable gates are UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data). 3. **READY WITH WARNINGS**: all applicable gates are assessed, none FAIL or UNKNOWN, and one or more are WARN. 4. **READY**: every applicable gate is PASS. Format: `[PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>` ## Cost Awareness - **EKS Insights API** (Step 3) is free — always use first. - **CloudWatch Logs Insights** cost ~$0.0076/GB scanned. Default to 60-min windows. - **Extended support** costs $0.60/cluster/hour — upgrading saves money. - **Surge nodes** incur temporary EC2 cost during overlap period. ## Required Permissions **AWS IAM** — see README.md "Prerequisites → IAM Permissions" for the full read-only action list (`eks:Describe*`, `eks:List*`, `ec2:Describe*`, `autoscaling:Describe*`, `iam:GetRole`, `servicequotas:GetServiceQuota`). **Kubernetes RBAC** (only if `kubectl` access is available — the assessment still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB checks). Read-only `ClusterRole` covering every `kubectl get`/`describe` used in this skill: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: eks-upgrade-readiness-readonly rules: - apiGroups: [""] resources: - nodes - pods - configmaps - secrets - events - persistentvolumeclaims - certificatesigningrequests verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] verbs: ["get", "list", "watch"] - apiGroups: ["policy"] resources: ["poddisruptionbudgets"] verbs: ["get", "list", "watch"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] verbs: ["get", "list", "watch"] - apiGroups: ["admissionregistration.k8s.io"] resources: - validatingwebhookconfigurations - mutatingwebhookconfigurations verbs: ["get", "list", "watch"] - apiGroups: ["karpenter.sh"] resources: ["nodepools", "nodeclaims"] verbs: ["get", "list", "watch"] - apiGroups: ["karpenter.k8s.aws"] resources: ["ec2nodeclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["crd.k8s.amazonaws.com"] resources: ["eniconfigs"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses", "csinodes"] verbs: ["get", "list", "watch"] ``` Bind with a `ClusterRoleBinding` to the identity the agent assumes (e.g. via IRSA/Pod Identity or an EKS access entry). `secrets` read access is required only for the Helm stored-manifest scan (Step 4) — omit that rule and accept UNKNOWN on Helm checks if a customer's security policy disallows it. --- ## Step 1: Gather Cluster Context ```bash aws eks describe-cluster --name <cluster-name> --region <region> ``` Extract: `cluster.version`, `platformVersion`, `status` (must be ACTIVE), `kubernetesNetworkConfig`, `logging.clusterLogging` (audit log must be enabled), `resourcesVpcConfig.subnetIds`, `tags` (IaC ownership detection). Determine **target version**: ask user or default to current + 1 minor. Confirm target is in standard support via the EKS release calendar. ## Step 2: Verify Infrastructure Prerequisites Check these — failures are **BLOCKERs**: 1. **Subnet IP availability** — need ≥5 IPs per cluster subnet. Mode-aware: standard IPv4, prefix delegation, custom networking, IPv6, SGP. Use `aws ec2 describe-subnets` with cluster subnet IDs. 2. **EKS IAM role** — verify role exists with `eks.amazonaws.com` trust. 3. **KMS key** (if encryption enabled) — verify cluster role has key access. 4. **Service quota headroom** — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000) must have room for surge nodes. Use `aws service-quotas get-service-quota`. VPC CNI mode and capacity-input detection: ```bash kubectl get ds aws-node -n kube-system -o json | jq ' .spec.template.spec.containers[0].env[] | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET")) | {name, value}' ``` ### VPC CNI Surge-Capacity Gate Do not treat "mode detected" as capacity validated. First calculate the managed-node-group surge using `references/capacity-planning.md`, distribute it by the node group's AZ placement, then verify the relevant subnet/ENI resource for the selected mode. Record the inputs and calculations as evidence; a missing mode-specific input is `UNKNOWN`, not PASS. | Mode | Required assessment before a node surge | Pass condition | |------|------------------------------------------|----------------| | Standard IPv4 | Inspect `WARM_IP_TARGET`, `MINIMUM_IP_TARGET`, and `WARM_ENI_TARGET` on `aws-node`; use node `status.allocatable.pods` and current pod count to calculate the additional secondary-IP demand for every surge node. | Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand. | | Prefix delegation | Confirm `ENABLE_PREFIX_DELEGATION=true`; each IPv4 prefix consumes a `/28` (16 addresses). Calculate required additional prefixes as `ceil(additional_pod_ips / 16)` per affected subnet/AZ. | `floor(availableIpAddressCount / 16)` covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target. | | Custom networking | Confirm `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`, enumerate `ENIConfig` resources, and map each node AZ to its `spec.subnet` and security groups. | The **ENIConfig pod subnet**, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ. | | Security Groups for Pods | Confirm `ENABLE_POD_ENI=true`, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. | Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute. | | IPv6 | Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. | Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6. | ```bash # Custom networking: inspect every AZ-to-pod-subnet mapping kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}' # Security Groups for Pods: inspect trunk/branch interfaces after mode detection aws ec2 describe-network-interfaces \ --filters Name=interface-type,Values=trunk,branch \ --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}' ``` ### Client and CI Tooling Skew (WARN-level, not a blocker) Operator/CI tooling that is too far behind the target version causes confusing failures during and after the upgrade. Check installed versions where available: ```bash kubectl version --client -o json # client minor version eksctl version # if eksctl-managed helm version --short # if Helm-managed workloads ``` | Tool | Skew Rule | Risk if Violated | |------|-----------|-------------------| | kubectl | Must be within ±1 minor of the target `kube-apiserver` version (upstream [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/#kubectl)) | Unrecognized fields, API calls silently rejected or misinterpreted | | eksctl | Must support the target EKS version (check release notes for the version that added support) | `eksctl upgrade` commands fail or use stale defaults | | Helm | 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated | Chart operations may fail independent of the cluster upgrade | | Terraform AWS provider | Must be new enough to support any target-version-specific attributes in use (e.g. `upgrade_policy`, `compute_config` for Auto Mode) — check the [provider changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md) for the attribute | `terraform apply` fails validation or silently ignores the new attribute | Do not hardcode exact version floors here — they shift every EKS release. Report the installed version, the rule, and a WARN if it cannot be confirmed current; never treat "tool not detected" as PASS. ## Step 3: Check EKS Upgrade Insights **Primary authoritative signal.** Always query first. ```bash aws eks list-insights --cluster-name <cluster> \ --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}' aws eks describe-insight --cluster-name <cluster> --id <insight-id> ``` | Status | Gate | Action | |--------|------|--------| | ERROR | FAIL | Must fix before upgrade | | WARNING | WARN | Recommended fix | | PASSING | PASS | No action | | UNKNOWN | UNKNOWN | EKS could not evaluate the check; investigate and refresh | | None returned | UNKNOWN | Continue other checks | **Freshness gate:** For every returned summary, capture `lastRefreshTime` and `lastTransitionTime`, then call `DescribeInsight` to collect the status, affected resources, and recommendation. Treat data as **stale** when `lastRefreshTime` is more than 24 hours old at assessment time, or predates a known relevant workload/addon change. A stale, missing, inaccessible, or unpaginated insight set is `UNKNOWN`; never reuse it as PASS. The assessment must not call `StartInsightsRefresh` because this skill is read-only. Instead, ask the operator to refresh insights through an approved workflow and rerun the assessment after the refresh completes. **Critical:** Insights does NOT cover Helm stored manifests, CRD deprecations, StatefulSets, Karpenter, service quotas, PDBs, or capacity planning. ## Step 4: API Deprecation Analysis Check version-specific removal gates relevant to user's target: - ≥1.33: AL2 AMI unavailable (Critical) - ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed - ≥1.25: Dockershim and PodSecurityPolicy removed - ≥1.23: In-tree EBS provisioner deprecated **Helm stored manifests** — the #1 missed blocker. Scan latest deployed release secrets for deprecated `apiVersion` lines: ```bash kubectl get secrets -A -l owner=helm,status=deployed # Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest' ``` **Third-party CRD deprecations** — check Istio, cert-manager, Karpenter, Flux, Argo, Prometheus Operator versions against known deprecation timelines. Tools: `kubent`, `pluto detect-all-in-cluster`, `helm mapkubeapis --dry-run`. See `references/api-deprecations.md` for full removal schedule. ## Step 5: Addon Compatibility Check **Managed addons:** Use live API to build compatibility matrix: ```bash aws eks list-addons --cluster-name <cluster> aws eks describe-addon --cluster-name <cluster> --addon-name <name> aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target> ``` **Self-managed addons:** First compare `ListAddons` with the actual `kube-system` workloads. Explicitly detect the three core addons: `aws-node` (VPC CNI), `coredns`, and `kube-proxy`. If a core component is absent from the managed-addon inventory but present in-cluster, mark it self-managed/custom and inspect its image, args, and configuration before assessing target support: ```bash kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \ jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' kubectl -n kube-system get deployment coredns -o json | \ jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' kubectl -n kube-system get configmap coredns aws-node -o yaml ``` For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI, validate custom environment/config-map values and the mode-specific capacity gate in Step 2. For kube-proxy, validate its deployed mode and version against its upstream support policy. Then scan other self-managed components (aws-load-balancer, external-dns, metrics-server, cluster-autoscaler, cert-manager, ingress-nginx, argocd, flux); extract image tags and validate against their K8s support matrices. **Pod Identity Agent (`eks-pod-identity-agent`):** Check like any other managed addon via `DescribeAddon` / `DescribeAddonVersions` — its version gates which association features are available (e.g. multiple associations per pod, target IAM role sessions). If not installed but `kubectl get pods -A -o json` shows service accounts with `eks.amazonaws.com/role-arn` annotations instead, the cluster is on IRSA, not Pod Identity — note this for blue-green planning (see `references/upgrade-troubleshooting.md` → Identity Migration Considerations). **Upgrade order:** Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks. Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed. See `references/addon-version-matrix.md` for static fallback reference. ## Step 6: Full Data Plane Inventory Inventory ALL node populations. See `references/data-plane-inventory.md` for complete detection commands. - **MNG:** `aws eks list-nodegroups` + `describe-nodegroup` for each - **Self-managed ASGs:** Find by cluster tag in `describe-auto-scaling-groups` - **Karpenter:** NodePools, EC2NodeClasses, controller version and health - **Auto Mode:** Check `cluster.computeConfig.enabled` - **Fargate:** `aws eks list-fargate-profiles` + describe each - **Kubelet versions:** `kubectl get nodes` — confirm within skew window Version skew requires two independent predicates: 1. **Current-state upper bound:** no kubelet may be newer than the **current** control plane (`kubelet_minor <= current_control_plane_minor`). A node already newer than the current API server is an invalid state and must be corrected before planning the upgrade. 2. **Target lower bound:** for target 1.X, kubelet must be at least 1.(X-3) when X>=28, or 1.(X-2) when X<28. Any node violating either predicate is a **FAIL**. ## Step 7: AL2 → AL2023 Migration Assessment If AL2 detected and target ≥1.33: **CRITICAL** blocker (EKS releases AL2 AMIs only through 1.32). If AL2 is detected with a target <1.33: **WARNING** — upstream Amazon Linux 2 reaches end of life on June 30, 2026. Assess: bootstrap method (bootstrap.sh vs nodeadm), custom AMIs, user data compatibility (yum→dnf, kubelet-extra-args→NodeConfig), cgroup v2 workload compatibility, IMDSv2 readiness. See `references/al2-al2023-migration.md` for full detection commands and migration strategy. ## Step 8: Upgrade Ordering and Pre-Upgrade Alignment **Pre-CP:** Karpenter (if needed), Cluster Autoscaler (must match target), admission webhooks with `failurePolicy: Fail`, custom controllers using deprecated APIs. **Post-CP:** Standard addon and node group upgrade order (Step 5). Webhook check: ```bash kubectl get validatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")' kubectl get mutatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")' ``` ## Step 9: PDB, Topology Spread, and Workload Safety **PDB blockers:** `maxUnavailable: 0`, `minAvailable` == replicas, orphaned PDBs: ```bash kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)' ``` **Pre-drain safety (DRAIN-01 to DRAIN-06):** Bare pods, emptyDir data loss, custom finalizers, EBS AZ-pinning, webhook deadlock, CoreDNS SPOF. See `references/pre-drain-safety.md` for full detection commands. **TopologySpreadConstraints:** Flag multi-replica deployments without topology spread. **StatefulSet safety:** Check `terminationGracePeriodSeconds != 0`, PVC retention policy, single-replica without PDB, update strategy. **Scaled-to-zero workloads:** Detect and flag for separate validation. ## Step 10: Pre-Upgrade Cluster Health Baseline Confirm healthy steady state before upgrade. Failures compound on unhealthy clusters. - **Node health:** All nodes Ready, no MemoryPressure/DiskPressure/PIDPressure - **Pending CSRs:** Indicate node registration issues - **Crash-looping system pods:** Check kube-system, monitoring, ingress namespaces - **Metrics and DNS baseline:** Verify metrics-server and CoreDNS responding Record baselines for post-upgrade comparison. ## Step 11: Fargate Considerations Fargate pods upgrade when redeployed after CP upgrade. All Fargate pods must be restarted post-upgrade. Restart command is in Step 14 (mutation, requires approval). ## Step 12: Management Plane and IaC Ownership Detection Detect management method to route remediation correctly: | Detection | Management Plane | Mutation Routing | |-----------|-----------------|-----------------| | ACK CRD + Cluster CR | ACK | Patch ACK Cluster CR | | ACK CR with `kro.run/owned` | KRO over ACK | Patch kro instance | | Tags: `terraform:*` | Terraform | Update .tf, `terraform apply` | | Tags: `aws:cloudformation:*` | CloudFormation | Update template, stack update | | Tags: `aws:cdk:*` | CDK | Update construct, `cdk deploy` | | Tags: `eksctl.cluster.k8s.io/*` | eksctl | Update config, `eksctl upgrade` | | Labels: `argocd.argoproj.io/*` | ArgoCD | Update Git source, sync | | Labels: `kustomize.toolkit.fluxcd.io/*` | Flux | Update Git source, reconcile | | Tags: `pulumi:*` | Pulumi | Update program, `pulumi up` | | None found | unknown | Block mutations until confirmed | Route ALL remediation through the owning tool — never suggest direct AWS CLI when IaC is detected (causes drift). ## Step 13: Autoscaler Pause During Node Rotation During upgrades, autoscalers can interfere with rolling replacement. Check current Karpenter consolidation config and Cluster Autoscaler scale-down state. Recommend pausing both before node rotation and re-enabling after completion. Pause commands are in Step 14 (mutations, require operator approval). ## Step 14: Remediation Playbook (Operator Approval Required) > ⚠️ **ALL commands in this section are MUTATIONS.** The agent MUST NOT execute > these — present as a playbook for operator review. - **14.1** Helm stored manifest fix: `helm mapkubeapis` + `helm upgrade` - **14.2** Addon conflict resolution: first capture `DescribeAddon` output and `configurationValues`; use `--resolve-conflicts PRESERVE` to retain reviewed custom configuration, or `OVERWRITE` only after approving replacement with EKS defaults and recording rollback steps. `OVERWRITE` can discard custom configuration. - **14.3** Fargate pod restart: `kubectl rollout restart` across namespaces - **14.4** PDB temporary adjustment: `kubectl patch pdb` (revert after upgrade) - **14.5** Karpenter pause: `kubectl annotate nodepools --all "karpenter.sh/do-not-disrupt=true"` - **14.6** CA scale-down pause: patch CA config `scale-down-enabled=false` - **14.7** Node group upgrade: MNG via `update-nodegroup-version`, Karpenter via EC2NodeClass patch (drift), self-managed via launch template update ## Step 15: Post-Upgrade Functional Validation Present as validation checklist for operator: - DNS resolution (nslookup kubernetes.default) - Metrics server (kubectl top nodes/pods) - Pod scheduling (run test pod) - Load balancer health (target group check) - IRSA / Pod Identity (sts get-caller-identity from pod) - CoreDNS and kube-proxy pods running - Compare against Step 10 baseline (node count, no new CrashLoopBackOff) ## Step 16: Generate Upgrade Plan and Report **Execution Order:** 1. Pre-upgrade alignment (Karpenter/CA/webhooks) 2. Pause autoscalers 3. Control plane upgrade (15-40 min) 4. Wait for ACTIVE status 5. kube-proxy → vpc-cni → coredns → other managed addons 6. Self-managed addons 7. Node groups (one at a time, validate between) 8. Karpenter nodes (drift-based) 9. Self-managed nodes (launch template update) 10. Fargate pods (restart) 11. Re-enable autoscalers 12. Post-upgrade validation **Rollback Matrix:** | Component | Reversibility | Method | |-----------|--------------|--------| | Control plane | CONDITIONAL (7-day window) | `aws eks update-cluster-version --kubernetes-version <N-1>` | | Addons | FULL | Downgrade to previous version | | MNG | PARTIAL | Can halt; completed nodes stay | | Karpenter nodes | FULL | Revert EC2NodeClass | | Self-managed | FULL | Revert launch template | | Fargate | FULL | Redeploy previous config | **Rollback eligibility has two phases:** - **Pre-upgrade (advisory only):** confirm the planned upgrade is one minor, document the 7-day window and component rollback order, but do not claim the future cluster will be eligible. Rollback readiness insights do not exist until after an eligible upgrade completes. - **Post-upgrade (authoritative):** while the cluster is ACTIVE and still inside the 7-day window, run `aws eks list-insights --cluster-name <cluster> --filter '{"categories":["ROLLBACK_READINESS"]}'`, paginate, then `describe-insight` for each entry. `ERROR` blocks a normal rollback; `UNKNOWN` means EKS could not evaluate readiness and also blocks a normal rollback. Only `PASSING` insights support an eligible rollback. This assessment reports the result but never performs `update-cluster-version` or a forced rollback. ## Step 17: Report Format ``` ## EKS Upgrade Readiness Report **Cluster:** <name> (<region>) **Current Version:** <current> **Target Version:** <target> **Assessment Date:** <date> **Management Plane:** <detected> **Evidence Completeness:** <X>% (<performed>/<applicable>) **Overall Readiness:** READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE ### Pre-Upgrade Health Baseline - [PASS/FAIL] (confidence: HIGH) All nodes Ready - [PASS/FAIL] (confidence: HIGH) No pending CSRs - [PASS/FAIL] (confidence: HIGH) No crash-looping system pods - [PASS/FAIL] (confidence: HIGH) DNS resolution working - [PASS/FAIL] (confidence: HIGH) Metrics server responding ### Infrastructure Prerequisites - [PASS/FAIL] (confidence: HIGH) Subnet IP availability (mode: <type>) - [PASS/FAIL] (confidence: HIGH) EKS IAM role valid - [PASS/FAIL/N/A] (confidence: HIGH) KMS key access - [PASS/FAIL] (confidence: HIGH) EC2 vCPU quota headroom - [PASS/FAIL] (confidence: HIGH) EBS volume quota headroom ### EKS Upgrade Insights - [PASS/FAIL/UNKNOWN] (confidence: HIGH) <summary> ### Data Plane Inventory - Managed Node Groups: <count> (versions: <list>) - Self-Managed ASGs: <count> (versions: <list>) - Karpenter NodePools: <count> (version: <ver>) - Fargate Profiles: <count> - Total Nodes: <count> ### Blockers (must fix) 1. [FAIL] (confidence: HIGH) <description> — <remediation> ### Warnings (recommended) 1. [WARN] (confidence: MEDIUM) <description> — <recommendation> ### Passing Checks 1. [PASS] (confidence: HIGH) <description> ### Unknown / Not Assessed 1. [UNKNOWN] <gate> — <reason> ### Upgrade Plan <execution order from Step 16> ### Rollback Window - Rollback eligibility: ELIGIBLE / NOT ELIGIBLE / CHECK AFTER UPGRADE - Window: 7 days from CP upgrade completion - Note: Add-ons and node groups must be rolled back BEFORE CP ### Pre-Drain Risks - Bare pods (DRAIN-01): <count> - EmptyDir data loss (DRAIN-02): <count> - EBS AZ-pinning (DRAIN-04): <count> - Webhook deadlock (DRAIN-05): <assessment> - CoreDNS SPOF (DRAIN-06): <status> ### Estimated Timeline - Control plane: ~30 min - Addons: ~5 min each - Node groups: ~<X> min per group - Total: ~<Y> min ``` ### Machine-Readable Output When the operator asks for a structured result (CI/CD gating, scripted polling, dashboards), emit this JSON alongside — never instead of — the markdown report. Every gate in the markdown report must have a matching entry; the JSON is a serialization of the same evidence, not a summary. ```json { "cluster": "<name>", "region": "<region>", "assessmentTimestamp": "<ISO-8601>", "currentVersion": "<current>", "targetVersion": "<target>", "overallVerdict": "READY | READY_WITH_WARNINGS | NOT_READY | CANNOT_DETERMINE", "evidenceCompletenessPct": 0, "gates": [ { "id": "<check-id from required-check-registry.yaml, e.g. NODE-04, ADDON-02, INFRA-01>", "name": "<human-readable check name>", "status": "PASS | FAIL | WARN | UNKNOWN | N_A", "confidence": "HIGH | MEDIUM | LOW", "evidence": "<short evidence string, same as markdown bullet>", "remediation": "<remediation text, or null if PASS>", "checkedAt": "<ISO-8601>" } ], "rollback": { "eligible": true, "windowExpiresAt": "<ISO-8601 or null>" } } ``` `gates[].id` maps 1:1 to the IDs in `references/required-check-registry.yaml` (prefixes: `PF-` pre-flight, `INFRA-` infrastructure, `NODE-` node assessment, `ADDON-` addon assessment, `WKLD-` workload assessment, `KARP-` Karpenter, `DRAIN-` pre-drain safety, `ROLL-` rollback), so a CI pipeline can gate on specific check categories (e.g. fail only on `NODE-*` or `ADDON-*` FAILs, warn-only on others) instead of just the overall verdict. `overallVerdict` follows the same rules as the markdown report — it is never `READY` while any gate is `UNKNOWN`. ## References See `references/` directory for: - `safety-invariants.md` — Hard safety rules, knowledge hierarchy, operation classification - `required-check-registry.yaml` — All 60+ checks with IDs, categories, and severity - `pre-flight-checks.yaml` — Blocking vs warning checks, timeouts, soak periods, rollback conditions - `api-deprecations.md` — Full K8s API removal schedule by version - `addon-version-matrix.md` — EKS addon compatibility per version (static fallback) - `capacity-planning.md` — FDCR/ODCR and surge capacity guidance - `upgrade-troubleshooting.md` — Common failures, feature removals, and tools - `karpenter-checks.md` — Full 14-check Karpenter registry (KARP-01 to KARP-14) - `pre-drain-safety.md` — DRAIN-01 to DRAIN-06 detection and remediation - `al2-al2023-migration.md` — AL2→AL2023 migration assessment details - `data-plane-inventory.md` — MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.