[
  {
    "name": "basic_upgrade_readiness_check",
    "prompt": "Check if my EKS cluster prod-cluster in us-east-1 is ready to upgrade to version 1.31",
    "assertions": [
      "should call eks describe-cluster or equivalent API",
      "should check EKS upgrade insights (ListInsights) as primary signal",
      "should produce a readiness verdict (READY, NOT READY, READY WITH WARNINGS, or CANNOT DETERMINE)",
      "should mention addon compatibility check via DescribeAddonVersions",
      "should inventory all node populations (MNG, self-managed, Karpenter, Fargate)",
      "should check kubelet version skew"
    ]
  },
  {
    "name": "deprecated_api_investigation",
    "prompt": "What deprecated Kubernetes APIs would break if I upgrade my EKS cluster from 1.30 to 1.31?",
    "assertions": [
      "should reference the API deprecation schedule",
      "should identify specific APIs removed in 1.31",
      "should suggest replacement APIs",
      "should recommend checking EKS upgrade insights first",
      "should mention Helm stored manifest scanning for deployed revisions",
      "should mention third-party CRD deprecation checks"
    ]
  },
  {
    "name": "addon_compatibility_check",
    "prompt": "Are my EKS addons compatible with Kubernetes 1.31? My cluster is running vpc-cni v1.16.0, coredns v1.10.1, and kube-proxy v1.29.0",
    "assertions": [
      "should use DescribeAddonVersions API to check each addon against target",
      "should identify kube-proxy version mismatch (1.29 vs target 1.31)",
      "should recommend upgrade order (control plane first, then addons)",
      "should mention using describe-addon-versions for live verification",
      "should detect self-managed addons not in EKS addon list"
    ]
  },
  {
    "name": "capacity_planning_large_cluster",
    "prompt": "I need to upgrade a 50-node EKS cluster across 3 AZs with m6g.2xlarge instances. What capacity do I need?",
    "assertions": [
      "should calculate surge node requirements based on maxUnavailable",
      "should mention capacity reservation options (ODCR or FDCR)",
      "should discuss instance type availability",
      "should recommend an upgrade strategy appropriate for cluster size",
      "should mention pausing Karpenter consolidation or CA scale-down during rotation"
    ]
  },
  {
    "name": "pdb_validation",
    "prompt": "Will my Pod Disruption Budgets block the EKS node group upgrade? I have PDBs with maxUnavailable: 0 on critical services",
    "assertions": [
      "should identify maxUnavailable: 0 as a blocker for node drains",
      "should recommend adjusting PDBs before upgrade",
      "should suggest at least maxUnavailable: 1 during upgrade window",
      "should place PDB mutation in remediation section (not inline)",
      "should warn this is a read-only assessment, operator must approve changes"
    ]
  },
  {
    "name": "full_upgrade_plan",
    "prompt": "Create a complete upgrade plan for my EKS cluster from 1.29 to 1.30 including control plane, addons, and node groups",
    "assertions": [
      "should produce a step-by-step ordered plan",
      "should list pre-upgrade alignment steps (Karpenter/CA if needed)",
      "should include autoscaler pause before node rotation",
      "should specify addon upgrade order after control plane",
      "should include node group upgrade after addons",
      "should include post-upgrade validation smoke tests",
      "should include rollback gates between steps",
      "should include a pre-upgrade checklist"
    ]
  },
  {
    "name": "n2_version_skew_mixed_fleet",
    "prompt": "My EKS cluster is on 1.27 with kubelet versions ranging from 1.25 to 1.27 across nodes. Some are managed node groups, some are self-managed ASGs with custom AMIs. Can I upgrade to 1.28?",
    "assertions": [
      "should identify that 1.27 uses N-2 skew policy (kubelet >= 1.25 required)",
      "should confirm 1.25 kubelet is within tolerance for current 1.27 CP",
      "should warn that after upgrade to 1.28, N-3 policy applies but 1.25 nodes become N-3 (borderline)",
      "should inventory both managed and self-managed node populations",
      "should flag custom AMIs as requiring manual launch template update",
      "should NOT mark missing data as PASS"
    ]
  },
  {
    "name": "n3_version_skew_violation",
    "prompt": "My cluster is on 1.30 and I want to go to 1.31. I have some nodes still running kubelet 1.27. Is that okay?",
    "assertions": [
      "should identify version skew violation (1.27 kubelet with 1.31 CP = N-4, exceeds N-3)",
      "should mark this as FAIL/BLOCKER",
      "should recommend upgrading those nodes before control plane upgrade",
      "should distinguish between MNG and self-managed remediation paths",
      "should NOT produce READY verdict while this violation exists"
    ]
  },
  {
    "name": "missing_insights_unknown_verdict",
    "prompt": "Check upgrade readiness for my cluster staging-app. I don't have eks:ListInsights permission.",
    "assertions": [
      "should mark Insights gate as UNKNOWN (not PASS, not N/A)",
      "should continue with remaining checks despite missing Insights",
      "should produce CANNOT DETERMINE overall verdict (not READY)",
      "should list the missing permission and recommend granting it",
      "should still check infrastructure prerequisites, addons, node groups"
    ]
  },
  {
    "name": "custom_bootstrap_al2_to_al2023",
    "prompt": "We use AL2 with custom bootstrap scripts (kubelet-extra-args, custom yum packages, cgroup v1 assumptions). Target is 1.33. Assess readiness.",
    "assertions": [
      "should identify AL2 as CRITICAL blocker for 1.33+ (AMIs unavailable)",
      "should flag bootstrap.sh to nodeadm migration requirement",
      "should identify kubelet-extra-args needing conversion to NodeConfig YAML",
      "should flag yum to dnf change",
      "should mention cgroup v2 compatibility concerns",
      "should mention IMDSv2 default on AL2023",
      "should route remediation through launch template update"
    ]
  },
  {
    "name": "pagination_and_access_denied",
    "prompt": "I have a cluster with 200 node groups and 50 addons. Check if it's ready to upgrade. Note: I don't have permission to describe subnets.",
    "assertions": [
      "should paginate ListNodegroups results (not stop at first page)",
      "should paginate ListAddons results",
      "should mark subnet IP check as UNKNOWN due to AccessDenied",
      "should NOT mark subnet check as PASS or N/A",
      "should produce CANNOT DETERMINE overall verdict (UNKNOWN gate exists)",
      "should note which specific permission is missing"
    ]
  },
  {
    "name": "karpenter_drift_disabled",
    "prompt": "My cluster runs Karpenter v0.32 with Drift feature gate disabled and expireAfter set to Never on all NodePools. Planning to upgrade from 1.30 to 1.31.",
    "assertions": [
      "should flag Drift disabled as a problem (nodes won't auto-replace)",
      "should flag expireAfter Never as a problem (nodes stay on old AMI forever)",
      "should check Karpenter version compatibility with target (v0.32 may not support 1.31)",
      "should recommend upgrading Karpenter before control plane (pre-upgrade alignment)",
      "should place mutation commands in remediation section only",
      "should check disruption budgets"
    ]
  },
  {
    "name": "gitops_iac_detection",
    "prompt": "My cluster is managed by Terraform and ArgoCD deploys all workloads. Check upgrade readiness and tell me how to proceed.",
    "assertions": [
      "should detect Terraform ownership from cluster tags",
      "should detect ArgoCD from addon labels or deployments",
      "should route control plane upgrade through terraform apply (not direct aws CLI)",
      "should route workload changes through ArgoCD Git source",
      "should warn against direct aws eks update-cluster-version (causes drift)",
      "should still perform all readiness checks regardless of IaC tool"
    ]
  },
  {
    "name": "post_upgrade_validation",
    "prompt": "We just finished upgrading our cluster to 1.31. What should we validate?",
    "assertions": [
      "should include DNS resolution test",
      "should include metrics-server check",
      "should include pod scheduling test",
      "should include load balancer health check",
      "should include IRSA or Pod Identity validation",
      "should recommend comparing against pre-upgrade baseline",
      "should check for new CrashLoopBackOff pods"
    ]
  },
  {
    "name": "self_managed_addon_detection",
    "prompt": "My cluster shows only vpc-cni and kube-proxy in EKS managed addons, but I know we have AWS Load Balancer Controller, external-dns, and cert-manager installed via Helm. Check addon compatibility for 1.31.",
    "assertions": [
      "should use describe-addon-versions for managed addons",
      "should detect self-managed addons by scanning deployments/Helm releases",
      "should check AWS LBC, external-dns, cert-manager versions against target compatibility",
      "should recommend migrating self-managed to EKS managed where possible",
      "should flag any self-managed addon with unknown K8s version support"
    ]
  },
  {
    "name": "pre_upgrade_health_failure",
    "prompt": "Before upgrading, I ran kubectl get nodes and two nodes show NotReady. Also, coredns pods are in CrashLoopBackOff. Should I proceed with the upgrade?",
    "assertions": [
      "should FAIL the pre-upgrade health baseline gate",
      "should recommend investigating NotReady nodes before upgrade",
      "should flag CrashLoopBackOff coredns as critical (DNS will break)",
      "should produce NOT READY verdict",
      "should NOT recommend proceeding with the upgrade",
      "should suggest remediation for each issue before retrying assessment"
    ]
  },
  {
    "name": "cli_tooling_skew_check",
    "prompt": "My local kubectl is v1.26 and I'm upgrading a cluster from 1.30 to 1.31. I also use eksctl and an old Terraform AWS provider pinned to 4.x. Anything I should fix first?",
    "assertions": [
      "should flag kubectl v1.26 as outside the +/-1 minor skew window for a v1.31 target (WARN, not a blocker)",
      "should recommend upgrading kubectl before running further commands against the upgraded cluster",
      "should note eksctl version should be checked against target EKS version support",
      "should flag the Terraform AWS provider as likely too old to support target-version-specific attributes",
      "should NOT hardcode a specific 'minimum' version number as gospel — should recommend checking current release notes/changelog",
      "should NOT block the overall assessment on tooling skew alone (WARN-level only)"
    ]
  },
  {
    "name": "pod_identity_blue_green_migration",
    "prompt": "We use EKS Pod Identity (not IRSA) for most workloads and are considering a blue-green cluster migration. What identity work is needed?",
    "assertions": [
      "should explain that Pod Identity IAM role trust policies do NOT need to change (cluster-agnostic principal)",
      "should explain that Pod Identity associations are scoped per-cluster and must be explicitly recreated with create-pod-identity-association on the new cluster",
      "should contrast this with IRSA, where existing role trust policies must be updated to trust the new cluster's OIDC provider",
      "should recommend inventorying existing associations via list-pod-identity-associations before cutover",
      "should place the recreate-associations step in the Remediation Playbook (mutation, operator approval), not execute it automatically",
      "should check the eks-pod-identity-agent addon version via DescribeAddonVersions if Pod Identity Agent add-on is in use"
    ]
  },
  {
    "name": "machine_readable_output_request",
    "prompt": "Check upgrade readiness for cluster ci-cluster and give me the result as JSON so I can gate our pipeline on it.",
    "assertions": [
      "should produce the standard markdown report AND a structured JSON verdict block",
      "should include gate-level entries in the JSON, each with an id matching required-check-registry.yaml prefixes (PF-, INFRA-, NODE-, ADDON-, WKLD-, KARP-, DRAIN-, ROLL-)",
      "should include an overallVerdict field using the same READY/NOT_READY/READY_WITH_WARNINGS/CANNOT_DETERMINE values as the markdown report",
      "should NOT set overallVerdict to READY if any gate status is UNKNOWN",
      "should include confidence and evidence fields per gate, not just a boolean pass/fail"
    ]
  },
  {
    "name": "current_control_plane_upper_skew_violation",
    "prompt": "My EKS control plane is 1.31 and I plan to upgrade to 1.32. One self-managed node reports kubelet 1.32 already. Is that okay because 1.32 is my target?",
    "assertions": [
      "should identify kubelet 1.32 as newer than the current 1.31 control plane and therefore an existing version-skew violation",
      "should mark the current-state upper-bound gate as FAIL",
      "should explain that the planned target does not make an already-invalid current state acceptable",
      "should recommend correcting the node version before proceeding with the control-plane upgrade",
      "should produce NOT READY"
    ]
  },
  {
    "name": "stale_upgrade_insights_unknown",
    "prompt": "Assess upgrade readiness for a cluster targeting 1.32. ListInsights returned PASSING entries, but their lastRefreshTime is 36 hours old and the VPC CNI was updated yesterday.",
    "assertions": [
      "should mark the Upgrade Insights gate UNKNOWN because lastRefreshTime is older than the skill's 24-hour freshness threshold and predates a relevant addon change",
      "should NOT treat stale PASSING insights as proof that the upgrade is safe",
      "should recommend an operator-triggered insights refresh followed by a rerun, rather than invoking StartInsightsRefresh during this read-only assessment",
      "should produce CANNOT DETERMINE if no independent gate has failed"
    ]
  },
  {
    "name": "custom_coredns_corefile_and_self_managed_core_addons",
    "prompt": "Our cluster has no managed CoreDNS, kube-proxy, or VPC CNI addons in ListAddons. aws-node and kube-proxy are DaemonSets, CoreDNS is a Deployment, and the Corefile has custom rewrite and forward rules. Can we upgrade to 1.32?",
    "assertions": [
      "should identify aws-node, kube-proxy, and CoreDNS as self-managed or custom core addons rather than assuming ListAddons is complete",
      "should inspect their deployed image versions, arguments, and configuration",
      "should require a Corefile migration/compatibility check for the custom Corefile",
      "should mark compatibility UNKNOWN rather than PASS if target-version support cannot be established from the self-managed components' documentation",
      "should NOT recommend an unreviewed OVERWRITE of the custom CoreDNS configuration"
    ]
  },
  {
    "name": "vpc_cni_custom_networking_prefix_and_pod_sg_capacity",
    "prompt": "Our EKS 1.31 cluster is moving to 1.32. aws-node has ENABLE_PREFIX_DELEGATION=true, AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, and ENABLE_POD_ENI=true. We use ENIConfigs in two AZs and need a 6-node surge. Validate network capacity.",
    "assertions": [
      "should enumerate ENIConfig resources and evaluate the alternate pod subnets per AZ, not only the cluster subnets",
      "should calculate prefix delegation capacity using /28 prefixes and 16 IPv4 addresses per prefix",
      "should check instance-type-specific branch-ENI limits for Security Groups for Pods rather than using generic ENI limits",
      "should include the surge-node and additional pod-IP demand in the capacity calculation",
      "should mark the capacity gate UNKNOWN if required subnet, branch-ENI, or warm-prefix inputs are unavailable"
    ]
  },
  {
    "name": "managed_addon_configuration_conflict",
    "prompt": "DescribeAddon shows our VPC CNI has custom configurationValues, and the target add-on update reports ConfigurationConflict. Should I use PRESERVE or OVERWRITE?",
    "assertions": [
      "should state that this is an operator-approved mutation and must not be executed by the read-only assessment",
      "should require capturing the current DescribeAddon output and configurationValues before selecting a resolution",
      "should recommend PRESERVE first when the custom configuration is intentional and target-compatible",
      "should explain that OVERWRITE replaces conflicting customer configuration with EKS defaults and can discard custom behavior",
      "should require a reviewed backup and rollback plan before recommending OVERWRITE",
      "should not present OVERWRITE as the only or default resolution"
    ]
  }
]

[
  {
    "query": "Which skill would help me assess if my EKS cluster is ready to upgrade? Just name it; do not run it.",
    "should_trigger": true
  },
  {
    "query": "Is there a skill available for planning an EKS Kubernetes version upgrade? Answer yes or no with the skill name; do not execute it.",
    "should_trigger": true
  },
  {
    "query": "Name the skill that covers EKS deprecated API detection before upgrading. Do not run any checks.",
    "should_trigger": true
  },
  {
    "query": "Which skill checks EKS addon compatibility with a target Kubernetes version? Just name it.",
    "should_trigger": true
  },
  {
    "query": "Is there a skill for validating Pod Disruption Budgets before an EKS node group upgrade? Name only.",
    "should_trigger": true
  },
  {
    "query": "Name the skill that helps with capacity planning for EKS cluster upgrades. Do not execute.",
    "should_trigger": true
  },
  {
    "query": "Which skill detects AL2 to AL2023 migration issues before an EKS upgrade? Name only.",
    "should_trigger": true
  },
  {
    "query": "Is there a skill that checks kubelet version skew policy before upgrading EKS? Just name it.",
    "should_trigger": true
  },
  {
    "query": "Name the skill that helps prepare for EKS extended support ending and an automatic version upgrade. Do not run it.",
    "should_trigger": true
  },
  {
    "query": "Which skill helps identify which deprecated Kubernetes APIs broke after an EKS version upgrade? Just name it; do not run it.",
    "should_trigger": true
  },
  {
    "query": "Is there a skill for comparing in-place versus blue-green EKS upgrade strategies across multiple versions? Name only.",
    "should_trigger": true
  },
  {
    "query": "Name the skill that explains what to do about Fargate pods after an EKS control plane version upgrade. Do not execute.",
    "should_trigger": true
  },
  {
    "query": "Which skill helps troubleshoot RDS Aurora database performance issues? Name only.",
    "should_trigger": false
  },
  {
    "query": "Is there a skill for setting up new ECS Fargate services? Just name it.",
    "should_trigger": false
  },
  {
    "query": "Name the skill that handles S3 bucket security configuration audits. Do not run it.",
    "should_trigger": false
  },
  {
    "query": "Which skill investigates Lambda function timeout issues? Name only.",
    "should_trigger": false
  },
  {
    "query": "Is there a skill for VPC networking and subnet configuration? Answer with skill name only.",
    "should_trigger": false
  },
  {
    "query": "Name the skill that covers MSK Kafka consumer lag troubleshooting. Do not execute.",
    "should_trigger": false
  },
  {
    "query": "Which skill helps set up Karpenter from scratch on a new cluster? Name only.",
    "should_trigger": false
  },
  {
    "query": "Is there a skill for EKS pod OOM troubleshooting and memory right-sizing? Just name it.",
    "should_trigger": false
  },
  {
    "query": "Which skill helps troubleshoot EKS API server 429 throttling and high kubectl latency? Just name it.",
    "should_trigger": false
  },
  {
    "query": "Is there a skill for reviewing EKS RBAC, pod security standards, and network policies? Name only.",
    "should_trigger": false
  },
  {
    "query": "Name the skill that analyzes EKS cluster cost and right-sizing for underutilized nodes. Do not run it.",
    "should_trigger": false
  },
  {
    "query": "Which skill sets up CloudWatch monitoring and alerting for an EKS cluster? Just name it.",
    "should_trigger": false
  }
]

# EKS Addon Version Compatibility Matrix

> Static fallback reference. Last verified: 2026-08-17. `DescribeAddonVersions`
> for the current cluster and target Kubernetes version is the compatibility
> authority. Do not select a version, or mark a blocker, from this table alone.

This reference shows historically recommended addon version families per EKS
Kubernetes version. Use it only when live API data is unavailable, and report
that the relevant compatibility gate as `UNKNOWN` rather than `PASS`.

## Core Addons

| EKS Version | kube-proxy | vpc-cni | coredns | aws-ebs-csi-driver |
|-------------|------------|---------|---------|-------------------|
| 1.32 | v1.32.x | v1.19+ | v1.12+ | v1.38+ |
| 1.31 | v1.31.x | v1.18+ | v1.11+ | v1.35+ |
| 1.30 | v1.30.x | v1.18+ | v1.11+ | v1.33+ |
| 1.29 | v1.29.x | v1.16+ | v1.11+ | v1.28+ |
| 1.28 | v1.28.x | v1.15+ | v1.10+ | v1.25+ |

## Addon Upgrade Rules

### kube-proxy
- Treat matching the target control-plane minor as the normal **post-upgrade
  alignment recommendation**, not a hard-coded compatibility rule.
- Use `DescribeAddonVersions` to decide whether the installed version blocks
  the target upgrade. The live API result overrides this static reference.
- Upgrade promptly after the control plane when the live API identifies a
  target-compatible version. During the transition, respect the Kubernetes
  kube-proxy skew policy rather than assuming an exact minor match is the only
  valid state.

### vpc-cni (amazon-vpc-cni-k8s)
- Generally backward compatible across 2-3 minor versions
- New features (prefix delegation, Security Groups for Pods, network policy)
  may require specific minimum versions
- Safe to run a newer vpc-cni on an older control plane

### coredns
- Backward compatible across multiple minor versions
- New EKS versions may require minimum coredns for new features
- Check `coredns:coredns/corefile-migration` for Corefile compatibility

### aws-ebs-csi-driver
- Version constraints driven by CSI spec version and sidecar compatibility
- Newer versions add volume snapshot, resize, and topology awareness features
- Check for deprecation of `kubernetes.io/aws-ebs` in-tree provisioner

## How to Check Compatibility

```bash
# List available versions for an addon on target EKS version
aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.31 \
  --query 'addons[0].addonVersions[*].{version:addonVersion,default:compatibilities[0].defaultVersion}' \
  --output table

# Check current addon versions on a cluster
aws eks list-addons --cluster-name <cluster> --output text
for addon in $(aws eks list-addons --cluster-name <cluster> --output text --query 'addons[]'); do
  echo "$addon: $(aws eks describe-addon --cluster-name <cluster> --addon-name $addon --query 'addon.addonVersion' --output text)"
done
```

## Self-Managed and Custom-Configured Addons

First compare `aws eks list-addons` with the in-cluster inventory. A core
component that exists in `kube-system` but is absent from `ListAddons` is
self-managed (or replaced) and must not be assumed compatible from EKS managed
addon APIs alone.

```bash
# Core components that may be self-managed: images, args, and configuration
kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get deployment coredns -o json | \
  jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}'
kubectl -n kube-system get configmap coredns aws-node -o yaml
```

- **VPC CNI (`aws-node`)** — inspect image, environment variables, and the
  `aws-node` ConfigMap. A custom image or configuration needs its own
  compatibility validation; use the mode-aware capacity gate in SKILL.md Step 2.
- **CoreDNS** — inspect image and Corefile. A custom Corefile requires the
  CoreDNS migration check for the target release; do not overwrite it without
  a reviewed backup and migration plan.
- **kube-proxy** — inspect image, mode/configuration, and DaemonSet arguments.
  Use the upstream component documentation plus target-version testing when it
  is not EKS managed.
- For other self-managed addons, check the addon's release notes for Kubernetes
  version support: Karpenter, AWS Load Balancer Controller, ExternalDNS,
  cert-manager, ingress-nginx, Argo CD, and Flux.

## Upgrade Order

1. Pre-control-plane compatibility remediation where needed (Karpenter,
   Cluster Autoscaler, webhooks, and controllers that must span source/target).
2. Control plane.
3. kube-proxy, VPC CNI, CoreDNS, CSI drivers, other managed addons, then
   self-managed addons — use target-compatible versions returned by live APIs
   or each self-managed addon's support matrix.

### Configuration-Conflict Strategy (operator approval required)

Before updating a managed addon, save `DescribeAddon` output and any
`configurationValues`. Select a conflict mode deliberately:

- `PRESERVE` retains customer configuration values. Use it first when custom
  configuration is intentional and compatible with the target addon.
- `OVERWRITE` replaces conflicting customer configuration with EKS defaults.
  It can discard custom behavior; use it only after review, backup, and a
  documented rollback plan.

Neither option is part of the read-only assessment. They belong in the
operator-approved remediation playbook.

# AL2 → AL2023 Migration Assessment

This document covers the full AL2 to AL2023 migration checks for EKS upgrade
readiness assessments. If any node group or EC2NodeClass uses AL2, and the target
version is 1.33+, this is a **CRITICAL** blocker because EKS stopped releasing
AL2 AMIs after 1.32. For targets < 1.33, flag AL2 usage as a **WARNING**:
upstream Amazon Linux 2 reaches end of life on June 30, 2026.

## Detection Commands

```bash
# Check MNG AMI types
aws eks list-nodegroups --cluster-name <cluster> --query 'nodegroups[]' --output text | \
  xargs -I {} aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name {} \
  --query 'nodegroup.amiType' --output text

# Check Karpenter EC2NodeClass amiFamily
kubectl get ec2nodeclasses -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.amiFamily}{"\n"}{end}'
```

## Launch Template Analysis

```bash
# Get launch template user data for bootstrap method detection
aws ec2 describe-launch-template-versions --launch-template-id <lt-id> \
  --versions <version> --query 'LaunchTemplateVersions[0].LaunchTemplateData.UserData' | \
  base64 -d
```

## Bootstrap Differences

| Aspect | AL2 (bootstrap.sh) | AL2023 (nodeadm) |
|--------|--------------------|--------------------|
| Bootstrap script | `/etc/eks/bootstrap.sh` | `nodeadm` with YAML NodeConfig |
| Config format | CLI flags | `/etc/nodeadm/nodeconfig.yaml` |
| Cgroup driver | cgroup v1 | cgroup v2 (unified) |
| IMDS | v1 enabled by default | v2 only (IMDSv2) |
| Container runtime | containerd (since 1.24) | containerd |
| Kernel | 5.10 | 6.1 |

## Custom AMI Detection

If a node group uses a custom AMI (not EKS-optimized):

```bash
# Check if AMI is EKS-optimized or custom
AMI_ID=$(aws ec2 describe-launch-template-versions --launch-template-id <lt-id> \
  --versions <version> --query 'LaunchTemplateVersions[0].LaunchTemplateData.ImageId' --output text)
aws ec2 describe-images --image-ids $AMI_ID --query 'Images[0].Name' --output text
# EKS-optimized pattern: amazon-eks-node-<version>-*
# Custom: anything else
```

If custom AMI is detected:
- Flag that automated AMI updates won't work
- User must rebuild their AMI pipeline for AL2023 base
- Check if user data scripts are AL2-specific (yum vs dnf, systemd units, etc.)

## User Data / Bootstrap Compatibility

Check user data for AL2-specific patterns that break on AL2023:

- `--kubelet-extra-args` in bootstrap.sh → must convert to NodeConfig YAML
- `/etc/docker/daemon.json` → irrelevant on AL2023 (containerd only)
- `yum install` → must change to `dnf install`
- `/etc/sysctl.d/` settings → verify cgroup v2 compatibility
- IMDSv1 assumptions → AL2023 defaults to IMDSv2 only

### Example: Converting bootstrap.sh to nodeadm NodeConfig

**AL2 (bootstrap.sh):**
```bash
/etc/eks/bootstrap.sh my-cluster \
  --kubelet-extra-args '--max-pods=110 --node-labels=workload=compute'
```

**AL2023 (nodeadm NodeConfig):**
```yaml
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: my-cluster
    apiServerEndpoint: https://...
    certificateAuthority: ...
  kubelet:
    config:
      maxPods: 110
    flags:
      - --node-labels=workload=compute
```

## Cgroup v2 Compatibility

AL2023 uses cgroup v2 (unified hierarchy). Check for workloads that assume cgroup v1:

- **Java apps with `-XX:+UseContainerSupport`:** Works on both, but check JDK version.
  JDK 15+ has full cgroup v2 support. JDK 8u372+ and 11.0.16+ have partial support.
- **Monitoring agents reading `/sys/fs/cgroup/memory/`:** This is the v1 path; v2 uses
  `/sys/fs/cgroup/memory.max` etc. Agents that hardcode v1 paths will break.
- **Custom init containers manipulating cgroup files directly:** Any direct cgroup
  filesystem manipulation needs updating.
- **Resource monitoring tools:** cAdvisor < 0.43 has limited cgroup v2 support.

### Detection

```bash
# Find pods that mount cgroup filesystem directly
kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.hostPath.path | test("/sys/fs/cgroup")) | {
  ns: .metadata.namespace,
  name: .metadata.name,
  mounts: [.spec.volumes[] | select(.hostPath.path | test("/sys/fs/cgroup")) | .hostPath.path]
}'

# Check Java version in common images (requires exec access)
# kubectl exec <pod> -- java -version 2>&1 | head -1
```

## IMDSv2 Compatibility

AL2023 defaults to IMDSv2 only (requires token-based requests). Check for
workloads that use IMDSv1 (simple HTTP GET without token):

### Common IMDSv1 Patterns That Break

- AWS SDK versions before credential provider chain update (SDK v1 < 1.11.x)
- Custom scripts using `curl http://169.254.169.254/latest/meta-data/`
  without first obtaining a session token
- Legacy EC2 metadata queries without `X-aws-ec2-metadata-token` header

### Detection

```bash
# Check launch template IMDS settings
aws ec2 describe-launch-template-versions --launch-template-id <lt-id> \
  --versions <version> --query 'LaunchTemplateVersions[0].LaunchTemplateData.MetadataOptions'

# Check node group IMDS configuration
aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name <ng> \
  --query 'nodegroup.launchTemplate'
```

### Remediation

- Update AWS SDK to latest version (all modern SDKs support IMDSv2)
- Replace `curl http://169.254.169.254/...` with token-based access:
  ```bash
  TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
  curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/
  ```
- Use IRSA or Pod Identity instead of IMDS for AWS credentials (preferred)

## Migration Strategy Summary

1. **Inventory:** Identify all AL2 node groups and Karpenter EC2NodeClasses
2. **Bootstrap:** Convert all bootstrap.sh args to nodeadm NodeConfig YAML
3. **Custom AMIs:** Rebuild AMI pipelines with AL2023 base
4. **User Data:** Update package managers (yum→dnf), systemd units, scripts
5. **Cgroup v2:** Validate workloads with cgroup v2 compatibility
6. **IMDSv2:** Ensure all metadata access uses tokens or IRSA/Pod Identity
7. **Test:** Deploy AL2023 node group in parallel, migrate workloads gradually
8. **Cutover:** Drain AL2 nodes after validation

# Kubernetes API Deprecations and Removals by EKS Version

> Static fallback reference. Last verified: 2026-08-17. Confirm the target
> version against live EKS Upgrade Insights and the Kubernetes release notes;
> this table is not a compatibility authority.

This reference maps deprecated and removed Kubernetes APIs to EKS versions.
Use this to identify workloads that must be updated before upgrading.

## How to Read This Table

- **Deprecated**: API still works but emits warnings in audit logs
- **Removed**: API returns 404 — workloads using it will break

## Removals by Target Version

### EKS 1.32 (Kubernetes 1.32)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `flowcontrol.apiserver.k8s.io/v1beta3` | `flowcontrol.apiserver.k8s.io/v1` | FlowSchema, PriorityLevelConfiguration |

### EKS 1.29 (Kubernetes 1.29)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `flowcontrol.apiserver.k8s.io/v1beta2` | `flowcontrol.apiserver.k8s.io/v1` | FlowSchema, PriorityLevelConfiguration |

### EKS 1.27 (Kubernetes 1.27)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `storage.k8s.io/v1beta1` (CSIStorageCapacity) | `storage.k8s.io/v1` | CSIStorageCapacity |

### EKS 1.26 (Kubernetes 1.26)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `flowcontrol.apiserver.k8s.io/v1beta1` | `flowcontrol.apiserver.k8s.io/v1beta3` | FlowSchema, PriorityLevelConfiguration |
| `autoscaling/v2beta2` | `autoscaling/v2` | HorizontalPodAutoscaler |

### EKS 1.25 (Kubernetes 1.25)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `policy/v1beta1` | `policy/v1` | PodDisruptionBudget, PodSecurityPolicy (removed entirely) |
| `batch/v1beta1` | `batch/v1` | CronJob |
| `discovery.k8s.io/v1beta1` | `discovery.k8s.io/v1` | EndpointSlice |
| `events.k8s.io/v1beta1` | `events.k8s.io/v1` | Event |
| `autoscaling/v2beta1` | `autoscaling/v2` | HorizontalPodAutoscaler |
| `node.k8s.io/v1beta1` | `node.k8s.io/v1` | RuntimeClass |

### EKS 1.22 (Kubernetes 1.22)

| API | Replacement | Resources Affected |
|-----|-------------|-------------------|
| `networking.k8s.io/v1beta1` | `networking.k8s.io/v1` | Ingress, IngressClass |
| `rbac.authorization.k8s.io/v1beta1` | `rbac.authorization.k8s.io/v1` | ClusterRole, ClusterRoleBinding, Role, RoleBinding |
| `admissionregistration.k8s.io/v1beta1` | `admissionregistration.k8s.io/v1` | MutatingWebhookConfiguration, ValidatingWebhookConfiguration |
| `apiextensions.k8s.io/v1beta1` | `apiextensions.k8s.io/v1` | CustomResourceDefinition |

## Detection Methods

### Via EKS Upgrade Insights (recommended)
```
aws eks list-insights --cluster-name <cluster> \
  --filter '{"categories":["UPGRADE_READINESS"]}'
```

### Via Kubernetes audit logs (if enabled)
Search for `k8s.io/deprecated=true` annotation in API server audit logs:
```
fields @timestamp, objectRef.resource, objectRef.apiVersion, user.username
| filter annotations.`k8s.io/deprecated` = "true"
| stats count() by objectRef.apiVersion, objectRef.resource
```

### Via kubectl (requires cluster access)
```bash
# Check for deprecated APIs using kubectl
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
```

## Remediation Pattern

For each deprecated API usage:
1. Identify the controller/workload using it (from insight or audit log)
2. Update the manifest `apiVersion` field to the replacement
3. Check if the resource spec changed between versions (some fields moved)
4. Apply the updated manifest
5. Verify the workload is healthy
6. Confirm no more deprecation warnings in audit logs

# EKS Upgrade Capacity Planning

This reference covers capacity planning for EKS node group upgrades,
including surge node calculations and Capacity Reservation strategies.

## Surge Node Calculation

This follows the actual Amazon EKS managed node group update algorithm
(see [Understand each phase of node updates](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html)),
not a simplified per-AZ percentage model. There are four phases:

1. **Setup** — a new launch template version is created and applied to the
   ASG. `updateConfig` (`maxUnavailable` or `maxUnavailablePercentage`, capped
   at 100 nodes) determines how many nodes can be replaced in parallel.
2. **Scale up** — the ASG's max and desired size are incremented **before**
   any old node is touched, so capacity never drops during the upgrade
   (default strategy). New nodes land in the same AZs as the nodes they
   replace, using EC2 Auto Scaling Availability Zone Rebalancing.
3. **Upgrade** — old nodes are cordoned once a replacement is `Ready`,
   drained (15-minute timeout, `PodEvictionFailure` without `--force`), then
   terminated after a 60-second post-eviction wait. This repeats in batches
   of `maxUnavailable` until every node runs the new launch template version.
4. **Scale down** — the ASG max/desired size is decremented back to the
   pre-upgrade value once the rollout completes (skipped if Cluster
   Autoscaler is actively scaling the group at that moment).

### Formula

The scale-up increment is **not** a per-AZ percentage of existing nodes — it
is the larger of two values, applied once to the whole ASG:

```
maxUnavailable_count = min(100, maxUnavailable OR ceil(desired_size * maxUnavailablePercentage / 100))
Total surge at peak  = max(2 * number_of_azs, maxUnavailable_count)
```

Because EKS guarantees at least one new node per AZ where old nodes exist
(and up to two per AZ to satisfy AZ Rebalancing), a node group spread across
many AZs can surge by more than `maxUnavailable` even when `maxUnavailable`
is small — plan capacity for `2 * numAZs`, not just `maxUnavailable`.

### Examples (3 AZs)

| Desired Size | updateConfig | maxUnavailable_count | Surge at Peak (max of 2×AZ, maxUnavailable) |
|--------------|-------------|----------------------|----------------------------------------------|
| 15 | maxUnavailable: 1 | 1 | 6 (2×3 AZ dominates) |
| 15 | maxUnavailable: 10 | 10 | 10 (maxUnavailable dominates) |
| 30 | maxUnavailablePercentage: 20% | 6 | 6 (maxUnavailable dominates) |
| 150 | maxUnavailablePercentage: 33% | 50 | 50 (maxUnavailable dominates) |

Use `--force` awareness when planning timelines: if `PodEvictionFailure`
occurs (aggressive PDBs, taint-tolerant deployments), the batch stalls at
the 15-minute drain timeout until an operator intervenes — factor this into
maintenance-window sizing, don't assume `force` is used automatically (it
requires an explicit operator-approved `update-nodegroup-version --force`).

## Capacity Reservation Strategies

For large clusters or instance types with limited availability,
use EC2 Capacity Reservations to guarantee surge capacity.

### On-Demand Capacity Reservations (ODCR)

- Immediate availability, billed whether used or not
- Best for: short upgrade windows where you want guaranteed capacity
- Create just before upgrade, cancel immediately after

### Flexible Duration Capacity Reservations (FDCR)

- Scheduled future capacity, minimum 24-hour duration
- Best for: planned upgrades with known schedules
- Create days in advance, auto-activate at scheduled time

### Targeting Strategies

| Strategy | How It Works | When to Use |
|----------|-------------|-------------|
| Open match | Any instance in the AZ consumes slots | Single workload in the AZ |
| Targeted + Resource Group | Only ASG instances consume slots | Multiple workloads in same AZ |

### Resource Group + ASG Targeting (Recommended)

```bash
# 1. Create resource group
aws resource-groups create-group \
  --name eks-upgrade-capacity \
  --configuration \
    '{"Type":"AWS::EC2::CapacityReservationPool"}' \
    '{"Type":"AWS::ResourceGroups::Generic","Parameters":[{"Name":"allowed-resource-types","Values":["AWS::EC2::CapacityReservation"]}]}'

# 2. Add CRs to group
aws resource-groups group-resources \
  --group eks-upgrade-capacity \
  --resource-arns arn:aws:ec2:<region>:<account>:capacity-reservation/<cr-id>

# 3. Configure ASG to target the group
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <asg-name> \
  --capacity-reservation-specification \
    '{"CapacityReservationTarget":{"CapacityReservationResourceGroupArn":"arn:aws:resource-groups:<region>:<account>:group/eks-upgrade-capacity"}}'
```

### Important Notes

- FDCRs start as "targeted" — must switch to "open" after activation OR use resource group
- Cannot modify instance eligibility while instances are consuming the reservation
- If using "open" match, other workloads with the same instance type in the AZ may consume slots
- Calculate reservation size as: existing nodes + surge nodes (all get replaced during rolling update)

## When NOT to Use Capacity Reservations

- Instance types with broad availability (t3, m5, m6i in major regions)
- Small clusters (< 10 nodes) where InsufficientCapacity is unlikely
- Clusters using diversified instance types (Karpenter with multiple types)
- Spot-based node groups (CRs are for On-Demand only)

## Troubleshooting Capacity Issues During Upgrade

| Symptom | Cause | Resolution |
|---------|-------|-----------|
| `InsufficientInstanceCapacity` during upgrade | AZ lacks capacity for instance type | Use FDCR or switch to open CR match |
| CR shows "Available: 0" but no instances running | Other workloads consumed open CR slots | Switch to targeted + resource group |
| ASG not consuming targeted CR | Launch template missing CR specification | Use resource group targeting on ASG instead |
| FDCR not activating | Still in "Scheduled" state | Wait until start time; cannot modify while scheduled |

# Data Plane Inventory

A complete picture of the data plane is required for upgrade readiness assessment.
Missing any node population means the upgrade plan has blind spots.

## Managed Node Groups (MNG)

```bash
# List all node groups (paginate!)
NODEGROUPS=$(aws eks list-nodegroups --cluster-name <cluster> --query 'nodegroups[]' --output text)

# For each node group, get full details
for ng in $NODEGROUPS; do
  aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name "$ng" \
    --query '{name:nodegroup.nodegroupName, version:nodegroup.version,
      amiType:nodegroup.amiType, instanceTypes:nodegroup.instanceTypes,
      desiredSize:nodegroup.scalingConfig.desiredSize,
      maxSize:nodegroup.scalingConfig.maxSize,
      updateConfig:nodegroup.updateConfig,
      launchTemplate:nodegroup.launchTemplate,
      health:nodegroup.health.issues}'
done
```

For each MNG, record:
- Current K8s version vs target (version skew check)
- AMI type (AL2, AL2023, BOTTLEROCKET, WINDOWS_CORE, CUSTOM)
- Update strategy (`maxUnavailable` or `maxUnavailablePercentage`)
- Launch template ID and version (for custom AMI detection)
- Health issues (any existing problems block upgrade)

### MNG Update Algorithm

When you initiate a node group update, EKS:
1. Creates new nodes with the updated config (up to `maxUnavailable` count)
2. Cordons old nodes
3. Drains old nodes (respects PDBs — will wait/retry for up to 15 min)
4. If drain fails after timeout, ForceEviction applies (pods deleted)
5. Old nodes are terminated
6. Repeats until all nodes are updated

Understanding this is critical for capacity planning — at peak, you have
`existing_nodes + maxUnavailable` nodes running simultaneously.

## Self-Managed Node Groups (ASGs)

Self-managed nodes are EC2 instances in ASGs that joined the cluster via
bootstrap script but aren't tracked by EKS node group APIs.

### Detection

```bash
# Find ASGs with EKS cluster tag
aws autoscaling describe-auto-scaling-groups \
  --query "AutoScalingGroups[?Tags[?Key=='kubernetes.io/cluster/<cluster>' || Key=='eks:cluster-name']].[AutoScalingGroupName,LaunchTemplate.LaunchTemplateId,LaunchTemplate.Version,DesiredCapacity]" \
  --output table

# Get launch template details for AMI ID
aws ec2 describe-launch-template-versions --launch-template-id <lt-id> \
  --versions <version> --query 'LaunchTemplateVersions[0].LaunchTemplateData.ImageId'

# Resolve AMI to K8s version
aws ec2 describe-images --image-ids <ami-id> --query 'Images[0].[Name,Description]'
```

### Kubelet Version from Self-Managed Nodes

```bash
# Extract kubelet version from node labels (if kubectl available)
kubectl get nodes -l eks.amazonaws.com/nodegroup!=<any-mng> \
  -o jsonpath='{range .items[*]}{.metadata.name}: {.status.nodeInfo.kubeletVersion}{"\n"}{end}'
```

For self-managed nodes:
- Check if AMI is custom or EKS-optimized (from AMI name pattern)
- Identify bootstrap method (see `al2-al2023-migration.md`)
- Note: self-managed nodes require manual launch template updates

## Karpenter Managed Nodes

```bash
# Check Karpenter version
kubectl get deploy karpenter -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

# List NodePools and their config
kubectl get nodepools -o json | jq '.items[] | {
  name: .metadata.name,
  expireAfter: .spec.disruption.expireAfter,
  consolidateAfter: .spec.disruption.consolidateAfter,
  budgets: .spec.disruption.budgets
}'

# List EC2NodeClasses (AMI config)
kubectl get ec2nodeclasses -o json | jq '.items[] | {
  name: .metadata.name,
  amiFamily: .spec.amiFamily,
  amiSelectorTerms: .spec.amiSelectorTerms
}'

# Check feature gates (Drift)
kubectl get deploy karpenter -n kube-system -o json | jq '.spec.template.spec.containers[0].env[] | select(.name=="FEATURE_GATES")'
```

See `karpenter-checks.md` for the full 14-check registry (KARP-01 to KARP-14).

## EKS Auto Mode

If `cluster.computeConfig.enabled` is `true`:
- Data plane upgrades happen automatically after control plane upgrade
- Monitor with: `aws eks describe-cluster --name <cluster> --query 'cluster.computeConfig'`
- Verify PDBs won't block automatic rotation

### Detection

```bash
aws eks describe-cluster --name <cluster> --query 'cluster.computeConfig'
```

If Auto Mode is enabled, the node rotation happens without operator action
after the control plane upgrade completes. The key checks become:
- PDBs must allow disruption
- Workloads must tolerate rolling replacement
- No bare pods or emptyDir-dependent workloads on Auto Mode nodes

## Fargate Profiles

```bash
# List Fargate profiles
aws eks list-fargate-profiles --cluster-name <cluster>

# Describe each profile
aws eks describe-fargate-profile --cluster-name <cluster> --fargate-profile-name <name> \
  --query '{name:fargateProfile.fargateProfileName, selectors:fargateProfile.selectors, subnets:fargateProfile.subnets}'
```

Fargate pods:
- Are automatically upgraded when redeployed after control plane upgrade
- Support the same version skew as managed node groups (N-3 for 1.28+)
- Require explicit restart after CP upgrade (see Remediation Playbook)

## Kubelet Version Inventory

Regardless of node management method, confirm actual kubelet versions running:

```bash
# Full kubelet version map (requires kubectl)
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}: kubelet={.status.nodeInfo.kubeletVersion}, os={.status.nodeInfo.osImage}, arch={.metadata.labels.kubernetes\.io/arch}{"\n"}{end}'

# Summary: versions that violate skew policy
kubectl get nodes -o json | jq --arg target "<target-version>" '.items[] | select(.status.nodeInfo.kubeletVersion | test("v1\\.(\\d+)") | not) | {name: .metadata.name, version: .status.nodeInfo.kubeletVersion}'
```

### Version Skew Check

Evaluate the current control-plane version and target version separately:

1. **Current-state upper bound:** kubelet must never be newer than the current
   control plane (`kubelet_minor <= current_control_plane_minor`). A node at
   1.32 with a current control plane at 1.31 is already invalid, even if 1.32
   is the intended target.
2. **Target lower bound:** for target version 1.X, kubelet must be >= 1.(X-3)
   when X >= 28 (N-3), or >= 1.(X-2) when X < 28 (N-2).

Any node violating either predicate is a **FAIL** — correct it before the
control-plane upgrade proceeds.

## Inventory Summary Template

After running all discovery commands, produce a summary:

```
### Data Plane Inventory
- Managed Node Groups: <count> (versions: <list>)
  - <ng-name>: <version>, <ami-type>, <instance-types>, <desired>/<max> nodes
- Self-Managed ASGs: <count> (versions: <list>)
  - <asg-name>: <ami-id>, <instance-type>, <count> nodes
- Karpenter NodePools: <count> (Karpenter version: <ver>)
  - <pool-name>: <amiFamily>, expireAfter=<val>, budgets=<val>
- EKS Auto Mode: <enabled|disabled>
- Fargate Profiles: <count>
  - <profile-name>: selectors=<namespaces>
- Total Nodes: <count>
- Kubelet Versions: <distinct versions found>
- Version Skew Violations: <count> nodes outside allowed window
```

# Karpenter Upgrade Readiness Checks (KARP-01 to KARP-14)

This document contains the full Karpenter check registry and detection commands
for EKS upgrade readiness assessments.

## Discovery Commands

```bash
# Check Karpenter version
kubectl get deploy karpenter -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

# List NodePools and their config
kubectl get nodepools -o json | jq '.items[] | {
  name: .metadata.name,
  expireAfter: .spec.disruption.expireAfter,
  consolidateAfter: .spec.disruption.consolidateAfter,
  budgets: .spec.disruption.budgets
}'

# List EC2NodeClasses (AMI config)
kubectl get ec2nodeclasses -o json | jq '.items[] | {
  name: .metadata.name,
  amiFamily: .spec.amiFamily,
  amiSelectorTerms: .spec.amiSelectorTerms
}'

# Check feature gates (Drift)
kubectl get deploy karpenter -n kube-system -o json | jq '.spec.template.spec.containers[0].env[] | select(.name=="FEATURE_GATES")'
```

## Check Registry

| ID | Check | Pass Criteria | Severity |
|----|-------|---------------|----------|
| KARP-01 | Version compatibility | Karpenter release supports target K8s version | Critical |
| KARP-02 | Drift enabled | Feature gate on (default since v0.33) | High |
| KARP-03 | expireAfter set | Not `Never` on any NodePool | High |
| KARP-04 | Disruption budgets | At least 1 node can be disrupted (not `nodes: "0"`) | High |
| KARP-05 | AMI not pinned | amiSelectorTerms not pinned to specific AMI ID | High |
| KARP-06 | AMI family valid | amiFamily not AL2 when target >= 1.33 | Critical |
| KARP-07 | Not self-hosted | Controller pods NOT on Karpenter-managed nodes | Critical |
| KARP-08 | NodeClassRef valid | Every NodePool's nodeClassRef points to existing EC2NodeClass | High |
| KARP-09 | Consolidation interference | Short consolidateAfter + active drift = race condition | Medium |
| KARP-10 | Schedule conflict | Disruption budget schedule doesn't block upgrade window | Medium |
| KARP-11 | ExpireAfter timing | NodeClaim expiry won't trigger during upgrade window | Medium |
| KARP-12 | Drift throughput | Estimated time to replace all nodes vs acceptable window | Low |
| KARP-13 | Controller health | All replicas ready, no crash-looping | Critical |
| KARP-14 | v1alpha5 orphans | No leftover Provisioner CRD from incomplete migration | Medium |

## KARP-07: Self-Hosted Detection (Critical)

If Karpenter runs on nodes it manages, it may evict itself during drift-based
replacement, halting all further node rotation.

```bash
# Check if karpenter pods run on karpenter-managed nodes
KARP_NODES=$(kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}')
for node in $KARP_NODES; do
  kubectl get node $node -o jsonpath='{.metadata.labels}' | grep -q "karpenter.sh/nodepool" && echo "FAIL: Karpenter self-hosted on $node"
done
```

## KARP-08: Dangling NodeClassRef Detection

Dangling nodeClassRef prevents Karpenter from launching replacement nodes after
drift fires:

```bash
# Verify all NodePool nodeClassRefs resolve
NODECLASSES=$(kubectl get ec2nodeclasses -o jsonpath='{.items[*].metadata.name}')
kubectl get nodepools -o json | jq --arg ncs "$NODECLASSES" '.items[] | select(.spec.template.spec.nodeClassRef.name as $ref | ($ncs | split(" ") | index($ref)) == null) | {pool: .metadata.name, danglingRef: .spec.template.spec.nodeClassRef.name}'
```

## KARP-09: Consolidation Interference

Short `consolidateAfter` combined with active drift creates a race condition
where consolidation may terminate nodes that drift is trying to replace.

Check:
```bash
kubectl get nodepools -o json | jq '.items[] | select(.spec.disruption.consolidateAfter != null and .spec.disruption.consolidateAfter != "Never") | {name: .metadata.name, consolidateAfter: .spec.disruption.consolidateAfter}'
```

If consolidateAfter is < 30m and drift is active, flag as WARN.

## KARP-10: Schedule Conflict

Disruption budget schedules that overlap with the planned upgrade window can
block Karpenter from replacing nodes:

```bash
kubectl get nodepools -o json | jq '.items[] | select(.spec.disruption.budgets[]?.schedule != null) | {name: .metadata.name, budgets: .spec.disruption.budgets}'
```

## KARP-11: ExpireAfter Timing

If NodeClaims are close to their expiry time, they may trigger replacement
during the upgrade window causing unexpected churn:

```bash
kubectl get nodeclaims -o json | jq '.items[] | {name: .metadata.name, created: .metadata.creationTimestamp, expireAfter: .spec.expireAfter}'
```

## KARP-13: Controller Health

```bash
kubectl get deploy karpenter -n kube-system -o json | jq '{
  replicas: .spec.replicas,
  ready: .status.readyReplicas,
  available: .status.availableReplicas,
  conditions: .status.conditions
}'

# Check for crash loops
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter -o json | jq '.items[] | {
  name: .metadata.name,
  ready: .status.containerStatuses[0].ready,
  restarts: .status.containerStatuses[0].restartCount,
  state: .status.containerStatuses[0].state
}'
```

## KARP-14: v1alpha5 Orphan Detection

After migration from Karpenter < 0.33, leftover Provisioner CRDs may remain:

```bash
# Check for old Provisioner CRD
kubectl get crd provisioners.karpenter.sh 2>/dev/null && echo "WARN: Legacy Provisioner CRD still exists"

# Check for leftover Provisioner resources
kubectl get provisioners 2>/dev/null && echo "WARN: Legacy Provisioner resources found"

# Check for old AWSNodeTemplate CRD
kubectl get crd awsnodetemplates.karpenter.k8s.aws 2>/dev/null && echo "WARN: Legacy AWSNodeTemplate CRD still exists"
```

# Pre-Drain Safety Checks (DRAIN-01 to DRAIN-06)

Beyond PDBs, node drains can fail or cause damage in 6 additional ways.
Check these BEFORE including drain in the upgrade plan.

## Check Summary

| ID | Check | Risk | Severity |
|----|-------|------|----------|
| DRAIN-01 | Bare pods (no ownerReferences) | Not rescheduled after eviction; kubectl drain refuses without --force | High |
| DRAIN-02 | Pods with emptyDir volumes | Data lost on drain (--delete-emptydir-data required) | Medium |
| DRAIN-03 | Custom finalizers on pods | Can hang eviction to timeout if finalizer controller is unhealthy | Medium |
| DRAIN-04 | EBS AZ-pinned PVCs | Cross-AZ reschedule strands the volume (Pending forever) | High |
| DRAIN-05 | Fail-closed webhooks on drain-target nodes | Evicting webhook pods deadlocks all further eviction cluster-wide | Critical |
| DRAIN-06 | CoreDNS SPOF | All CoreDNS replicas on same drain batch = cluster-wide DNS outage | Critical |

## DRAIN-01: Bare Pods

Pods without ownerReferences are not managed by a controller and will NOT be
rescheduled after eviction. `kubectl drain` refuses to evict them without
`--force`, which can stall automated node replacement.

```bash
kubectl get pods -A -o json | jq '.items[] | select(.metadata.ownerReferences == null) | {
  ns: .metadata.namespace,
  name: .metadata.name,
  node: .spec.nodeName
}'
```

**Remediation:** Wrap bare pods in a Deployment/Job, or acknowledge data loss
and allow `--force` drain.

## DRAIN-02: Pods with emptyDir Volumes

emptyDir volumes are ephemeral — data is lost when the pod is evicted.
`kubectl drain` requires `--delete-emptydir-data` flag to proceed.

```bash
kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.emptyDir != null) | {
  ns: .metadata.namespace,
  name: .metadata.name,
  node: .spec.nodeName,
  emptyDirVolumes: [.spec.volumes[] | select(.emptyDir != null) | .name]
}'
```

**Remediation:** Ensure any important data in emptyDir is either ephemeral
(caches, temp files) or backed by external storage. Flag pods using emptyDir
for actual state (e.g., Prometheus WAL without persistent storage).

## DRAIN-03: Custom Finalizers on Pods

Pods with custom finalizers can hang eviction indefinitely if the finalizer
controller is unhealthy or slow:

```bash
kubectl get pods -A -o json | jq '.items[] | select(.metadata.finalizers != null and (.metadata.finalizers | length > 0)) | {
  ns: .metadata.namespace,
  name: .metadata.name,
  finalizers: .metadata.finalizers
}'
```

**Remediation:** Verify finalizer controllers are healthy. Consider removing
non-critical finalizers before upgrade or setting an eviction timeout.

## DRAIN-04: EBS AZ-Pinned PVCs

EBS volumes are AZ-bound. If a pod is drained to a node in a different AZ,
the PVC cannot be attached — the pod stays Pending forever.

```bash
# Find EBS PVCs with zone affinity
kubectl get pv -o json | jq '.items[] | select(.spec.csi.driver == "ebs.csi.aws.com") | {
  name: .metadata.name,
  zone: .spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[] | select(.key == "topology.ebs.csi.aws.com/zone") | .values[0],
  claim: .spec.claimRef.namespace + "/" + .spec.claimRef.name
}'

# Cross-reference with node group AZ distribution
kubectl get nodes -o json | jq '.items[] | {
  name: .metadata.name,
  zone: .metadata.labels["topology.kubernetes.io/zone"]
}'
```

**Remediation:** Ensure node groups span the same AZs as EBS volumes. For
StatefulSets with EBS, use `topologySpreadConstraints` or node affinity to
keep pods in the same AZ as their volumes. During upgrade, ensure surge
nodes are launched in every AZ that has EBS volumes.

## DRAIN-05: Fail-Closed Webhooks on Drain-Target Nodes

If ALL endpoint pods for a `failurePolicy: Fail` webhook are on nodes being
drained simultaneously, evicting those pods causes a cluster-wide deadlock —
no further evictions can proceed because the webhook rejects all API calls.

```bash
# Find fail-closed webhooks and their backing endpoints
kubectl get validatingwebhookconfigurations -o json | jq '.items[].webhooks[] | select(.failurePolicy == "Fail") | {name: .name, service: .clientConfig.service}'
kubectl get mutatingwebhookconfigurations -o json | jq '.items[].webhooks[] | select(.failurePolicy == "Fail") | {name: .name, service: .clientConfig.service}'

# For each webhook service, check endpoint pod distribution
# Example for a webhook service named "webhook-svc" in namespace "system":
kubectl get endpoints webhook-svc -n system -o json | jq '.subsets[].addresses[].nodeName'

# Cross-reference with nodes scheduled for drain
```

**Remediation:**
- Ensure webhook pods have anti-affinity to spread across nodes/AZs
- Set PDBs on webhook deployments to prevent all replicas from draining simultaneously
- Consider `failurePolicy: Ignore` for non-critical webhooks during upgrade
- Drain webhook-hosting nodes LAST

## DRAIN-06: CoreDNS Single Point of Failure

If all CoreDNS replicas end up on nodes in the same drain batch, the entire
cluster loses DNS resolution — new pods can't resolve services, health checks
fail, and cascading failures follow.

```bash
# Check CoreDNS pod distribution
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# Detailed node placement
kubectl get pods -n kube-system -l k8s-app=kube-dns -o json | jq '.items[] | {
  name: .metadata.name,
  node: .spec.nodeName,
  ready: .status.conditions[] | select(.type == "Ready") | .status
}'

# Check if CoreDNS has topology spread or anti-affinity
kubectl get deploy coredns -n kube-system -o json | jq '{
  replicas: .spec.replicas,
  topologySpread: .spec.template.spec.topologySpreadConstraints,
  affinity: .spec.template.spec.affinity
}'
```

**Remediation:**
- Ensure CoreDNS has at least 2 replicas (ideally 3+)
- Add `topologySpreadConstraints` to spread across AZs
- Set a PDB with `minAvailable: 2` (or appropriate for replica count)
- During node rotation, verify CoreDNS pods are rescheduled FIRST before
  proceeding with further drains
- Consider running CoreDNS on dedicated system nodes or Fargate

# EKS Upgrade Readiness — Pre-Flight Checks
# Version: 2.0.0
# Purpose: Defines blocking vs warning checks with timeouts and soak periods.
#          Blocking checks halt the assessment. Warning checks flag but continue.

blocking_checks:
  - id: cluster_active
    description: "Cluster status MUST be ACTIVE"
    api: "eks:DescribeCluster → status"
    failure_action: "STOP — do not proceed with any assessment"

  - id: cluster_not_mid_upgrade
    description: "Cluster MUST NOT have a pending platform update in progress"
    api: "eks:ListUpdates (filter status=InProgress)"
    failure_action: "STOP — wait for current update to complete"

  - id: version_hop_valid
    description: "Target version MUST be exactly current + 1 minor version"
    api: "eks:DescribeCluster → version comparison"
    failure_action: "STOP — EKS requires sequential minor version upgrades"

  - id: version_skew_gate
    description: "No kubelet is newer than the current control plane, and all kubelets meet the target lower bound (N-3 for >=1.28, N-2 for <1.28)"
    api: "eks:DescribeCluster, eks:DescribeNodegroup (version), kubectl get nodes"
    failure_action: "STOP — upgrade or correct node groups first to meet both skew predicates"

  - id: ami_hard_gate
    description: "No AL2 AMI usage when target >= 1.33"
    api: "eks:DescribeNodegroup → amiType, kubectl get ec2nodeclasses"
    failure_action: "STOP — migrate to AL2023 or Bottlerocket before upgrading"

  - id: subnet_minimum_ips
    description: "Cluster subnets MUST have at least 5 available IPs"
    api: "ec2:DescribeSubnets"
    failure_action: "STOP — insufficient IP capacity for control plane upgrade"

  - id: iam_cluster_role_valid
    description: "Cluster IAM role MUST exist with correct trust policy"
    api: "iam:GetRole for cluster role ARN"
    failure_action: "STOP — cluster role missing or insufficient permissions"

  - id: kms_key_valid
    description: "If encryption enabled, KMS key MUST exist and be enabled"
    api: "eks:DescribeCluster → encryptionConfig"
    condition: "Only if cluster has encryptionConfig"
    failure_action: "STOP — KMS key invalid, upgrade will fail silently"

  - id: karpenter_controller_healthy
    description: "Karpenter controller deployment healthy (if Karpenter detected)"
    api: "kubectl get deploy karpenter -n kube-system"
    condition: "Only if Karpenter detected"
    failure_action: "STOP — unhealthy Karpenter cannot replace nodes post-upgrade"

  - id: karpenter_not_self_hosted
    description: "Karpenter pods NOT on Karpenter-managed nodes"
    api: "kubectl get pods karpenter + node labels"
    condition: "Only if Karpenter detected"
    failure_action: "STOP — self-hosted Karpenter may evict itself during drift"

warning_checks:
  - id: insights_warnings
    description: "EKS Insights show warnings (not errors) for target version"
    api: "eks:ListInsights"
    warning_action: "Present findings, recommend remediation before proceeding"

  - id: pdb_blocking
    description: "PDBs with zero allowed disruptions detected"
    api: "kubectl get pdb -A"
    condition: "Only if kubectl available"
    warning_action: "Present specific PDBs, recommend adjustment"

  - id: deprecated_apis
    description: "Deprecated API usage detected for target version"
    api: "eks:ListInsights (primary) or kubectl"
    warning_action: "Present API migration requirements"

  - id: self_managed_addons
    description: "Self-managed add-ons detected (not via EKS add-on API)"
    api: "kubectl get deploy vs eks:ListAddons"
    condition: "Only if kubectl available"
    warning_action: "Flag for manual version verification"

  - id: webhooks_fail_policy
    description: "Webhooks with failurePolicy=Fail targeting broad API groups"
    api: "kubectl get validating/mutatingwebhookconfigurations"
    condition: "Only if kubectl available"
    warning_action: "Present webhook details, assess compatibility"

  - id: control_plane_logging_disabled
    description: "Control plane audit logging not enabled"
    api: "eks:DescribeCluster → logging"
    warning_action: "Recommend enabling before upgrade for debugging"

  - id: helm_stored_deprecated_apis
    description: "Helm stored manifests contain deprecated APIs for target"
    api: "kubectl get secrets -l owner=helm,status=deployed"
    condition: "Only if kubectl + helm available"
    warning_action: "Recommend running helm mapkubeapis before upgrade"

  - id: bare_pods_detected
    description: "Pods without ownerReferences found on cluster"
    api: "kubectl get pods -A"
    condition: "Only if kubectl available"
    warning_action: "Flag — these pods won't be rescheduled after drain"

  - id: ebs_az_pinning
    description: "EBS PVCs zone-pinned with multi-AZ node groups"
    api: "kubectl get pvc,pv + ec2:DescribeSubnets"
    condition: "Only if kubectl available"
    warning_action: "Flag — cross-AZ reschedule will strand these volumes"

  - id: karpenter_consolidation_active
    description: "Karpenter consolidation may race with drift during upgrade"
    api: "kubectl get nodepools (consolidateAfter)"
    condition: "Only if Karpenter detected"
    warning_action: "Recommend pausing consolidation during upgrade window"

  - id: management_plane_unknown
    description: "Cannot determine cluster management plane (manual/IaC/ACK/KRO)"
    api: "eks:DescribeCluster → tags, kubectl get crd"
    warning_action: "Ask operator to confirm management approach"

# Timeout and soak period guidance for upgrade execution
timeouts:
  control_plane_upgrade:
    expected_minutes: 25
    alert_at_minutes: 40
    escalate_at_minutes: 60
    note: "CRD-heavy clusters (>200 CRDs) may take 60-90 min"

  node_group_upgrade:
    per_node_estimate_seconds: 120
    default_minutes: 60
    note: "Based on node count × per-node-time × (1/maxUnavailable)"

  addon_update:
    default_minutes: 15
    verification_wait_seconds: 30

  node_bootstrap:
    default_seconds: 900
    gpu_instance_seconds: 1200
    windows_instance_seconds: 1200

soak_periods:
  post_control_plane_minutes: 15
  post_first_nodegroup_minutes: 30
  between_nodegroup_batches_minutes: 10
  post_addon_update_seconds: 30
  post_all_upgrades_minutes: 60
  note: "Soak periods allow detection of delayed failure modes (OOM, connection exhaustion, cert expiry)"

# Rollback awareness
rollback:
  control_plane:
    available_since: "July 2026"
    window_days: 7
    conditions:
      - "Cluster was upgraded to current version (not created at it)"
      - "Within 7 days of upgrade completion"
      - "Single version rollback only (N → N-1)"
      - "Target rollback version still supported by EKS"
      - "Cluster status is ACTIVE"
      - "No features enabled at current version incompatible with N-1"
    what_rolls_back:
      - "API server and control plane components"
      - "Platform version"
      - "Auto Mode nodes (automatically)"
    what_does_not_roll_back:
      - "etcd data"
      - "Workloads and their configs"
      - "Add-ons (manual downgrade required)"
      - "Persistent volumes"
      - "Managed node groups (manual)"
      - "Self-managed nodes (manual)"
      - "Fargate pods (must delete/recreate)"
    post_upgrade_check: "eks:ListInsights --filter categories=ROLLBACK_READINESS"
    initiate_command: "aws eks update-cluster-version --name <cluster> --kubernetes-version <N-1>"
    force_flag: "--force (bypasses insight checks but NOT prerequisite validations)"

  addons:
    reversibility: "full"
    method: "aws eks update-addon --addon-version <previous-version>"
    window: "indefinite"

  managed_node_groups:
    reversibility: "partial"
    note: "Can halt mid-upgrade; completed nodes stay at new version"
    method: "Cannot downgrade — must recreate node group at old version if needed"

  karpenter_nodes:
    reversibility: "full"
    method: "Revert EC2NodeClass amiSelectorTerms, delete drifted NodeClaims"
    window: "indefinite"

  self_managed_nodes:
    reversibility: "full"
    method: "Revert launch template version, terminate new nodes"
    window: "indefinite"

# EKS Upgrade Readiness — Required Check Registry
# Version: 2.0.0
# Purpose: Defines all checks for the upgrade readiness assessment.
#          Evidence Completeness = checks_performed / total_applicable_checks
#
# Fields:
#   id            - Unique identifier
#   description   - What the check verifies
#   category      - pre_flight | node_assessment | addon_assessment |
#                   workload_assessment | infrastructure | networking |
#                   karpenter | pre_drain | rollback
#   required      - always | if_kubectl | if_helm | if_karpenter |
#                   if_fargate | if_encrypted | if_custom_ami |
#                   if_self_managed_nodes | if_auto_mode | if_post_upgrade
#   api           - Primary API or method used
#   severity      - critical | high | medium | low (if check fails)

checks:

  # ── PRE-FLIGHT ────────────────────────────────────────────────────────────

  - id: PF-01
    description: "Cluster status is ACTIVE"
    category: pre_flight
    required: always
    api: "eks:DescribeCluster → status"
    severity: critical

  - id: PF-02
    description: "Cluster version identified, target determined"
    category: pre_flight
    required: always
    api: "eks:DescribeCluster → version"
    severity: critical

  - id: PF-03
    description: "Cluster not mid-upgrade (no pending platform update)"
    category: pre_flight
    required: always
    api: "eks:ListUpdates"
    severity: high

  - id: PF-04
    description: "Control plane logging enabled (audit + authenticator)"
    category: pre_flight
    required: always
    api: "eks:DescribeCluster → logging"
    severity: medium

  - id: PF-05
    description: "EKS Upgrade Insights queried (primary signal)"
    category: pre_flight
    required: always
    api: "eks:ListInsights, eks:DescribeInsight"
    severity: critical

  - id: PF-06
    description: "Cluster IAM role valid with correct trust policy"
    category: pre_flight
    required: always
    api: "iam:GetRole, iam:ListAttachedRolePolicies"
    severity: critical

  - id: PF-07
    description: "KMS key valid and accessible (if encryption enabled)"
    category: pre_flight
    required: if_encrypted
    api: "eks:DescribeCluster → encryptionConfig"
    severity: critical

  - id: PF-08
    description: "All nodes Ready, no pending CSRs, no crash-looping system pods"
    category: pre_flight
    required: if_kubectl
    api: "kubectl get nodes, kubectl get csr, kubectl get pods -n kube-system"
    severity: high

  - id: PF-09
    description: "DNS resolution working (CoreDNS healthy)"
    category: pre_flight
    required: if_kubectl
    api: "kubectl get pods -n kube-system -l k8s-app=kube-dns"
    severity: high

  - id: PF-10
    description: "Metrics server responding"
    category: pre_flight
    required: if_kubectl
    api: "kubectl top nodes"
    severity: low

  - id: PF-11
    description: "Version skew lower-bound gate — kubelets meet target N-3 (1.28+) or N-2 (<1.28)"
    category: pre_flight
    required: always
    api: "eks:DescribeNodegroup (version), kubectl get nodes"
    severity: critical

  - id: PF-12
    description: "Current-state upper-bound gate — no kubelet is newer than the current control plane"
    category: pre_flight
    required: always
    api: "eks:DescribeCluster → version, kubectl get nodes → status.nodeInfo.kubeletVersion"
    severity: critical

  - id: PF-13
    description: "Management plane detected (self-managed/ACK/KRO/GitOps/unknown)"
    category: pre_flight
    required: always
    api: "eks:DescribeCluster → tags, kubectl get crd clusters.eks.services.k8s.aws"
    severity: medium

  # ── INFRASTRUCTURE ────────────────────────────────────────────────────────

  - id: INFRA-01
    description: "Subnet and VPC CNI mode-specific surge capacity (node/pod subnets, prefixes, or branch ENIs)"
    category: infrastructure
    required: always
    api: "ec2:DescribeSubnets, aws-node env/config, kubectl get eniconfig"
    severity: critical

  - id: INFRA-02
    description: "EC2 vCPU quota headroom for surge nodes"
    category: infrastructure
    required: always
    api: "servicequotas:GetServiceQuota (L-1216C47A)"
    severity: high

  - id: INFRA-03
    description: "EBS volume quota headroom"
    category: infrastructure
    required: always
    api: "servicequotas:GetServiceQuota (L-7A658000)"
    severity: high

  - id: INFRA-04
    description: "VPC CNI mode detected (standard/prefix-delegation/custom-networking/IPv6/SGP)"
    category: networking
    required: always
    api: "eks:DescribeAddon (vpc-cni), kubectl get ds aws-node env vars"
    severity: medium

  - id: INFRA-06
    description: "Mode-specific VPC CNI capacity calculation (prefix /28, ENIConfig pod subnet, or branch-ENI limit)"
    category: networking
    required: if_kubectl
    api: "ec2:DescribeSubnets, kubectl get eniconfig, ec2:DescribeNetworkInterfaces"
    severity: critical

  - id: INFRA-05
    description: "Security groups allow required control-plane to node communication"
    category: networking
    required: always
    api: "ec2:DescribeSecurityGroups"
    severity: high

  # ── NODE ASSESSMENT ───────────────────────────────────────────────────────

  - id: NODE-01
    description: "Managed node group inventory (all MNGs described)"
    category: node_assessment
    required: always
    api: "eks:ListNodegroups, eks:DescribeNodegroup"
    severity: critical

  - id: NODE-02
    description: "Self-managed ASG inventory"
    category: node_assessment
    required: if_self_managed_nodes
    api: "autoscaling:DescribeAutoScalingGroups (filtered by cluster tag)"
    severity: high

  - id: NODE-03
    description: "AMI family detection (AL2, AL2023, Bottlerocket, Windows, Custom)"
    category: node_assessment
    required: always
    api: "eks:DescribeNodegroup → amiType"
    severity: critical

  - id: NODE-04
    description: "AL2 hard gate check (AL2 unavailable for target >= 1.33)"
    category: node_assessment
    required: always
    api: "eks:DescribeNodegroup → amiType vs target version"
    severity: critical

  - id: NODE-05
    description: "Custom launch template AMI detection"
    category: node_assessment
    required: always
    api: "ec2:DescribeLaunchTemplateVersions → ImageId"
    severity: high

  - id: NODE-06
    description: "Fargate profile inventory"
    category: node_assessment
    required: if_fargate
    api: "eks:ListFargateProfiles, eks:DescribeFargateProfile"
    severity: medium

  - id: NODE-07
    description: "Auto Mode detection and config"
    category: node_assessment
    required: if_auto_mode
    api: "eks:DescribeCluster → computeConfig"
    severity: medium

  - id: NODE-08
    description: "Custom AMI bootstrap method (bootstrap.sh vs nodeadm)"
    category: node_assessment
    required: if_custom_ami
    api: "ec2:DescribeLaunchTemplateVersions → UserData"
    severity: high

  # ── ADDON ASSESSMENT ──────────────────────────────────────────────────────

  - id: ADDON-01
    description: "EKS managed addon inventory with current versions"
    category: addon_assessment
    required: always
    api: "eks:ListAddons, eks:DescribeAddon"
    severity: critical

  - id: ADDON-02
    description: "Addon compatibility verified against target via live API"
    category: addon_assessment
    required: always
    api: "eks:DescribeAddonVersions"
    severity: critical

  - id: ADDON-03
    description: "Self-managed addon detection including core VPC CNI, CoreDNS, and kube-proxy images/config"
    category: addon_assessment
    required: if_kubectl
    api: "kubectl get daemonset aws-node,kube-proxy; deployment coredns; configmaps"
    severity: high

  - id: ADDON-04
    description: "Addon upgrade order calculated (dependency chain)"
    category: addon_assessment
    required: always
    api: "derived from ADDON-01 + ADDON-02"
    severity: medium

  - id: ADDON-05
    description: "Third-party controller compatibility (LBC, cert-manager, external-dns, etc.)"
    category: addon_assessment
    required: if_kubectl
    api: "kubectl get deploy (image tags) vs compatibility matrix"
    severity: high

  # ── WORKLOAD ASSESSMENT ───────────────────────────────────────────────────

  - id: WKLD-01
    description: "Deprecated API scan (live resources)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl, kubent, pluto"
    severity: high

  - id: WKLD-02
    description: "Helm stored manifest API scan (deployed revision only)"
    category: workload_assessment
    required: if_helm
    api: "kubectl get secrets -l owner=helm,status=deployed"
    severity: critical

  - id: WKLD-03
    description: "Third-party CRD API deprecation check (vendor/version-aware)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get crds, kubectl get deploy (controller versions)"
    severity: high

  - id: WKLD-04
    description: "PDB blocking analysis (maxUnavailable=0 or minAvailable=replicas)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get pdb -A"
    severity: high

  - id: WKLD-05
    description: "Admission webhook compatibility (failurePolicy=Fail on broad groups)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get validatingwebhookconfigurations, mutatingwebhookconfigurations"
    severity: high

  - id: WKLD-06
    description: "StatefulSet safety (grace period, PVC retention, single-replica)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get statefulsets -A"
    severity: medium

  - id: WKLD-07
    description: "Scaled-to-zero workload detection"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get deploy,sts (replicas=0)"
    severity: low

  - id: WKLD-08
    description: "TopologySpreadConstraints on critical workloads"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get deploy -A (check spec)"
    severity: low

  - id: WKLD-09
    description: "Service mesh detection (Istio, App Mesh, Linkerd)"
    category: workload_assessment
    required: if_kubectl
    api: "kubectl get deploy, kubectl get crd"
    severity: medium

  # ── KARPENTER ─────────────────────────────────────────────────────────────

  - id: KARP-01
    description: "Karpenter version compatibility with target K8s version"
    category: karpenter
    required: if_karpenter
    api: "kubectl get deploy karpenter -n kube-system (image tag)"
    severity: critical

  - id: KARP-02
    description: "Drift feature gate enabled (default on v0.33+)"
    category: karpenter
    required: if_karpenter
    api: "kubectl get deploy karpenter env FEATURE_GATES"
    severity: high

  - id: KARP-03
    description: "expireAfter not set to Never on any NodePool"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodepools"
    severity: high

  - id: KARP-04
    description: "Disruption budgets allow at least 1 node disruption"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodepools (spec.disruption.budgets)"
    severity: high

  - id: KARP-05
    description: "EC2NodeClass amiSelectorTerms not pinned to specific AMI ID"
    category: karpenter
    required: if_karpenter
    api: "kubectl get ec2nodeclasses"
    severity: high

  - id: KARP-06
    description: "EC2NodeClass amiFamily not AL2 when target >= 1.33"
    category: karpenter
    required: if_karpenter
    api: "kubectl get ec2nodeclasses (spec.amiFamily)"
    severity: critical

  - id: KARP-07
    description: "Karpenter controller not self-hosted on Karpenter-managed nodes"
    category: karpenter
    required: if_karpenter
    api: "kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter (nodeSelector)"
    severity: critical

  - id: KARP-08
    description: "NodePool nodeClassRef points to existing EC2NodeClass"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodepools, kubectl get ec2nodeclasses"
    severity: high

  - id: KARP-09
    description: "Consolidation interference check (short consolidateAfter + drift)"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodepools (spec.disruption)"
    severity: medium

  - id: KARP-10
    description: "Disruption budget schedule does not conflict with upgrade window"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodepools (spec.disruption.budgets[].schedule)"
    severity: medium

  - id: KARP-11
    description: "NodeClaim expireAfter won't trigger during upgrade window"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodeclaims"
    severity: medium

  - id: KARP-12
    description: "Drift throughput estimation (time to replace all nodes)"
    category: karpenter
    required: if_karpenter
    api: "kubectl get nodes (karpenter-managed count) + budget rate"
    severity: low

  - id: KARP-13
    description: "Karpenter controller health (all replicas ready, no crash-loop)"
    category: karpenter
    required: if_karpenter
    api: "kubectl get deploy karpenter -n kube-system"
    severity: critical

  - id: KARP-14
    description: "v1alpha5 Provisioner orphan detection (incomplete migration)"
    category: karpenter
    required: if_karpenter
    api: "kubectl get crd provisioners.karpenter.sh"
    severity: medium

  # ── PRE-DRAIN SAFETY ──────────────────────────────────────────────────────

  - id: DRAIN-01
    description: "Bare pods (no ownerReferences) on nodes — not rescheduled after eviction"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get pods -A (filter no ownerReferences)"
    severity: high

  - id: DRAIN-02
    description: "Pods with emptyDir volumes — data lost on drain"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get pods -A (volume type emptyDir)"
    severity: medium

  - id: DRAIN-03
    description: "Pods with custom finalizers — can hang eviction indefinitely"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get pods -A (metadata.finalizers)"
    severity: medium

  - id: DRAIN-04
    description: "EBS AZ-pinned PVCs — cross-AZ reschedule strands the volume"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get pvc,pv + ec2:DescribeSubnets (AZ match)"
    severity: critical

  - id: DRAIN-05
    description: "Fail-closed webhooks on drain-target nodes — eviction deadlock"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get webhookconfigs + kubectl get endpoints"
    severity: critical

  - id: DRAIN-06
    description: "CoreDNS SPOF — all replicas on same drain batch"
    category: pre_drain
    required: if_kubectl
    api: "kubectl get pods -n kube-system -l k8s-app=kube-dns (node distribution)"
    severity: critical

  # ── ROLLBACK ──────────────────────────────────────────────────────────────

  - id: ROLL-01
    description: "Pre-upgrade rollback advisory (eligibility conditions and 7-day window documented; not yet verifiable)"
    category: rollback
    required: always
    api: "advisory from upgrade plan"
    severity: medium

  - id: ROLL-02
    description: "Post-upgrade ROLLBACK_READINESS insights are complete, fresh, and passing"
    category: rollback
    required: if_post_upgrade
    api: "eks:ListInsights --filter categories=ROLLBACK_READINESS; eks:DescribeInsight"
    severity: high

# SAFETY INVARIANTS — Load First, Keep in Context

> This file defines non-negotiable safety rules for the EKS Upgrade Readiness
> skill. Every rule here is a hard constraint. Violating any Hard Rule is a
> critical defect in the assessment.

## Knowledge Hierarchy (highest authority wins)

```
1. Live API response from THIS session (eks:Describe*, ec2:Describe*, kubectl get)
   ▼ overrides
2. Config files in this skill (YAML references, check registry)
   ▼ overrides
3. Fetched documentation (EKS docs, upstream K8s release notes)
   ▼ overrides
4. Training data / model knowledge
```

When sources conflict, the higher-numbered source is WRONG. Live API responses
are ground truth. Never override a live API response with cached or trained
knowledge.

**Example:** If `DescribeAddonVersions` returns a version list, use ONLY those
versions — never suggest a version remembered from training data.

## Hard Rules

| # | Rule | Rationale |
|---|------|-----------|
| H1 | NEVER select an add-on version not returned by `DescribeAddonVersions` in THIS session | Stale version data causes upgrade failures |
| H2 | NEVER execute a mutating API call — this skill is READ-ONLY | All mutations are in the Remediation Playbook for operator approval |
| H3 | NEVER skip a required check from the registry — mark it `SKIPPED` with reason | Evidence Completeness must be accurate |
| H4 | NEVER mark a gate PASS when data is missing or access was denied | UNKNOWN is the only valid verdict for missing data |
| H5 | NEVER produce a READY verdict while any gate is UNKNOWN | Operator must investigate unknowns before proceeding |
| H6 | ALWAYS document rollback classification before recommending any step | Operator must know recovery options |
| H7 | ALWAYS paginate API results to exhaustion — partial results produce LOW confidence | Incomplete data is dangerous |
| H8 | NEVER claim the control plane upgrade is irreversible without checking rollback eligibility | EKS supports 7-day rollback (since July 2026) under specific conditions |
| H9 | NEVER operate on a cluster without first confirming target identity (name, account, region) | Wrong-cluster assessments are useless |
| H10 | NEVER assume node group AMI type without checking `DescribeNodegroup` | Custom AMIs have unpredictable behavior |

## Operation Classification

This skill is **read-only by design**. All operations are Tier 3 (Allowed):

| Tier | Operations | This Skill |
|------|-----------|------------|
| Tier 3: ALLOWED | `describe*`, `list*`, `get*`, `kubectl get/describe` | ✅ All assessment work |
| Tier 2: REVIEW-REQUIRED | `update-addon`, `patch`, `helm upgrade` | ❌ In Remediation Playbook only |
| Tier 1: BLOCKED | `update-cluster-version`, `delete-*`, `drain` | ❌ In Remediation Playbook only |

## Uncertainty Handling

```
Agent encounters unknown condition
     │
     ▼
Can it be verified via live API call?
     │
     ├─ Yes → Call API, use response as ground truth
     │
     └─ No → Mark gate as UNKNOWN
              Report: what was encountered, what was attempted, why uncertain
              Overall verdict: CANNOT DETERMINE
```

- NEVER guess at version compatibility — verify via API or declare UNKNOWN
- NEVER infer cluster configuration from naming conventions alone
- NEVER assume an add-on is EKS-managed without checking `ListAddons`
- Empty API results ≠ PASS (means "no data" not "no problem")
- Pagination exhausted without `nextToken` = complete. Stopped early = LOW confidence.

## Scope Restrictions

- All operations scoped to the cluster(s) the operator specified
- CloudWatch queries (if any) target only `/aws/eks/<cluster-name>/cluster`
- Cross-cluster operations forbidden unless explicitly requested
- Cross-account operations require explicit account list from operator

## AccessDenied Protocol

When any API returns `AccessDeniedException` or `Forbidden`:
1. Log which permission is missing
2. Mark the affected gate as **UNKNOWN**
3. Continue with remaining checks (do not abort entire assessment)
4. Include in report: "Gate X: UNKNOWN — AccessDenied on `<API>`"
5. Overall verdict: CANNOT DETERMINE (UNKNOWN gates exist)

# EKS Upgrade Troubleshooting

Common failures during EKS upgrades and their resolutions.

## Control Plane Upgrade Failures

### Upgrade stuck in "Updating" for > 60 minutes

**Causes:**
- Webhook configurations blocking API server startup
- Custom admission controllers not compatible with new version
- Insufficient IAM permissions for EKS service role

**Resolution:**
1. Check EKS update status: `aws eks describe-update --name <cluster> --update-id <id>`
2. Review CloudTrail for EKS API errors
3. If webhook is blocking: the control plane will eventually recover by
   skipping the webhook; no user action needed but it causes delays
4. If IAM: verify `AmazonEKSClusterPolicy` is attached to cluster role

### Control plane upgrade succeeded but kubectl fails

**Causes:**
- kubeconfig pointing to old endpoint
- aws-auth ConfigMap missing after upgrade (rare)
- Client version too old for new API server

**Resolution:**
1. Update kubeconfig: `aws eks update-kubeconfig --name <cluster>`
2. Verify: `kubectl version` — client should be within one minor of server
3. Check aws-auth: `kubectl get configmap aws-auth -n kube-system`

## Addon Upgrade Failures

### Addon update returns "ConfigurationConflict"

**Cause:** Addon was manually modified outside of EKS addon management.

**Resolution:** This is an operator-approved mutation. Inspect and back up the
existing managed-addon configuration first, then choose deliberately:

```bash
# Preserve intentional customer configuration when it is target-compatible.
aws eks update-addon --cluster-name <cluster> \
  --addon-name <addon> --addon-version <version> \
  --resolve-conflicts PRESERVE

# OVERWRITE replaces conflicting customer configuration with EKS defaults.
# Use only after reviewing the diff, recording configurationValues, and
# confirming the rollback plan.
aws eks update-addon --cluster-name <cluster> \
  --addon-name <addon> --addon-version <version> \
  --resolve-conflicts OVERWRITE
```

### CoreDNS not running after upgrade

**Causes:**
- Corefile incompatible with new version
- Pod scheduling issues (taints, resource limits)

**Resolution:**
1. Check pods: `kubectl get pods -n kube-system -l k8s-app=kube-dns`
2. Check events: `kubectl describe pod <coredns-pod> -n kube-system`
3. If Corefile issue: check `kubectl get configmap coredns -n kube-system -o yaml`

## Node Group Upgrade Failures

### Nodes not draining (upgrade stuck)

**Causes:**
- PDB with `maxUnavailable: 0` blocking eviction
- Pod with no controller (standalone pod without owner)
- Local storage preventing eviction (emptyDir with data)
- Finalizers blocking pod deletion

**Resolution:**
1. Check PDBs: `kubectl get pdb --all-namespaces`
2. Identify blocking pods from node group update events:
   ```
   aws eks describe-update --name <cluster> --update-id <id> --nodegroup-name <ng>
   ```
3. Temporarily adjust PDB: `kubectl patch pdb <name> -p '{"spec":{"maxUnavailable":1}}'`
4. For standalone pods: delete manually or add controller

### New nodes joining but pods not scheduling

**Causes:**
- Taints on new nodes not tolerated by workloads
- Node labels changed between AMI versions
- Resource requests exceed new node capacity

**Resolution:**
1. Check node taints: `kubectl describe node <new-node> | grep Taint`
2. Check pending pods: `kubectl get pods --field-selector=status.phase=Pending`
3. Check events: `kubectl describe pod <pending-pod>`

### InsufficientInstanceCapacity during node group upgrade

**Cause:** EC2 cannot launch the required instance type in the AZ.

**Resolution:**
1. Check which AZ is constrained from the update error
2. Options:
   - Wait and retry (capacity may free up)
   - Use Capacity Reservations (see `capacity-planning.md`)
   - Add alternative instance types to the node group
   - Reduce `maxUnavailablePercentage` to lower simultaneous surge

### Launch template version mismatch

**Cause:** Custom launch template AMI doesn't match target EKS version.

**Resolution:**
1. Check LT: `aws ec2 describe-launch-template-versions --launch-template-id <lt-id>`
2. Update AMI to match target version:
   ```
   aws ssm get-parameter --name /aws/service/eks/optimized-ami/<version>/amazon-linux-2023/x86_64/standard/recommended/image_id
   ```
3. Create new LT version with correct AMI
4. Update node group to use new LT version

## Karpenter-Specific Issues

### Karpenter not launching nodes with new AMI after upgrade

**Cause:** `amiSelectorTerms` in EC2NodeClass pinned to old version.

**Resolution:**
1. Check EC2NodeClass: `kubectl get ec2nodeclass -o yaml`
2. Update `amiSelectorTerms` to include new version or use `amiFamily` for auto-discovery
3. Roll nodes: `kubectl delete nodes -l karpenter.sh/nodepool=<pool>`

### Karpenter version incompatible with new EKS version

**Cause:** Old Karpenter release doesn't support new K8s API version.

**Resolution:**
1. Check Karpenter compatibility matrix in release notes
2. Upgrade Karpenter BEFORE or alongside control plane upgrade
3. For Karpenter v1.x, check minimum EKS version in docs

## Rollback Scenarios

Control-plane rollback is **conditional**, not one-way: EKS makes it available
only to an eligible cluster for seven days after a successful upgrade. During
that window, query `ListInsights` with `ROLLBACK_READINESS` and resolve every
`ERROR` or `UNKNOWN` before the operator attempts rollback. Outside the window,
or when eligibility conditions are not met, the control plane must be fixed
forward.

| Component | Rollback Possible? | How |
|-----------|-------------------|-----|
| Control plane | CONDITIONAL (7 days after eligible upgrade) | Operator performs rollback only after `ROLLBACK_READINESS` insights PASS; otherwise fix forward or follow the documented forced-rollback procedure |
| Addons | YES | `aws eks update-addon --addon-version <old-version>` |
| Managed node groups | PARTIAL | Can halt; completed nodes stay at new version |
| Self-managed nodes | YES | Revert launch template, terminate new nodes |
| Karpenter nodes | YES | Revert EC2NodeClass AMI, delete nodes |

## Prevention Checklist

- [ ] Run EKS Upgrade Insights before starting
- [ ] Test upgrade in a non-production cluster first
- [ ] Enable cluster audit logging before upgrade
- [ ] Take Velero backup of critical resources
- [ ] Verify all webhooks are compatible with target version
- [ ] Confirm no pending node group health issues
- [ ] Schedule upgrade during low-traffic window
- [ ] Have rollback plan for addons and node groups

## Feature-Specific Removal Guidance

### Dockershim Removal (EKS 1.25)

EKS Optimized AMI for 1.25+ no longer includes Dockershim. If workloads mount
the Docker socket (`/var/run/docker.sock`), they will break.

**Detection:**
```bash
# Install and run Detector for Docker Socket (DDS)
kubectl krew install dds
kubectl dds
```

**Resolution:** Remove Docker socket dependencies. Use containerd-compatible
alternatives or CRI APIs directly.

### PodSecurityPolicy Removal (EKS 1.25)

PSP was removed in Kubernetes 1.25. Clusters using PSP must migrate before
upgrading.

**Detection:**
```bash
kubectl get psp
# If any PSPs exist, migration is required
```

**Migration options:**
1. Pod Security Standards (PSS) with Pod Security Admission (PSA) — built-in
2. Policy-as-code: OPA Gatekeeper or Kyverno

See [AWS PSP removal FAQ](https://docs.aws.amazon.com/eks/latest/userguide/pod-security-policy-removal-faq.html).

### In-Tree Storage Driver Deprecation (EKS 1.23)

The `kubernetes.io/aws-ebs` in-tree provisioner is deprecated. Must use the
EBS CSI driver (`ebs.csi.aws.com`) before upgrading to 1.23+.

**Detection:**
```bash
kubectl get sc -o jsonpath='{range .items[*]}{.metadata.name}: {.provisioner}{"\n"}{end}'
# Look for kubernetes.io/aws-ebs
kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.csi.driver // "in-tree"}{"\n"}{end}'
```

**Resolution:** Install the [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) and create new StorageClasses
using `ebs.csi.aws.com`. Existing PVs will be handled by CSI migration (automatic).

## Useful Upgrade Tools

| Tool | Purpose | Link |
|------|---------|------|
| kubent | Scan cluster for deprecated APIs | https://github.com/doitintl/kube-no-trouble |
| pluto | Detect deprecated APIs in cluster and Helm charts | https://pluto.docs.fairwinds.com |
| kubectl-convert | Convert manifests between API versions | https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-convert-plugin |
| eksup (ClowdHaus) | EKS upgrade guidance CLI | https://clowdhaus.github.io/eksup |
| GoNoGo | Determine upgrade confidence for add-ons | https://github.com/FairwindsOps/GoNoGo |
| DDS | Detect Docker socket dependencies | https://github.com/aws-containers/kubectl-detector-for-docker-socket |
| Velero | Cluster backup before upgrade | https://velero.io |
| AWS Backup | Managed backup for EKS | https://docs.aws.amazon.com/eks/latest/userguide/integration-backup.html |

## Blue-Green Cluster Strategy

For very large clusters or when skipping multiple minor versions is required:

**Benefits:**
- Can jump multiple EKS versions at once
- Able to switch back to old cluster if issues arise
- Creates a fresh cluster with latest configurations

**Downsides:**
- API endpoint and OIDC change (requires updating all consumers: kubectl, CI/CD, IRSA)
- Two clusters running in parallel (cost, capacity limits)
- Load balancers and external DNS cannot easily span clusters
- Stateful workload migration requires careful planning (data backup + restore)
- More coordination needed if workloads depend on each other

**When to consider:**
- Cluster is 3+ minor versions behind
- In-place sequential upgrades would take too long or be too risky
- Cluster was created with legacy tooling and needs to be rebuilt with modern IaC
- Compliance requires a clean-state cluster

### Identity Migration Considerations (IRSA vs. Pod Identity)

Both mechanisms need work on a new cluster, but the work is different — don't
assume "no IAM changes" means "no identity work":

| Mechanism | What Must Happen on the New Cluster | Effort |
|-----------|--------------------------------------|--------|
| **IRSA** | Each new cluster has its own OIDC provider ARN. Existing IAM role trust policies must be updated to also trust the new cluster's OIDC provider (a trust policy can list multiple issuers, but is capped at 4096 characters — roles shared across many clusters can hit this limit). | Edit IAM role trust policies |
| **Pod Identity** | The IAM role's trust policy does not change (it trusts the cluster-agnostic `pods.eks.amazonaws.com` service principal). However, associations (service account ↔ role mappings) are stored as an EKS resource scoped to one cluster — each association must be explicitly recreated with `aws eks create-pod-identity-association` on the new cluster. Nothing carries over automatically. | Recreate every association (no IAM edits) |

Before a blue-green cutover, inventory both:
```bash
# IRSA: service accounts with role-arn annotations
kubectl get sa -A -o json | jq '.items[] | select(.metadata.annotations["eks.amazonaws.com/role-arn"] != null) | {ns: .metadata.namespace, sa: .metadata.name, role: .metadata.annotations["eks.amazonaws.com/role-arn"]}'

# Pod Identity: existing associations on the source cluster
aws eks list-pod-identity-associations --cluster-name <source-cluster>
```
Every association returned by `list-pod-identity-associations` needs an
equivalent `create-pod-identity-association` call against the new cluster
before cutting workloads over — this is a mutation and belongs in the
Remediation Playbook (Step 14 of SKILL.md), not something the assessment
executes automatically.