# Karpenter Consolidation Eligibility

## What consolidation does

Karpenter consolidation (enabled via `consolidationPolicy: WhenUnderutilized` or `WhenEmpty` in a NodePool) automatically removes underutilized nodes and reschedules their pods to better-packed nodes. This reduces idle node cost without requiring manual intervention.

For a pod to be a candidate for consolidation, Karpenter must be able to disrupt it safely and reschedule it onto a smaller or fewer nodes.

## Eligibility conditions

A pod is consolidation-eligible when all five conditions are met:

### 1. No blocking PodDisruptionBudget

A PodDisruptionBudget (PDB) blocks consolidation if it would prevent Karpenter from evicting the pod at the time of consolidation.

Blocking PDB configurations:
- `maxUnavailable: 0` — zero replicas can be disrupted; Karpenter cannot evict any pod.
- `minAvailable` equal to the current replica count — effectively blocks all evictions.
- `maxUnavailable: 0%` — same as `maxUnavailable: 0` in percentage form.

Non-blocking PDB configurations:
- `maxUnavailable: 1` or higher — at least one replica can be disrupted.
- `minAvailable` less than the current replica count — some replicas can be disrupted.

To check for blocking PDBs: the caller should inspect `kubectl get pdb -n <namespace>` output and confirm whether any PDB covers the workload.

### 2. No consolidation-blocking pod anti-affinity

Pod anti-affinity rules that spread replicas across nodes by `kubernetes.io/hostname` (host anti-affinity) can prevent Karpenter from co-locating pods that are currently on separate nodes.

If every replica of a Deployment has a `requiredDuringSchedulingIgnoredDuringExecution` anti-affinity rule against other replicas on the same host, Karpenter cannot pack them together, blocking node consolidation.

`preferredDuringSchedulingIgnoredDuringExecution` anti-affinity is soft and does not block consolidation.

### 3. No topologySpreadConstraints with WhenUnsatisfiable: DoNotSchedule

`topologySpreadConstraints` with `whenUnsatisfiable: DoNotSchedule` and a spread domain of `kubernetes.io/hostname` prevents Karpenter from scheduling multiple pods onto the same node, blocking consolidation of node pairs.

`whenUnsatisfiable: ScheduleAnyway` is soft and does not block consolidation.

### 4. No local storage (hostPath or local PV)

Pods that use `hostPath` volumes or PersistentVolumes backed by a `local` StorageClass are pinned to the node where the volume exists. Karpenter cannot reschedule these pods to a different node without data loss.

Pods using network-attached storage (EBS, Azure Disk, GCE PD, OCI Block Volume) are not pinned and are eligible for consolidation — the volume can be re-attached to the new node.

Note: EBS volumes in AWS have an availability-zone constraint; consolidation targets must be in the same AZ as the volume.

### 5. No nodeName selector pinning

A pod with `spec.nodeName` set to a specific node name is pinned to that node and cannot be rescheduled. Karpenter will not consolidate the host node as long as pinned pods remain on it.

Similarly, a `nodeSelector` that matches only one specific node (e.g., `kubernetes.io/hostname: <exact-node-name>`) effectively pins the pod.

## Summary eligibility table

| Condition | Eligible when | Blocks consolidation when |
|---|---|---|
| PodDisruptionBudget | No PDB, or PDB with `maxUnavailable >= 1` | PDB with `maxUnavailable: 0` or `minAvailable == replica count` |
| Pod anti-affinity | No anti-affinity, or `preferredDuring...` only | `requiredDuring...` host anti-affinity between replicas |
| TopologySpreadConstraints | No constraints, or `whenUnsatisfiable: ScheduleAnyway` | `whenUnsatisfiable: DoNotSchedule` on hostname topology |
| Storage | Network-attached PV only | `hostPath` volume or `local` StorageClass PV |
| Node selector | No `nodeName` and no single-node `nodeSelector` | `spec.nodeName` set or `nodeSelector` matching one node |

## Consolidation policy modes

| Policy | Behavior |
|---|---|
| `WhenEmpty` | Consolidates only fully empty nodes (all pods evicted or completed). Conservative. |
| `WhenUnderutilized` | Consolidates nodes when doing so would reduce total node count without violating pod scheduling constraints. Aggressive. |

For FinOps purposes, `WhenUnderutilized` provides the greater cost reduction but requires that pods pass all five eligibility conditions above.

## Disruption budget in Karpenter v0.33+

Karpenter v0.33 introduced a `disruption.budgets` field in the NodePool spec that limits the rate of node replacement across all consolidation actions. A low disruption budget (e.g., `nodes: "10%"`) slows consolidation speed but reduces service disruption risk.

The disruption budget does not affect eligibility; it affects when Karpenter will act on an eligible pod.

# Metric Sources

This reference describes how to gather p50, p95, and p99 CPU and memory utilization metrics from common observability platforms for use as input to the rightsize-recommendation skill. These are recipes and query patterns, not commands to be executed by the skill.

## Why percentile metrics matter

Using only the average (mean) of CPU or memory utilization understates peaks and leads to under-resourced pods. Using the p99 alone for requests leads to over-provisioning for workloads with rare spikes. The combination of p50, p95, and p99 over a 7-14 day window captures normal load, burst behavior, and outlier events.

- p50: median utilization; represents steady-state load.
- p95: captures the top 5% of utilization events; used for request sizing.
- p99: captures the top 1% of utilization events; used for limit sizing.

## Prometheus (self-hosted or managed)

### CPU utilization per pod (millicores)

```
# p50 CPU (millicores) over 14 days
quantile_over_time(0.50,
  rate(container_cpu_usage_seconds_total{
    namespace="<namespace>",
    pod=~"<pod-name-prefix>.*",
    container!="POD",
    container!=""
  }[5m])[14d:5m]
) * 1000

# p95 CPU (millicores) over 14 days
quantile_over_time(0.95, ...) * 1000

# p99 CPU (millicores) over 14 days
quantile_over_time(0.99, ...) * 1000
```

Replace `<namespace>` and `<pod-name-prefix>` with your values. Adjust the window (`14d`) and step (`5m`) as needed. For a Deployment or StatefulSet, aggregate across all replica pods by including `pod=~"<deployment-name>-.*"`.

### Memory working set per pod (MiB)

```
# p95 memory working set (MiB) over 14 days
quantile_over_time(0.95,
  container_memory_working_set_bytes{
    namespace="<namespace>",
    pod=~"<pod-name-prefix>.*",
    container!="POD",
    container!=""
  }[14d:5m]
) / 1048576
```

Use `container_memory_working_set_bytes` (not `container_memory_usage_bytes`) because the working set excludes file-backed pages that can be evicted by the kernel without causing an OOMKill. This gives a more accurate picture of the memory the container actually needs to retain.

## Google Cloud Monitoring (GKE)

Use the Cloud Monitoring Metrics Explorer or the Metrics Query Language (MQL) console.

Relevant metric types:

- CPU: `kubernetes.io/container/cpu/request_utilization` (fraction of requested CPU used) or `kubernetes.io/container/cpu/core_usage_time` (cumulative core-seconds)
- Memory: `kubernetes.io/container/memory/request_utilization` (fraction of requested memory used) or `kubernetes.io/container/memory/used_bytes`

To compute p95 over 14 days using MQL:

```
fetch k8s_container
| metric 'kubernetes.io/container/memory/used_bytes'
| filter (resource.namespace_name == '<namespace>'
      && resource.container_name != 'POD')
| group_by [resource.pod_name, resource.container_name],
    percentile(value.used_bytes, 95)
| within 14d
```

Divide the result by 1,048,576 to convert bytes to MiB before passing to the skill.

## Azure Monitor (AKS)

Use Container Insights or the Azure Monitor Metrics blade.

Relevant metrics:

- CPU: `cpuUsageNanoCores` (container CPU usage in nanocores; divide by 1,000,000 to convert to millicores)
- Memory: `memoryWorkingSetBytes` (working set in bytes; divide by 1,048,576 for MiB)

To query p95 over 14 days using Azure Monitor Logs (KQL):

```kql
InsightsMetrics
| where Namespace == "container.azm.ms/cpuUsageNanoCores"
| where Tags contains '"namespace":"<namespace>"'
| summarize p95 = percentile(Val, 95) by bin(TimeGenerated, 5m)
| where TimeGenerated > ago(14d)
| summarize p95_cpu_mc = percentile(p95, 95) / 1000000
```

## OpenCost (in-cluster)

If OpenCost is deployed, it exposes a REST API that returns pre-computed allocation data including max usage over a window. The skill cannot call this API directly, but the caller can retrieve the data using:

```
curl -G http://<opencost-service>:9003/allocation \
  --data-urlencode 'window=14d' \
  --data-urlencode 'aggregate=pod' \
  --data-urlencode 'namespace=<namespace>'
```

From the response, extract `cpuCoreRequestAverage`, `cpuCoreUsageAverage`, `ramByteRequestAverage`, and `ramByteUsageAverage`. Note: OpenCost exposes averages, not percentiles. If percentile data is required, use Prometheus directly.

## Vertical Pod Autoscaler (VPA) recommendations as a starting point

If VPA is deployed in recommendation mode (not auto), it exposes `LowerBound`, `Target`, `UpperBound`, and `UncappedTarget` for CPU and memory. These can serve as an approximate p50-to-p99 range:

- `LowerBound` ≈ p50 equivalent (conservative minimum)
- `Target` ≈ p95 equivalent (VPA recommendation)
- `UpperBound` ≈ p99 equivalent (headroom ceiling)

These are VPA's internal heuristics, not directly equivalent to Prometheus percentiles. Label them as `assumed` when using VPA output as a substitute for percentile metrics.

To retrieve VPA recommendations without modifying the cluster:

```
# Output from: kubectl describe vpa <vpa-name> -n <namespace>
# Paste the "Container Recommendations:" block into the skill input
```