{
  "schema_version": 1,
  "skill_name": "kubernetes",
  "evals": [
    {
      "id": "crashloopbackoff-diagnosis",
      "prompt": "A pod in my cluster is in CrashLoopBackOff: it starts and dies every few seconds. I ran kubectl get pods and see the state but nothing else. How do I diagnose this systematically, from the fastest checks to the deeper ones?",
      "expected_output": "A systematic CrashLoopBackOff diagnosis ordered by evidence value: describe the pod (kubectl describe pod) to see events, restart counts, and why it is being killed; read the container logs (kubectl logs with the previous container flag for the dying attempt) for the actual error; distinguish the failure classes — application error at startup (bad config, missing env var, crash in code), readiness or liveness probe failures killing a healthy app (probe path, port, or timeout wrong), resource limits being exceeded (OOMKilled in the container status), and image or command problems (imagePullBackOff, wrong entrypoint). The response then maps each class to its fix: fix the config or code for app errors, correct the probe definition for probe failures, raise limits or fix the memory profile for OOM, and check image tags and pull secrets for image errors. It prescribes verifying the fix by watching the pod reach Running and Ready.",
      "assertions": [
        "Diagnosis starts with describe and logs, including the previous container's logs",
        "Failure classes are distinguished: app crash, probe failures, OOM, image problems",
        "Each class is mapped to its specific fix",
        "Probe-related deaths are identified as a distinct class where the app may be healthy",
        "Verification is by watching the pod reach Running and Ready"
      ]
    },
    {
      "id": "rbac-networkpolicy-design",
      "prompt": "I am deploying a three-tier app: frontend, API, and database. I want least-privilege access: the frontend may talk only to the API, the API only to the database, and nothing external may reach the database. How do I implement this with Kubernetes RBAC and NetworkPolicy?",
      "expected_output": "A design that uses NetworkPolicy as the primary isolation mechanism and RBAC for control-plane access: the response defines a deny-by-default posture with policies that allow the specific flows (frontend to API on the API port, API to database on the database port) using pod selectors and ports, and a policy for the database that only the API pods can reach. For RBAC it specifies the identities and roles: separate service accounts per tier, role bindings scoped to what each workload needs, and the principle that pods get credentials only through their own service accounts. The response explains the practical gotchas: NetworkPolicy is enforced by the CNI (default-deny requires explicit policies, and a namespace with no policies allows all), ingress and egress policy fields work independently, and selector-based policy must match the actual pod labels. It prescribes verifying with a connectivity test between tiers and from outside.",
      "assertions": [
        "NetworkPolicy is used for data-plane isolation with a deny-by-default posture",
        "Policies specify exact pod selectors and ports for each allowed flow",
        "RBAC uses per-tier service accounts with scoped role bindings",
        "The response explains that NetworkPolicy enforcement depends on the CNI and default behavior",
        "Verification via connectivity tests between tiers and from outside is prescribed"
      ]
    },
    {
      "id": "ingress-routing-troubleshoot",
      "prompt": "Traffic to my service works when I port-forward but returns 503 through the Ingress. The ingress controller is running. What should I check to find where the path breaks?",
      "expected_output": "A routing diagnosis that walks the path layer by layer: check the ingress resource itself (host and path rules match the request, the service name and port in the ingress backend are correct, annotations are valid), check the service (selector matches the pod labels, the targetPort exists, endpoints are populated — a service with no endpoints returns 503), and check the controller (ingress class matches the controller, the controller can reach the pods). The response distinguishes 503 from 404 (503 means the ingress found the route but the backend was unreachable, pointing at service endpoints, while 404 points at the routing rules), and it explains the port-forward-works-but-ingress-fails pattern: the service selector or namespace mismatch is the usual culprit since port-forward bypasses the service. It prescribes checking kubectl get endpoints as the fastest discriminator.",
      "assertions": [
        "The diagnosis walks ingress resource, service, and controller layers in order",
        "The 503-versus-404 distinction is explained and used to narrow the cause",
        "Service selector and endpoint population are checked as the prime suspect",
        "The port-forward-works-but-ingress-fails pattern is explained",
        "kubectl get endpoints is prescribed as the fast discriminator"
      ]
    },
    {
      "id": "upgrade-planning",
      "prompt": "We run a self-managed cluster on k3s with a few production workloads and are several minor versions behind. I want to plan an upgrade that does not take down the workloads. What does a safe upgrade plan look like?",
      "expected_output": "An upgrade plan that treats version drift as the primary risk and sequencing as the control: the response starts by inventorying the current versions (server, kubelet, and the client tools) and the Kubernetes minor-version skew policy, checks the target version's deprecations against the workloads' API usage (verify the resource API versions the manifests use are still served), and upgrades in controlled stages: back up etcd state first, upgrade one node or a non-production cluster as a rehearsal, then roll the control plane and worker nodes, draining nodes before upgrades and uncordoning after. It includes the rollback path (restore backup, or downgrade within supported bounds) and verification at each stage: node versions, workload health, and the API-version compatibility checks. It also flags the common failure: upgrading the control plane without checking deprecated APIs breaks workloads after the upgrade, and using the in-place k3s install script without reading the release notes.",
      "assertions": [
        "The plan inventories versions and checks the skew policy before upgrading",
        "Deprecated API usage in the workloads' manifests is checked against the target version",
        "Upgrades are staged with etcd backup, a rehearsal environment, and drain-then-upgrade node rolling",
        "A rollback path is defined for each stage",
        "Verification at each stage covers node versions, workload health, and API compatibility"
      ]
    },
    {
      "id": "autoscaling-rightsizing",
      "prompt": "Our API pods run at 30% CPU average but the cluster sometimes spikes and the HPA scales to 20 replicas that mostly sit idle. I want autoscaling that matches demand without waste. How should I configure it?",
      "expected_output": "An autoscaling and rightsizing design: the response starts with the workloads' actual profile — measure sustained CPU and memory percentiles over a representative period (not the average), set resource requests from the p95/p99 so the scheduler reserves honestly without over-reserving, and configure the HPA on the metric that reflects demand (CPU utilization relative to requests, or a custom metric such as request latency or queue depth if CPU is a poor proxy). It explains the HPA tuning knobs: min/max replicas chosen from the measured demand curve, target utilization set so it scales before latency degrades but not on noise, and the scaling-delay parameters (stabilization window, cooldown) that prevent the thrash between 5 and 20 replicas. The response also covers the interplay with cluster autoscaling: the HPA's ceiling must be reconcilable with node capacity, and idle spike behavior is addressed by the utilization target and scale-down stabilization rather than more replicas.",
      "assertions": [
        "Rightsizing starts from measured percentile CPU and memory, with requests set from the high percentile",
        "The HPA metric is chosen to match demand, with custom metrics considered over CPU alone",
        "Min/max replicas and target utilization come from the measured demand curve",
        "Stabilization windows are used to prevent replica thrash",
        "The HPA ceiling and cluster autoscaling are reconciled"
      ]
    }
  ]
}

# API, discovery, and versioning

## Discovery first

Use the target API server as the authority:

```sh
kubectl version -o json
kubectl api-resources -o wide
kubectl api-versions
kubectl explain deployment.spec --recursive
kubectl get crd
```

Do not infer that an API exists because a YAML example exists. Validate with `api-resources`, `api-versions`, `explain`, and server-side dry-run.

## Resource and scope rules

Every request has a group/version/resource, and resources are either namespaced or cluster-scoped. Subresources such as `status`, `scale`, `logs`, `exec`, and `portforward` have distinct behavior. Discover the resource and scope before constructing a URL or command.

## Server-side apply

Server-side apply is stable and tracks field ownership. Use an explicit field manager. A conflict is a coordination signal, not permission to force overwrite. Resolve by inspecting `metadata.managedFields`, changing intent, or obtaining explicit approval for `--force-conflicts`.

```sh
kubectl apply --server-side --field-manager=agent-kubernetes -f manifest.yaml --dry-run=server
kubectl diff -f manifest.yaml
kubectl apply --server-side --field-manager=agent-kubernetes -f manifest.yaml
```

## Deprecation

GA, beta, and alpha APIs have different stability guarantees. Detect deprecated API use from API warnings, audit annotations, metrics such as `apiserver_requested_deprecated_apis`, and release/deprecation guides. Never use `v1beta1` because it appears in an old tutorial.

## CRDs and operators

Separate these checks:

1. CRD is registered and served.
2. Custom object passes schema/admission.
3. Controller/operator is installed.
4. Controller is healthy and reconciling.
5. Object conditions represent the desired state.

A CRD object can be accepted while no controller exists to act on it. Check controller logs, events, owner references, conditions, and served versions.

## Current feature-state discipline

The 2026-07-11 research pass found version-sensitive features including streaming lists (1.34 beta), Pod-level resource specification (1.34 beta), and newer alpha APIs. Do not operationalize an alpha/beta feature without checking the target server's feature gates and the current official page.

## Sources

- https://kubernetes.io/docs/reference/using-api/api-concepts/
- https://kubernetes.io/docs/reference/using-api/server-side-apply/
- https://kubernetes.io/docs/reference/using-api/deprecation-policy/
- https://kubernetes.io/docs/reference/using-api/deprecation-guide/
- https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
- https://kubernetes.io/releases/

# Backup and restore

## Choose the state to protect

- API/resource state: etcd or the distribution datastore.
- Persistent volume data: CSI snapshots, provider snapshots, application backup, or Velero.
- Cluster identity/configuration: PKI, kubeconfigs, distribution config, manifests, provider resources.
- Observability/policy configuration: audit policy, admission/webhook configuration, monitoring rules.

A YAML export is not an etcd backup. An etcd snapshot is not an application-consistent PV backup.

## Preflight

1. Identify the distribution and datastore.
2. Capture version and snapshot compatibility.
3. Check datastore health, quorum, disk space, and encryption/key custody.
4. Create a backup with a timestamp and retention policy.
5. Verify metadata/checksum and test restore in a disposable environment.

## Distribution boundary

Use the current official procedure for `kubeadm`/etcd, k3s `etcd-snapshot`, RKE2 backup/restore, k0s backup, Talos `talosctl`, or managed-provider backup. Do not improvise a control-plane restore command from a blog post. The exact k3s/k0s restore syntax remains version-sensitive and must be refreshed before execution.

## Velero and CSI

Velero and CSI VolumeSnapshots are ecosystem components, not built-in guarantees. Confirm plugin/provider support, credentials, snapshot class, namespace/resource filters, PV data mode, and restore hooks. Test an actual restore, including application readiness and external dependencies.

## Sources

- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
- https://kubernetes.io/docs/concepts/storage/volume-snapshots/
- https://velero.io/docs/latest/
- https://docs.k3s.io/backup-restore
- https://docs.rke2.io/backup_restore/
- https://docs.k0sproject.io/stable/backup/
- https://docs.talos.dev/

# Native CLI reference

## Context and discovery

The bundled wrapper places global options before the subcommand:

```sh
scripts/k8s-cli --json get pods --namespace default
```

Native examples:

```sh
kubectl config get-contexts
kubectl config current-context
kubectl version -o json
kubectl api-resources -o wide
kubectl api-versions
kubectl get nodes -o wide
kubectl get --raw='/readyz?verbose'
```

## Bounded inspection

```sh
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp
kubectl logs POD -n NS --all-containers --tail=200 --since=1h
kubectl describe pod POD -n NS
```

Always add an explicit namespace, context, selector, tail, since, or timeout where the operation allows it.

## Validation and mutation

```sh
kubectl diff -f manifest.yaml
kubectl apply --server-side --field-manager=agent-kubernetes --dry-run=server -f manifest.yaml
kubectl apply --server-side --field-manager=agent-kubernetes -f manifest.yaml
kubectl delete -f manifest.yaml --dry-run=server
kubectl rollout status deployment/NAME -n NS --timeout=120s
kubectl wait --for=condition=available deployment/NAME -n NS --timeout=120s
```

## Output safety

Prefer `-o json` for machine parsing. Use `-o name`, `-o wide`, JSONPath, or custom columns for bounded views. Never use `kubectl config view --raw`, `get secret -o yaml`, or unbounded `logs -f` in chat output.

## Sources

- https://kubernetes.io/docs/reference/kubectl/
- https://kubernetes.io/docs/reference/kubectl/quick-reference/
- https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

# Advanced debugging

## Ephemeral containers

Use `kubectl debug` when a running Pod needs inspection without restarting the application. Select an appropriate debug profile, target a container only when necessary, and treat the resulting shell/output as potentially sensitive.

```sh
kubectl debug pod/NAME -n NAMESPACE -it --image=busybox:1.36 --target=CONTAINER
kubectl debug pod/NAME -n NAMESPACE --copy-to=NAME-debug --share-processes --image=busybox:1.36
```

Do not add privileged capabilities, host namespaces, or host mounts as a reflex. Explain why the chosen profile requires them.

## Must-gather pattern

Collect context/version, readiness, nodes, system workloads, CRDs, events, resource conditions, selected logs, policy objects, and provider/distribution markers. Bound every collection, omit Secrets and raw kubeconfig, and write a manifest of commands and timestamps.

## Sources

- https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/
- https://kubernetes.io/docs/tasks/debug/debug-cluster/
- https://kubernetes.io/docs/reference/kubectl/

# Distribution and provider overlays

Use this matrix as routing, not as a timeless release catalog. Refresh release/support fields before relying on them.

| Family | Distinguishing lifecycle/defaults | Load before |
|---|---|---|
| upstream/kubeadm | static control-plane Pods, etcd, install CNI separately, `kubeadm upgrade` | bootstrap, HA, upgrade |
| k3s | single binary, SQLite/embedded etcd/Kine, Flannel, Traefik, ServiceLB, local-path storage, `k3s etcd-snapshot` | any self-managed edge or lightweight operation |
| RKE2 | Rancher enterprise distribution, static Pods, etcd, Canal/Calico/Flannel, Rancher lifecycle | FIPS, Rancher, upgrade |
| MicroK8s | snap packaging, dqlite HA, addons, `microk8s` wrapper | snap/addon/HA operation |
| k0s | single binary, controller/worker separation, SQLite/etcd/Kine, k0sctl | k0sctl, backup, upgrade |
| Talos | immutable API-only OS, no SSH/shell/package manager, `talosctl`, etcd | node-level operations |
| OpenShift/OKD | Operators, CVO, OAuth, `oc`, OVN/Router | OpenShift-specific lifecycle/policy |
| Rancher | management platform, not a distribution; manages/imports downstream clusters | Fleet, Rancher auth, provisioning |
| kind | Docker/Podman node containers, disposable test clusters | local integration tests |
| Minikube | local single-node VM/container cluster, driver/addon lifecycle | local development |
| EKS | AWS-managed control plane, VPC CNI, EKS API, IRSA/Pod Identity, Standard/Auto Mode | AWS identity, load balancer, upgrade |
| AKS | Azure-managed control plane, Azure CNI/Cilium, Workload ID, Standard/Automatic | Azure identity, upgrade, node pools |
| GKE | Google-managed control plane, Standard/Autopilot, release channels, Workload Identity | GCP identity, upgrade, networking |

## Detection

Use explicit operator input first. Then inspect server version, node labels, system namespaces, component names, and provider CLI context. Detection is a hypothesis. Confirm against official docs and cluster evidence before applying defaults.

## Version baseline checked 2026-07-11

- Kubernetes project: maintained minor branches 1.36, 1.35, 1.34.
- k3s release: `v1.36.2+k3s1`.
- RKE2 release: `v1.36.2+rke2r1`.
- k0s release: `v1.36.2+k0s.0`.
- Talos release: `v1.13.6`.
- kind release: `v0.32.0`.
- Minikube release: `v1.38.1`.

These values are observations, not promises. Use the official release pages below for refresh.

## k3s first-class notes

Server nodes run control-plane and datastore components; agents run kubelet, runtime, and CNI. Single-server SQLite, HA embedded etcd, and external datastore/Kine topologies differ. K3s bundles CoreDNS, Traefik, ServiceLB, local-path storage, and networking services. Upgrade servers before agents and use the current k3s upgrade/backup procedure.

## Managed provider boundary

Do not inspect or mutate managed control-plane processes as if they were kubeadm static Pods. Use the provider API/CLI for cluster lifecycle and Kubernetes APIs for workloads. Provider identity and load-balancer resources require provider credentials and must be validated outside the cluster too.

## Sources

- https://kubernetes.io/docs/reference/setup-tools/kubeadm/
- https://docs.k3s.io/architecture
- https://docs.k3s.io/datastore
- https://docs.k3s.io/upgrades
- https://docs.rke2.io/architecture
- https://docs.rke2.io/upgrade
- https://microk8s.io/docs
- https://docs.k0sproject.io/stable/architecture/
- https://docs.talos.dev/
- https://docs.openshift.com/container-platform/latest/
- https://docs.rancher.com/rancher/
- https://kind.sigs.k8s.io/
- https://minikube.sigs.k8s.io/docs/
- https://docs.aws.amazon.com/eks/latest/userguide/
- https://learn.microsoft.com/en-us/azure/aks/
- https://cloud.google.com/kubernetes-engine/docs/
- https://kubernetes.io/releases/

# Networking, DNS, Services, and Gateway API

## Layered diagnosis

1. Pod IP and container listener.
2. Service selector and EndpointSlice membership.
3. Cluster DNS resolution from the calling namespace.
4. NetworkPolicy/CNI enforcement.
5. Ingress or Gateway API route and controller conditions.
6. Provider or distribution load balancer.
7. External DNS, TLS, firewall, and client path.

```sh
kubectl get svc,endpointslice -n NAMESPACE
kubectl describe svc NAME -n NAMESPACE
kubectl get networkpolicy -A
kubectl get ingress,gateway,gatewayclass,httproute -A
kubectl run dns-debug --rm -it --restart=Never --image=busybox:1.36 -- nslookup SERVICE.NAMESPACE.svc.cluster.local
```

NetworkPolicy has no effect unless the network plugin enforces it. Policies are additive; ingress and egress isolation are evaluated separately, and both source egress and destination ingress must allow a connection.

## Ingress and Gateway API

Ingress `networking.k8s.io/v1` remains important for compatibility. Gateway API is a separate SIG project with its own CRDs and controller implementations. Prefer Gateway API for new designs only after checking the installed Gateway API CRDs, controller support, and provider/distribution integration. Never assume `Gateway`, `GatewayClass`, or `HTTPRoute` exists because the Kubernetes API server is healthy.

## Sources

- https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
- https://kubernetes.io/docs/concepts/services-networking/service/
- https://kubernetes.io/docs/concepts/services-networking/network-policies/
- https://kubernetes.io/docs/concepts/services-networking/ingress/
- https://gateway-api.sigs.k8s.io/docs/

# Nodes, disruption, and reliability

## Node maintenance

```sh
kubectl get nodes -o wide
kubectl describe node NODE
kubectl cordon NODE
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=10m
kubectl uncordon NODE
```

Cordon prevents new scheduling; drain evicts workloads and is constrained by PodDisruptionBudgets. Review PDBs, daemonsets, local storage, topology, capacity, and workload replicas before draining. Verify the node is empty or only has intentionally retained Pods, then verify replacement workloads and external availability. Do not add `--force`, bypass PDBs, or remove finalizers by default.

## Pressure and conditions

Inspect `Ready`, memory/disk/PID pressure, taints, allocatable capacity, kubelet/runtime evidence, and provider node events. Kubelet eviction and API-initiated eviction are different paths. Resource pressure may be a node symptom of workload requests, image garbage collection, filesystem, or runtime failure.

## PDBs

Use `policy/v1` with one of `minAvailable` or `maxUnavailable`, a selector matching the intended workload, and an explicit disruption expectation. A PDB is not a guarantee against involuntary failure and can block maintenance if replicas or selectors are wrong.

## Sources

- https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
- https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/
- https://kubernetes.io/docs/concepts/scheduling-eviction/pod-disruption-budgets/
- https://kubernetes.io/docs/concepts/architecture/nodes/

# Observability and metrics

## Minimal health evidence

```sh
kubectl get --raw='/readyz?verbose'
kubectl get nodes
kubectl get pods -A
kubectl events -A --types=Warning
kubectl top nodes
kubectl top pods -A --sort-by=memory
```

`kubectl top` depends on the resource metrics pipeline and may be unavailable even when the API server is healthy. Distinguish missing metrics, stale metrics, and a measured zero.

## Logging

Use bounded `kubectl logs` with `--tail`, `--since`, and `--previous`. Cluster-level logging is normally implemented by a node agent, sidecar, or application-direct path. Container log rotation and runtime behavior are node/distribution concerns.

## Events and audit

Events are short-lived diagnostic evidence and should be correlated by involved object, reason, type, count, and timestamp. Audit logs answer who called the API and at which stage, but require a configured policy and backend. Keep audit policy changes separate from ordinary workload changes.

## Metrics selection

Use API-server deprecation metrics to locate deprecated API use. Use controller/node/workload metrics for health and capacity. Prometheus Operator, kube-state-metrics, and provider monitoring are ecosystem overlays; route to their official docs rather than pretending they are Kubernetes core.

## Audit policy boundary

Audit logging requires an API-server policy and a backend. The policy levels are `None`, `Metadata`, `Request`, and `RequestResponse`; the more detailed levels increase sensitivity and cost. Start from the minimum evidence needed, redact or restrict request bodies, and treat audit-policy changes as control-plane changes rather than ordinary workload configuration.

Audit records have `RequestReceived`, `ResponseStarted`, `ResponseComplete`, and `Panic` stages. The policy is ordered: the first matching rule wins, and an empty rules list is invalid. Backends are currently log files or webhooks. Audit increases API-server memory use, so measure and bound high-volume rules.

## Sources

- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/
- https://kubernetes.io/docs/concepts/cluster-administration/logging/
- https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
- https://kubernetes.io/docs/reference/instrumentation/metrics/
- https://kubernetes.io/docs/concepts/security/secrets-good-practices/

# Reliability, upgrades, backup, and observability

## Inventory before change

Record distribution/provider, control-plane and node versions, contexts, API resources, CRDs, CNI/CSI, admission, system workloads, nodes, quotas, and representative application health. Use the inventory template and redact credentials.

## Upgrade pattern

1. Confirm support window and version skew from current official docs.
2. Scan deprecated API usage and operator/webhook compatibility.
3. Verify backups and restore procedure, not only backup creation.
4. Check PDBs, capacity, quotas, maintenance windows, and external dependencies.
5. Upgrade according to the distribution/provider overlay, never generic Kubernetes memory.
6. Verify API discovery, nodes, system workloads, controllers, representative workloads, storage, routes, and observability.

Upgrades are not reversible by default. A provider may roll back infrastructure while a self-managed distribution may require restore or re-provisioning.

## Backups

Back up the actual stateful substrate: etcd or the distribution's datastore, PKI/configuration, manifests, persistent data, and provider resources. Test restoration into a disposable environment. An etcd snapshot alone is not automatically an application-consistent backup.

## Observability

Collect API audit, events, control-plane/component logs where accessible, node conditions, workload logs, metrics, HPA signals, and provider control-plane events. Distinguish “metrics unavailable” from “metric is zero.”

## Sources

- https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/
- https://kubernetes.io/docs/setup/release/version-skew-policy/
- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
- https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
- https://docs.k3s.io/backup-restore
- https://docs.rke2.io/backup_restore/
- https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
- https://learn.microsoft.com/en-us/azure/aks/upgrade-aks-cluster
- https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-upgrades

# Policy selection

Choose the smallest policy mechanism that satisfies the requirement:

| Need | Prefer | Boundary |
|---|---|---|
| Basic Pod hardening | Pod Security Admission | Namespace labels and workload compatibility |
| Quotas/defaults | ResourceQuota, LimitRange, built-in admission | Namespace/resource scope |
| Simple declarative validation/mutation | ValidatingAdmissionPolicy / MutatingAdmissionPolicy with CEL | Check target version and parameter support |
| Rego-based enterprise policy and audit | Gatekeeper | Install and maintain CRDs/webhooks; provider-specific |
| YAML-native validate/mutate/generate/verify | Kyverno | Install and maintain policy reports/webhooks |
| Image provenance/signatures | Sigstore/cosign plus an admission integration | Registry, identity, and key policy |

Policy is additive and can conflict operationally. Inventory existing policies, webhooks, failure policies, and exemptions before adding another enforcement layer. Test in audit/warn mode where supported and verify rejected objects plus user-facing diagnostics.

## Sources

- https://kubernetes.io/docs/concepts/security/pod-security-admission/
- https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
- https://open-policy-agent.github.io/gatekeeper/website/docs/
- https://kyverno.io/docs/
- https://docs.sigstore.dev/cosign/overview/

# Safety gates

## Read-only operations

Context, version, discovery, get, describe, events, logs, auth checks, and rollout status are read-only in the Kubernetes API, although logs/exec can expose sensitive data. Bound output and redact values.

## Reversible mutations

Apply, scale, rollout restart, and non-destructive patches can still cause outage. Require:

- explicit context and namespace
- manifest or target scope
- diff/server dry-run
- field manager
- timeout and rollback/verification plan
- explicit `--yes` to enact through `k8s-cli`

## Destructive or high-risk mutations

Delete, drain, node cordon, finalizer removal, force-conflicts, RBAC broadening, admission changes, Secret changes, datastore restore, cluster reset, and upgrades require a human-confirmed scope. The CLI refuses these by default.

## Verification gate

After a mutation, verify:

1. command exit status
2. API object conditions
3. events and controller state
4. rollout/Pod readiness
5. Service/EndpointSlice or route
6. external health check when relevant

A successful command is not a successful operation until the relevant boundary is healthy.

# Scheduling, resources, scaling, and storage

## Pending Pods

Inspect, in order: Pod events, node readiness/pressure, resource requests, taints/tolerations, selectors and affinity, topology spread, quotas, priorities, and admission policy. `Pending` is a symptom, not a root cause.

```sh
kubectl describe pod NAME -n NAMESPACE
kubectl get nodes -o wide
kubectl describe node NAME
kubectl get resourcequota,limitrange -n NAMESPACE
```

Requests influence scheduling. CPU limits throttle; memory-limit violations can result in OOM kills. Do not prescribe resource changes without observing usage, requests, limits, and node capacity.

## HPA

HPA needs a scalable target and a functioning metrics API. Missing resource requests can make utilization undefined. Check HPA conditions, target reference, current/desired metrics, and `metrics.k8s.io` availability before changing replicas.

HPA, VPA, and Cluster Autoscaler are separate control loops. HPA changes replica count, VPA changes resource recommendations or requests, and Cluster Autoscaler changes node capacity. Combining them without a signal/ownership design can cause oscillation, request inflation, evictions, or slow reaction. Treat VPA as recommendation-only until its eviction and stateful-workload behavior is understood; treat Cluster Autoscaler as a provider/distribution overlay.

```sh
kubectl get hpa -A
kubectl describe hpa NAME -n NAMESPACE
kubectl top pods -n NAMESPACE
kubectl get --raw='/apis/metrics.k8s.io/v1beta1/namespaces/NAMESPACE/pods'
```

Do not diagnose “HPA is not scaling” from replica count alone. Check metrics availability, target requests, HPA conditions, stabilization/behavior policies, pending Pods, and node-group capacity.

## Persistent storage

Separate PV, PVC, StorageClass, CSI driver, attachment/mount, and application filesystem evidence:

```sh
kubectl get pvc,pv,storageclass -A
kubectl describe pvc CLAIM -n NAMESPACE
kubectl get volumeattachment -o wide
kubectl get pods -A -l app=CSI-DRIVER
```

A Pending PVC may mean no matching class/capacity. A Terminating PVC may be protected while still in use. A Bound PVC does not prove the application mounted or can write the volume.

## Sources

- https://kubernetes.io/docs/concepts/scheduling-eviction/
- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
- https://kubernetes.io/docs/concepts/storage/volumes/
- https://kubernetes.io/docs/concepts/storage/persistent-volumes/
- https://kubernetes.io/docs/concepts/storage/storage-classes/
- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
- https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

# Security, identity, and policy

## Access

RBAC permissions are additive. Role is namespaced; ClusterRole is cluster-scoped but can be bound into a namespace. Prefer the narrowest subject, namespace, resource, and verb set. Check effective authorization before changing bindings:

```sh
kubectl auth can-i VERB RESOURCE --namespace NAMESPACE
kubectl auth can-i --list --namespace NAMESPACE
kubectl get role,rolebinding -n NAMESPACE
kubectl get clusterrole,clusterrolebinding
```

Never print `kubeconfig --raw`, token fields, or Secret data. Redact object output before sharing.

## Pod security

Pod Security Standards define Privileged, Baseline, and Restricted profiles. Namespace labels and admission behavior are version-sensitive. Before changing enforcement, audit existing workloads and test in warn/audit modes where supported.

## Admission and encryption

Admission webhooks can block or mutate every matching API request. Enumerate webhook configurations, inspect failure policies/timeouts, and consider built-in CEL admission for simple validation. Encryption at rest is not automatic merely because Secrets exist; inspect the control-plane/provider configuration.

## Network and workload identity

NetworkPolicy requires an enforcing plugin. Managed providers add identity layers: EKS IRSA/Pod Identity, AKS Workload ID/OIDC/Azure RBAC, and GKE Workload Identity Federation. These are not portable Kubernetes instructions.

## Privilege-escalation checks

Treat these permissions as high risk during RBAC review: `escalate`, `bind`, `impersonate`, `nodes/proxy`, `serviceaccounts/token`, CSR approval, admission webhook configuration, and broad access to Secrets. `list` and `watch` on Secrets can expose their contents. Creating workloads can indirectly grant access to mounted Secrets and service-account credentials. Review `system:masters` separately because it bypasses ordinary RBAC rules.

```sh
kubectl auth can-i --list --as=system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT -n NAMESPACE
kubectl get clusterrolebindings -o json
kubectl auth reconcile -f rbac.yaml --dry-run=client
```

Never broaden a RoleBinding merely to make a failing workload work. Identify the denied verb/resource, grant the smallest namespace-scoped permission, and verify with `auth can-i`.

## PSA rollout

Pod Security Admission is namespace-label driven. A safer migration is `warn` and `audit` first, then `enforce` after violations are remediated. Pin the profile version only after checking the target cluster’s supported profile version.

```yaml
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
# Add enforce only after reviewing violations.
```

## Secrets and image supply chain

Kubernetes Secrets are API objects, not a complete secrets-management or encryption-at-rest strategy. Secret values are base64-encoded, not encrypted, unless encryption at rest is configured. Avoid putting secret values in manifests, shell history, logs, or agent output. Restrict `get`, `list`, and `watch` access, disable automatic service-account token mounting when a workload does not need the API, and use short-lived workload identity where the provider supports it. External Secret Store CSI providers and admission/image-signature policy are ecosystem overlays that require separate validation. Verify image digest, registry trust, architecture, pull credentials, and admission policy separately.

## Sources

- https://kubernetes.io/docs/reference/access-authn-authz/rbac/
- https://kubernetes.io/docs/concepts/security/pod-security-standards/
- https://kubernetes.io/docs/concepts/security/pod-security-admission/
- https://kubernetes.io/docs/concepts/cluster-administration/admission-webhooks-good-practices/
- https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
- https://kubernetes.io/docs/concepts/security/rbac-good-practices/
- https://kubernetes.io/docs/concepts/security/secrets-good-practices/
- https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html
- https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity

# Source index and maintenance contract

Research and verification date: 2026-07-11

## Primary sources

### Kubernetes core

- https://kubernetes.io/docs/
- https://kubernetes.io/releases/
- https://kubernetes.io/docs/setup/release/version-skew-policy/
- https://kubernetes.io/docs/reference/using-api/api-concepts/
- https://kubernetes.io/docs/reference/using-api/deprecation-policy/
- https://kubernetes.io/docs/reference/using-api/deprecation-guide/
- https://kubernetes.io/docs/reference/kubectl/
- https://kubernetes.io/docs/concepts/workloads/
- https://kubernetes.io/docs/concepts/scheduling-eviction/
- https://kubernetes.io/docs/concepts/storage/
- https://kubernetes.io/docs/concepts/services-networking/
- https://kubernetes.io/docs/concepts/security/
- https://kubernetes.io/docs/tasks/debug/
- https://kubernetes.io/docs/tasks/administer-cluster/

### Distribution and provider overlays

- https://docs.k3s.io/
- https://docs.rke2.io/
- https://microk8s.io/docs
- https://docs.k0sproject.io/stable/
- https://docs.talos.dev/
- https://docs.openshift.com/container-platform/latest/
- https://docs.okd.io/latest/
- https://docs.rancher.com/rancher/
- https://kind.sigs.k8s.io/
- https://minikube.sigs.k8s.io/docs/
- https://docs.aws.amazon.com/eks/latest/userguide/
- https://learn.microsoft.com/en-us/azure/aks/
- https://cloud.google.com/kubernetes-engine/docs/

### Ecosystem overlays

- https://gateway-api.sigs.k8s.io/docs/
- https://helm.sh/docs/
- https://velero.io/docs/latest/
- https://open-policy-agent.github.io/gatekeeper/website/docs/
- https://kyverno.io/docs/
- https://docs.sigstore.dev/cosign/overview/

## Verified release observations

| Product | Observation | Verification |
|---|---|---|
| Kubernetes | 1.36, 1.35, 1.34 maintained branches | Official release page scrape |
| k3s | `v1.36.2+k3s1` | GitHub release API, 2026-07-11 |
| RKE2 | `v1.36.2+rke2r1` | GitHub release API, 2026-07-11 |
| k0s | `v1.36.2+k0s.0` | GitHub release API, 2026-07-11 |
| Talos | `v1.13.6` | GitHub release API, 2026-07-11 |
| kind | `v0.32.0` | GitHub release API, 2026-07-11 |
| Minikube | `v1.38.1` | GitHub release API, 2026-07-11 |

## Refresh procedure

1. Run `scripts/refresh-version-matrix.sh`.
2. Re-scrape the official Kubernetes release, version-skew, deprecation, distribution, and provider pages.
3. Compare release values, defaults, feature-state labels, and upgrade instructions against this reference.
4. Record discrepancies in the project’s research reconciliation log before editing guidance.
5. Do not update a version number without its source URL, retrieval date, and support interpretation.
6. Re-run the CLI tests, skill validator, and any available disposable-cluster integration tests.

The skill intentionally excludes unsupported or unverified alpha-feature instructions from executable command guidance. A source can be listed for future research without becoming an operational recommendation.

# Evidence-first troubleshooting

## Universal loop

1. State the symptom and boundary: API, object admission, scheduling, startup, readiness, service, external route, storage, node, or control plane.
2. Capture context, versions, namespace, labels, owner references, conditions, events, and recent changes.
3. Form competing hypotheses.
4. Run the least invasive discriminating check.
5. Change one layer at a time.
6. Verify both Kubernetes state and the user-visible boundary.
7. Preserve a bounded report with commands, timestamps, exit codes, and redacted output.

## Common symptoms

- `ImagePullBackOff`: inspect Events, image name/tag, registry auth, node egress, architecture, and image policy.
- `CrashLoopBackOff`: inspect previous logs, termination reason, probes, config/Secret mounts, limits, and recent rollout.
- `Pending`: follow scheduling/resource/taint/quota evidence; do not simply increase replicas or delete Pods.
- `Running` but unavailable: inspect readiness, Service selectors, EndpointSlices, DNS, NetworkPolicy, route/controller status, and external load balancer.
- `Terminating`: inspect finalizers, owner/controller health, admission/webhook behavior, and whether the resource is still in use. Removing finalizers is destructive and requires explicit justification.
- Upgrade blocked: inspect deprecated API use, PDBs, webhooks, node readiness, quotas, provider/distribution upgrade rules, and version skew.

## Output discipline

Use `-o json` or `-o jsonpath` for machine parsing. Bound events and logs by namespace, selector, tail lines, and time. Never present an aggregator, cached page, or intermediate tool result as proof that a live endpoint is healthy.

## Sources

- https://kubernetes.io/docs/tasks/debug/debug-cluster/
- https://kubernetes.io/docs/tasks/debug/debug-application/
- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/
- https://kubernetes.io/docs/concepts/cluster-administration/logging/
- https://kubernetes.io/docs/concepts/cluster-administration/system-logs/

# Version skew and freshness

## Kubernetes baseline

The official release page checked 2026-07-11 reported maintained minor branches 1.36, 1.35, and 1.34, with approximately one year of patch support for 1.19 and newer. Recheck the page before making a support claim.

## Compatibility checks

Compare client, API server, kubelet, controller, node image, CRD, operator, webhook, CNI, CSI, and provider versions. Kubernetes version skew rules differ by component and release. The API server is authoritative for served APIs; provider support windows can be narrower or differently shaped than upstream support.

## Upgrade evidence

Before upgrade: deprecated API metrics/warnings, CRD served/storage versions, webhook compatibility, PDBs, capacity, backups, and provider/distribution procedure. After upgrade: discovery, nodes, system workloads, controllers, representative workloads, storage, routes, metrics, and alerts.

## Sources

- https://kubernetes.io/releases/
- https://kubernetes.io/docs/setup/release/version-skew-policy/
- https://kubernetes.io/docs/reference/using-api/deprecation-policy/
- https://kubernetes.io/docs/reference/using-api/deprecation-guide/

# Workloads and rollouts

## Workload selection

- Deployment: stateless replicated applications and rolling updates.
- StatefulSet: stable identity and persistent storage; inspect ordinal identity, PVCs, and update strategy.
- DaemonSet: one or more Pods per eligible Node; inspect taints, selectors, and update strategy.
- Job: run-to-completion; inspect completions, parallelism, backoff, deadline, and TTL.
- CronJob: scheduled Jobs; inspect schedule, concurrency policy, deadlines, and history limits.

Use the highest-level controller that expresses the intent. Do not hand-edit ReplicaSets or controller-owned Pods unless debugging demands it.

## Evidence collection

```sh
kubectl get deploy,sts,ds,job,cronjob -A -o wide
kubectl describe deployment NAME -n NAMESPACE
kubectl get pods -n NAMESPACE -l app=NAME -o wide
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp
kubectl rollout status deployment/NAME -n NAMESPACE --timeout=120s
```

`Running` means the Pod has been scheduled and containers are running. Readiness controls endpoint membership; liveness controls restart behavior; startup probes protect slow-starting applications. A rollout is not healthy until its conditions, Pods, events, and relevant endpoint agree.

## Safe rollout pattern

1. Render or inspect the intended change.
2. Run `kubectl diff` and `kubectl apply --dry-run=server`.
3. Apply with an explicit field manager.
4. Wait for rollout and inspect conditions/events.
5. Verify Service/EndpointSlice or Gateway/HTTP response as appropriate.
6. Roll back only after recording the failed revision and evidence.

## Sources

- https://kubernetes.io/docs/concepts/workloads/
- https://kubernetes.io/docs/concepts/workloads/pods/probes/
- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
- https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
- https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
- https://kubernetes.io/docs/concepts/workloads/controllers/job/
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

#!/usr/bin/env bash
set -euo pipefail
OUT=${1:-k8s-gather-$(date -u +%Y%m%dT%H%M%SZ)}
mkdir -p "$OUT"
run() { name=$1; shift; "$@" >"$OUT/$name.txt" 2>&1 || printf 'command failed: %s\n' "$*" >>"$OUT/$name.txt"; }
run context kubectl config current-context
run version kubectl version -o json
run readiness kubectl get --raw='/readyz?verbose'
run api-resources kubectl api-resources -o wide
run api-versions kubectl api-versions
run nodes kubectl get nodes -o wide
run system-pods kubectl get pods -A -o wide
run events kubectl events -A --types=Warning --sort-by=.lastTimestamp
run crds kubectl get crd
run storage kubectl get pvc,pv,storageclass -A
run webhooks kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
printf 'Collected bounded diagnostics in %s\n' "$OUT"
printf 'Secrets and raw kubeconfig were intentionally omitted.\n'

#!/usr/bin/env bash
# Refresh release observations for human review. This script intentionally does
# not edit references automatically: version claims need source reconciliation.
set -euo pipefail
command -v gh >/dev/null || { echo 'gh is required' >&2; exit 127; }
printf 'Research date: '; date -u +%Y-%m-%d
printf '\nKubernetes docs:\nhttps://kubernetes.io/releases/\n'
for repo in k3s-io/k3s rancher/rke2 k0sproject/k0s siderolabs/talos kubernetes-sigs/kind kubernetes/minikube; do
  gh release view --repo "$repo" --json name,publishedAt,url
 done
cat <<'EOF'

Review output against references/distributions.md and references/operations.md before editing.
EOF

#!/usr/bin/env bash
set -euo pipefail
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
CLI="$ROOT/scripts/k8s-cli"
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
cat > "$TMP/kubectl" <<'FAKE'
#!/usr/bin/env bash
set -u
printf '{"argv":['
first=1
for arg in "$@"; do [ "$first" = 1 ] || printf ','; first=0; printf '%s' "$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$arg")"; done
printf '],"kind":"List","items":[]}'
FAKE
chmod +x "$TMP/kubectl"
run() { KUBECTL="$TMP/kubectl" "$CLI" --json "$@"; }

out=$(run get pods --namespace test)
python3 -c 'import json,sys; d=json.loads(sys.argv[1]); assert d["ok"] and d["result"]["kind"] == "List"' "$out"

if run delete pod/example --namespace test >/dev/null 2>&1; then
  echo 'delete must require --yes' >&2; exit 1
fi
run delete pod/example --namespace test --dry-run >/dev/null
run delete pod/example --namespace test --yes >/dev/null

if run raw /api/../secrets >/dev/null 2>&1; then
  echo 'raw traversal must be rejected' >&2; exit 1
fi

echo 'k8s-cli tests: PASS'

#!/usr/bin/env bash
set -euo pipefail
fail=0
kubectl get --raw='/readyz' >/dev/null || fail=1
if kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -qv $'\tTrue$'; then
  echo 'one or more nodes are not Ready' >&2; fail=1
fi
if kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}' | grep -qv $'\tRunning$'; then
  echo 'one or more kube-system pods are not Running' >&2; fail=1
fi
if [ "$fail" -eq 0 ]; then echo 'cluster health: PASS'; else echo 'cluster health: FAIL'; fi
exit "$fail"

{
  "schema": "kubernetes-cluster-inventory/v1",
  "captured_at": "",
  "context": {"name": "", "server": "", "namespace": ""},
  "versions": {"client": "", "server": ""},
  "distribution": {"name": "", "version": "", "confidence": "unknown"},
  "api": {"resources": [], "versions": [], "crds": []},
  "nodes": [],
  "system_workloads": [],
  "security": {"webhooks": [], "pod_security_namespaces": [], "network_policies": []},
  "storage": {"classes": [], "csi_drivers": []},
  "warnings": [],
  "redaction": {"secrets_omitted": true, "raw_kubeconfig_omitted": true}
}

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: example
  template:
    metadata:
      labels:
        app.kubernetes.io/name: example
    spec:
      containers:
        - name: example
          image: nginx:1.29
          ports:
            - name: http
              containerPort: 80
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 128Mi
          readinessProbe:
            httpGet:
              path: /
              port: http
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true

# Kubernetes diagnostic report

- Date/time:
- Operator:
- Context (redacted if sensitive):
- Distribution/provider:
- Server/client version:
- Namespace/scope:
- User-visible symptom:
- First observed:
- Recent changes:

## Evidence

- Objects and conditions:
- Events:
- Logs (bounded and redacted):
- Nodes/resources:
- Network/DNS/route checks:
- Storage checks:
- Policy/auth checks:

## Hypotheses

1.
2.
3.

## Actions

| Time | Command/action | Scope | Result | Reversible? |
|---|---|---|---|---|
| | | | | |

## Verification

- API state:
- Workload state:
- User-visible boundary:
- Remaining uncertainty:

## Sources

-

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: example-default-deny
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: example
  namespace: default
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: example

# Kubernetes upgrade runbook

## Scope

- Cluster/provider:
- Distribution:
- Current server version:
- Target version:
- Context:
- Maintenance window:
- Owner and approver:

## Preflight

- [ ] Current official support/version-skew policy checked
- [ ] Deprecated API usage checked
- [ ] CRDs/operators/webhooks checked for compatibility
- [ ] PDBs, capacity, quotas, and node health checked
- [ ] Backup completed and restore tested or explicitly accepted as a risk
- [ ] Provider/distribution-specific procedure selected
- [ ] Rollback or restore path documented

## Change

Record exact commands, target, operator, start/end time, and output location. Do not paste credentials or Secret values.

## Verification

- [ ] API server/version/discovery
- [ ] Nodes and system workloads
- [ ] CRDs/operators/controllers
- [ ] Representative stateless workloads
- [ ] Stateful workloads and PVCs
- [ ] Services, routes, DNS, and external endpoint
- [ ] Metrics, logs, audit, and alerts

## Sources

-