# Azure AKS Rollout Operations

Use this reference for current, source-grounded service behavior and the hard review gates that the lean `SKILL.md` intentionally does not carry.

## What people get wrong

- Calling a rollout safe because kubectl apply succeeded.
- Ignoring PDBs, replicas, readiness probes, and maxUnavailable until a drain is already blocked.
- Using maxUnavailable to avoid quota issues without admitting the disruption tradeoff.
- Starting a node-pool upgrade without IP, compute quota, and rollback-window evidence.
- Running rollback commands twice because no one tracked current rollout state.

## Officially grounded service shape

Microsoft Learn evidence says AKS rolling upgrades add surge nodes, cordon and drain old nodes, optionally wait for soak time, reimage nodes, repeat, and remove surge nodes. Production node pools are recommended to use max surge rather than disruptive in-place unavailability where possible. Max unavailable can disrupt workloads and increase failures from unsatisfied PodDisruptionBudgets. Blue-green node-pool upgrade includes a final soak window during which rollback is available before old nodes are removed.

- Deployment rollouts and AKS node-pool upgrades are separate but coupled failure domains.
- AKS node rolling upgrades use surge, cordon, drain, optional soak, reimage, and cleanup phases.
- PDBs can protect availability but can also block drains when configured too tightly for replicas and topology.
- Blue-green upgrade provides a validation and rollback window, but rollback disappears after the final commit/removal point.
- Read-only evidence can prove current health snapshots, not future rollout success.

## Non-negotiable design rules

- Stop if target subscription/resource group/cluster/namespace/workload/principal or approval is ambiguous.
- Require preflight evidence for pods, deployments, events, PDBs, HPA, nodes, quotas, and recent errors before mutation.
- Prefer pause/status/describe/dry-run/plan checks before apply, restart, drain, upgrade, or undo.
- Define rollback trigger, command, owner, and verification before execution.
- Sanitize outputs; never print tokens, kubeconfig contents, environment values, or customer data.

## Minimal safe implementation flow

- Confirm target and read-only context: cluster, namespace, workload, node pool, current principal, and requested operation.
- Collect rollout status, deployment spec, PDBs, replicas, readiness, events, node health, capacity, and monitoring signals.
- Compare rollout strategy, maxUnavailable/maxSurge, PDB, and drain/soak settings against documented AKS and Kubernetes behavior.
- Gate mutation on explicit approval, rollback plan, and health criteria.
- After action, verify deployment availability, events, logs/metrics summary, and rollback posture.

## High-risk assumptions to kill

- `kubectl apply` success is not rollout success; availability, events, probes, PDBs, HPA behavior, and monitored health must be checked.
- A PDB can protect or deadlock a rollout; the math must fit replica count, maxUnavailable, topology, and node drain behavior.
- Max surge consumes compute and IP quota, while max unavailable trades capacity for disruption; neither setting is safe without explicit evidence.
- Force or bypass paths that ignore PDBs can create complete service unavailability and need explicit emergency justification.
- Rollback is not real if revision history, previous ReplicaSet, blue-green soak window, or node-pool recovery path is unknown.

## Safe command/code verification targets

- Inspect deployment manifests for replicas, strategy, maxSurge, maxUnavailable, probes, termination grace, resource requests, and topology spread.
- Review PDB and HPA manifests together to prove eviction allowance cannot deadlock under expected rollout or drain conditions.
- Check AKS node-pool settings for max surge, max unavailable, drain timeout, soak duration, version skew, node image, zones, quotas, and subnet IP headroom.
- Verify preflight commands use status/describe/events/dry-run style evidence before mutation and avoid printing kubeconfig, tokens, or environment values.
- Confirm rollback playbook names the exact pause, undo, history, node-pool rollback/blue-green decision point, and post-rollback health checks.

## Safe verification targets

- Deployment has enough replicas and PDB allowance for expected disruption.
- Node pools have surge/capacity/IP quota or an explicit accepted maxUnavailable tradeoff.
- Readiness/liveness/startup probes and termination grace match rollout and drain behavior.
- Rollback command and validation window are known before mutation.
- Post-rollout status shows desired replicas available, no new critical events, and monitored health criteria met.

## When to push back

- The user asks to mutate a live cluster without explicit approval.
- PDBs or replicas make disruption mathematically unavoidable.
- Quota/capacity evidence is missing for surge-based upgrades.
- Rollback cannot be explained before starting the rollout.

# Documentation and Evidence Path

## Preferred evidence order

1. Microsoft Learn documentation through the user's configured documentation MCP for documented Azure behavior.
2. Sampled read-only Azure or Kubernetes evidence, when safely available, for current configured-environment observations.
3. Sanitized user-provided evidence.
4. Clearly labeled inference.

## What each evidence type can prove

- Microsoft Learn documentation can prove documented service behavior, supported concepts, and recommended patterns.
- Sampled read-only evidence can prove the sampled configured state at the time observed.
- Sanitized user evidence can prove only what the snippet shows.
- None of these alone prove broad regional availability, future success, full account posture, or production readiness.

## Safe usage pattern

- State whether each claim is documentation-based, sampled-current-state, user-provided, or inference.
- Use read-only queries before recommending changes.
- Do not include sensitive internal identifiers, tenant identifiers, subscription identifiers, or secrets in committed docs or final findings.
- If no sampled evidence is available, say the review is documentation-based and list the exact evidence still needed.

## Asset guidance

Use Microsoft Learn documentation through the user's configured documentation MCP for documented AKS behavior. Use sampled read-only Azure or Kubernetes evidence only for current cluster observations and label it as sampled evidence.

# Official Sources

Use these sources to ground the skill. Microsoft Learn documentation proves documented Azure behavior; it does not prove the user's tenant, RBAC, quotas, deployed resources, or production readiness.

## Primary Microsoft Learn sources

- https://learn.microsoft.com/azure/aks/upgrade-aks-node-pools-rolling
- https://learn.microsoft.com/azure/aks/upgrade-options
- https://learn.microsoft.com/azure/aks/upgrade-conceptual
- https://learn.microsoft.com/azure/aks/blue-green-node-pool-upgrade
- https://learn.microsoft.com/azure/architecture/operator-guides/aks/aks-upgrade-practices
- https://learn.microsoft.com/azure/aks/concepts-clusters-workloads
- https://learn.microsoft.com/azure/aks/operator-best-practices-cluster-security
- https://kubernetes.io/docs/tasks/run-application/configure-pdb/
- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment

## Grounding notes

- Documentation-based claim: Microsoft Learn evidence says AKS rolling upgrades add surge nodes, cordon and drain old nodes, optionally wait for soak time, reimage nodes, repeat, and remove surge nodes. Production node pools are recommended to use max surge rather than disruptive in-place unavailability where possible. Max unavailable can disrupt workloads and increase failures from unsatisfied PodDisruptionBudgets. Blue-green node-pool upgrade includes a final soak window during which rollback is available before old nodes are removed.
- Current-state claim: requires sampled read-only Azure evidence or sanitized user-provided evidence.
- Inference: allowed only when labeled and tied to observed fields or documented behavior.
- Do not include sensitive internal identifiers or secret material in findings.

## Source use rules

- Prefer Microsoft Learn documentation through the user's configured documentation MCP for current Azure service behavior.
- Use sampled read-only Azure evidence only to validate current configured-environment observations.
- If documentation and sampled evidence appear to conflict, report both and stop short of a production-ready verdict.
- Re-check official sources before changing high-risk guidance, because cloud behavior and feature availability can change.

## Current Microsoft Learn deltas checked on 2026-06-05

- Production AKS upgrade guidance uses surge capacity as the normal safety path, but more surge is not automatically safer when quota, subnet IPs, or workload capacity are constrained.
- MaxUnavailable behavior is constrained by node-pool type and can deadlock or disrupt workloads when PodDisruptionBudgets, drain timeout, or capacity are wrong.
- Force upgrade bypasses normal disruption protections and must be treated as a high-risk live operation, not a routine rollout fix.
- Rollback is not a generic one-command undo; capture current state and rollback limits before any node-pool or workload rollout mutation.

# Permission Model: Azure Live AKS Rollout Guard

## Azure RBAC (control plane — cluster credential access)

```json
{
  "Name": "AKS Rollout Guard",
  "IsCustom": true,
  "Description": "Read AKS cluster state and fetch user-level kubeconfig. No cluster admin rights.",
  "Actions": [
    "Microsoft.ContainerService/managedClusters/read",
    "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"
  ],
  "NotActions": [
    "Microsoft.ContainerService/managedClusters/delete",
    "Microsoft.ContainerService/managedClusters/agentPools/write"
  ],
  "AssignableScopes": [
    "<cluster-resource-scope>"
  ]
}
```

`listClusterUserCredential` grants a user-level kubeconfig. What the user can do inside
the cluster is governed by AKS-integrated Entra ID RBAC, not this control-plane role.
Use the exact AKS cluster resource scope from approved change records; do not paste raw
subscription identifiers into chat.

## Kubernetes RBAC (data plane — in-cluster namespace scope)

Bind the operator's Entra ID identity to a namespace-scoped Role (never ClusterRole):

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rollout-guard
  namespace: <namespace-name>
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list"]
```

## Do not assign

- `Azure Kubernetes Service Cluster Admin Role` — full cluster admin kubeconfig
- `cluster-admin` ClusterRoleBinding in Kubernetes
- `Microsoft.ContainerService/managedClusters/agentPools/delete`
- Subscription-level Contributor for routine rollout operations

# Preflight Commands: Azure Live AKS Rollout Guard

Use shell variables for examples instead of raw identifiers. Populate them from an approved change record or already configured shell context; never paste tenant, subscription, resource, or secret values into chat.

## Evidence-variable convention

Variables such as $AZURE_RESOURCE_GROUP_NAME, $APP_SERVICE_APP_NAME, or $KEY_VAULT_NAME are local operator placeholders. Do not commit real values, and redact them from shared evidence unless the change record explicitly allows disclosure.

Run these commands before any AKS rollout mutation. Paste sanitized output as evidence.

## 1. Confirm identity and cluster target

```bash
az account show --query "{subscriptionName:name, user:user.name}"
az aks show -g $AZURE_RESOURCE_GROUP_NAME -n $AKS_CLUSTER_NAME \
  --query "{provisioningState:provisioningState, kubernetesVersion:kubernetesVersion, fqdn:fqdn}"
```

## 2. Fetch user-level kubeconfig

```bash
az aks get-credentials -g $AZURE_RESOURCE_GROUP_NAME -n $AKS_CLUSTER_NAME --overwrite-existing
kubectl config current-context
```

## 3. Audit PodDisruptionBudgets in target namespace

```bash
kubectl get pdb -n $KUBERNETES_NAMESPACE -o wide
# minAvailable or maxUnavailable must leave at least one pod available during rollout
```

## 4. Check current deployment rollout status

```bash
kubectl rollout status deployment/$KUBERNETES_DEPLOYMENT_NAME -n $KUBERNETES_NAMESPACE
kubectl get deployment $KUBERNETES_DEPLOYMENT_NAME -n $KUBERNETES_NAMESPACE -o jsonpath='{.spec.strategy}'
```

## 5. Verify node readiness and resource headroom

```bash
kubectl get nodes -o wide
kubectl top nodes
kubectl get pods -n $KUBERNETES_NAMESPACE -o wide
```

## 6. Confirm maxSurge / maxUnavailable strategy

```bash
kubectl get deployment $KUBERNETES_DEPLOYMENT_NAME -n $KUBERNETES_NAMESPACE \
  -o jsonpath='{.spec.strategy.rollingUpdate}'
# maxUnavailable=0 is safest for production; maxSurge=1 is a conservative default
```

## 7. Check HorizontalPodAutoscaler (if present)

```bash
kubectl get hpa -n $KUBERNETES_NAMESPACE
# HPA minReplicas must exceed PDB minAvailable or the rollout will deadlock
```

# Rollback Playbook: Azure Live AKS Rollout Guard

## Immediate rollback — undo to previous ReplicaSet

```bash
# Pause the rollout first to stop further progress
kubectl rollout pause deployment/<deployment-name> -n <namespace-name>

# Check rollout history to identify the target revision
kubectl rollout history deployment/<deployment-name> -n <namespace-name>

# Undo to the immediately prior revision
kubectl rollout undo deployment/<deployment-name> -n <namespace-name>

# Or undo to a specific revision
kubectl rollout undo deployment/<deployment-name> -n <namespace-name> --to-revision=<N>
```

## Verify rollback success

```bash
kubectl rollout status deployment/<deployment-name> -n <namespace-name>
kubectl get pods -n <namespace-name> -o wide
kubectl describe deployment <deployment-name> -n <namespace-name> | grep -A 5 "Conditions:"
```

## Rollback limitations

- `kubectl rollout undo` reverts the pod template spec only (image, env, volumes).
- It does NOT revert ConfigMaps, Secrets, PVCs, or Service endpoint changes.
- If a schema migration ran as an init container, the rollback will reuse the new schema.
- HPA target replicas and PDB settings are not reverted by `rollout undo`.

## Escalation path

1. If rollback leaves pods in `CrashLoopBackOff`: check logs with `kubectl logs <pod-name> -n <namespace-name> --previous`
2. If node is under memory pressure: drain the node with `kubectl drain <node-name> --ignore-daemonsets`
3. If the cluster is unresponsive: escalate to AKS support via Azure portal → cluster → Support + troubleshooting

# Safety Checklist

## Evidence labels

- `documentation-based`: grounded in Microsoft Learn or official Kubernetes documentation where listed.
- `sampled-current-state`: grounded in read-only Azure or Kubernetes observations from the user's configured tools.
- `user-provided`: grounded in sanitized snippets supplied by the user.
- `inference`: reasoned from evidence but not directly proven.

## Mutation boundary

- Default to read-only review.
- Do not perform create, update, delete, rotate, purge, recover, apply, restart, drain, cordon, scale, rollout, role-assignment, policy-assignment, or network changes unless the user explicitly asks and approval is clear.
- Prefer preview, dry-run, status, describe, what-if, list, show, and policy evaluation evidence before any mutation.

## Credential and data boundary

- Never ask users to paste credentials, tokens, tenant IDs, subscription IDs, customer data, private keys, kubeconfig contents, CA requester credentials, secret values, or connection strings.
- Summarize sensitive evidence by field presence, control state, and risk; do not reproduce secret material.

## Risk gates

- Stop on ambiguous target, ambiguous principal, missing approval, missing rollback, or missing owner for high-impact assets.
- Treat broad permissions, permanent privileged access, public exposure, purge authority, destructive operations, and live rollout changes as high-risk.
- Separate documented product behavior from sampled configured-environment evidence.

## Asset-specific hard line

Never advance an AKS rollout without target, principal, approval, PDB audit, replica health, capacity, and rollback evidence. Treat undo, drain, cordon, scale, and node-pool upgrade operations as live mutations requiring explicit approval.

# Workflow and Output Contract

## Execution flow

1. Scope the exact asset, environment boundary, owner, and requested decision.
2. Load `official-sources.md`, then the component operations guide for service behavior and risk gates.
3. Gather sampled read-only evidence only when available and safe.
4. Compare observed posture against documented behavior, least-privilege expectations, and operational safety rules.
5. Return a verdict with evidence level, blockers, safe next actions, and open questions.

## Required output

- `verdict`: pass, warn, fail, or blocked.
- `evidence_level`: documentation-based, sampled-current-state, user-provided, inference, or mixed.
- `scope`: what was reviewed and what was not reviewed.
- `blockers`: issues that prevent a safe or production-ready conclusion.
- `findings`: severity-labeled risks with source labels.
- `safe_next_actions`: reversible actions first; mutation only with explicit approval.
- `open_questions`: missing facts that would change the verdict.

## Stress checks

- What assumption would make this recommendation unsafe?
- Which role, policy, network, lifecycle, or rollout action has the largest blast radius?
- What evidence would disprove the claimed readiness?
- Is the answer accidentally treating documentation as tenant-specific proof?

## Response discipline

Use Microsoft Learn documentation through the user's configured documentation MCP for documented AKS behavior. Use sampled read-only Azure or Kubernetes evidence only for current cluster observations and label it as sampled evidence.