sota-kubernetes
State-of-the-art Kubernetes platform security and operations (2026) for cloud-managed (EKS/GKE/AKS) and self-hosted clusters (kubeadm, k3s/k0s, Talos). Use when building, operating, hardening, or auditing the cluster PLATFORM layer: control plane and etcd, API server/kubelet, RBA
Install
npx skills add https://github.com/martinholovsky/SOTA-skills/tree/main/skills/sota-kubernetes
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install martinholovsky-sota-skills@llmmart
git clone https://github.com/martinholovsky/SOTA-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole martinholovsky/sota-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
SOTA Kubernetes Platform Security & Operations
Purpose
Engineer and audit the Kubernetes platform so that a compromised workload, a hostile chart, a leaked token, or a malicious controller cannot pivot to cluster-admin, read every Secret, or take the cluster down. This skill owns the layer above the pod: control plane and etcd, API server and kubelet hardening, RBAC and ServiceAccount identity, admission control and policy-as-code, GitOps controllers, operators/CRDs and admission webhooks, Helm/Kustomize at admission, multi-tenancy boundaries, cluster lifecycle, and K8s audit logging.
It does not re-teach pod isolation mechanics. Boundaries it defers:
- Pod
securityContext, seccomp, AppArmor, capabilities, PSA pod-level fields →sota-sandboxing(rules/03 containers & microVMs; rules/01 boundaries). This skill owns the admission-time enforcement of those fields, not their internals. - NetworkPolicy semantics, CNI choice, service mesh, mTLS →
sota-network-security. This skill states only the requirement (default-deny per namespace, enforced at admission). - OIDC/SSO and RBAC-role design methodology →
sota-identity-access. This skill owns the K8s RBAC mechanics and escalation traps. - Runtime/audit-log detection content (Falco/Tetragon rules, detections) →
sota-detection-engineering. This skill owns producing and shipping the audit stream. - Workload identity, Secret storage backends (ESO, sealed-secrets, CSI) →
sota-secrets-management(rules/01, rules/02). This skill owns etcd encryption-at-rest. - Cloud IAM, managed-K8s selection, DR →
sota-cloud-infrastructure(rules/02, rules/04, rules/07). - CI/CD provenance, image signing, IaC/GitOps pipeline, runtime ops →
sota-devsecops(rules/02, rules/04, rules/06, rules/07).
Two modes. Pick one explicitly at the start of the task.
BUILD mode
Use when provisioning a cluster, writing RBAC/policies/manifests, configuring a GitOps controller, or installing an operator/chart.
- Name the cluster topology first: managed (EKS/GKE/AKS) vs self-hosted (kubeadm, k3s/k0s, Talos), single- vs multi-tenant, who the threat actors are (workload → control plane? tenant → tenant? compromised CI → cluster?). Read the matching rules.
- Least privilege by default: no wildcard RBAC verbs/resources,
automountService AccountToken: falseunless the pod calls the API, scoped AppProjects, operators get the narrowest RBAC that works. Loosen only with a comment stating why. - Encrypt etcd at rest with a KMS provider (
rules/01) — Secrets are base64, not encrypted, by default. This is the single most common "we thought we were covered" gap. - Admission is fail-closed and ENFORCING, not auditing. A policy left in
audit/warnforever is documentation, not a control (rules/03). Ship the AUDIT→ENFORCE rollout plan with the policy. - GitOps is the only write path to the cluster. Humans propose via PR; the
controller reconciles. Scope the controller's own privileges and AppProjects tightly
(
rules/04). - Plan the upgrade before you build: version skew, EOL date, CVE-response runbook,
tested etcd restore (
rules/01,rules/07).
Deliverables: topology + threat statement, the concrete manifests/configs, the AUDIT→ENFORCE rollout, and the residual-risk/assumptions list (cloud-managed control plane internals you cannot see, org IAM, CNI behavior).
AUDIT mode
Use when reviewing an existing cluster, its RBAC, policies, GitOps config, or operators.
Procedure: inventory the cluster surface (control plane flags or managed equivalent,
RBAC graph, admission policies, GitOps controllers, operators/CRDs, namespaces/tenancy);
for each, walk the relevant rules-file Audit checklist; verify empirically with
kubectl, kubectl auth can-i, rbac-tool/krane, kube-bench, and helm template where
possible; report findings in the format below. Do not report style nits as security
findings.
Severity conventions
| Severity | Meaning | Examples |
|---|---|---|
| Critical | Cluster-admin, all-Secrets read, or cluster takedown reachable now | anonymous-auth enabled on API server/kubelet; etcd unencrypted AND reachable; ClusterRoleBinding granting cluster-admin to a workload SA or system:authenticated; wildcard */* ClusterRole bound broadly; Argo CD AppProject clusterResourceWhitelist: [{group: '*', kind: '*'}] with broad SSO; unpatched control plane on a known-RCE CVE |
| High | Escalation/secret-read by an in-cluster or contributor principal, or a single event from it | escalate/bind/impersonate verbs granted; secret-reader → token-mint → privilege chain; admission policy in audit mode for a control that should enforce; image-verification policy only Audits signatures; operator with cluster-wide secrets:*; kubelet read-only-port open; no etcd backup or untested restore |
| Medium | Weakens defense in depth or detection | PSA not enforced (only warn/audit); automountServiceAccountToken defaulted on for non-API pods; no default-deny NetworkPolicy (requirement-level; depth → network-security); audit policy missing or None/Metadata-only for Secret access; aggregated ClusterRole accreting verbs; no PDB on critical workloads |
| Low | Hygiene, hardening headroom | RBAC subjects for departed users; unused ClusterRoles; namespaces without resource quotas; :latest image tags admitted; missing PolicyException expiry |
Severity is judged by reachability (anonymous > workload/tenant > contributor > admin) × yield (cluster-admin/all-Secrets > namespace compromise > info leak > availability).
Finding format
file:line | rule | severity | effort | fix
Where effort is one of trivial / small / medium / large, rule is the rules-file
section (e.g. 02 §RBAC-wildcards), and fix is the concrete change. Expand below the
line with: what is wrong, who exploits it and how (concrete attack path), and a snippet
when short. Example:
clusterrole-ci.yaml:14 | 02 §escalate-verbs | Critical | small | remove the bind/escalate grant; scope to the 3 named ClusterRoles the controller actually applies
Issue: ClusterRole bound to the CI ServiceAccount grants rbac.authorization.k8s.io/{bind,escalate} on clusterroles.
Attack path: anyone who can run a job as this SA (any merged PR) can bind themselves cluster-admin.
End every audit with: counts per severity, top 3 fixes by risk-reduction-per-effort, and an explicit OUT-OF-SCOPE list (managed control-plane internals, org IAM, CNI/mesh, runtime detection content).
Rules index
| File | Read this when... |
|---|---|
| rules/01-control-plane-etcd.md | HA control plane, API server flags (anonymous-auth, authz modes, audit), etcd encryption-at-rest with KMS v2 + backup/defrag/restore, kubelet hardening, node + immutable-distro hardening (Talos no-SSH/machine-config/SecureBoot+TPM, k3s/k0s), CIS benchmark + kube-bench |
| rules/02-rbac-serviceaccounts.md | Roles/ClusterRoles least-privilege, the escalation traps (wildcards, bind/escalate/impersonate, cluster-admin bindings, aggregated roles, secret-reader chains), ServiceAccount hygiene (automount off, bound/projected/audience-scoped tokens, no long-lived token Secrets), RBAC auditing (auth can-i, rbac-tool/krane/who-can) |
| rules/03-admission-policy.md | Pod Security Admission (restricted/baseline/privileged, enforce/audit/warn) and its limits, Kyverno vs Gatekeeper/OPA vs ValidatingAdmissionPolicy/MutatingAdmissionPolicy, the AUDIT→ENFORCE rollout discipline, image verification at admission (cosign/Kyverno verifyImages), PolicyException discipline |
| rules/04-gitops-controllers.md | Argo CD / Flux security: AppProject scoping (the clusterResourceWhitelist:[{*,*}] trap), project/RBAC/SSO, the controller's own privileges and self-management, repo/SSH creds, ApplicationSet injection, auto-sync vs approval, drift, recent Argo CD CVEs, promotion/rollback as git ops |
| rules/05-operators-crds-webhooks.md | The operator privilege problem (broad RBAC → CRD-mediated escalation), vetting operator RBAC, CRD validation/security, trusted controllers (ESO, cert-manager), admission webhooks as attack surface (failurePolicy, timeout, TLS, namespaceSelector) |
| rules/06-workloads-tenancy.md | Resource requests/limits as availability+QoS, PodDisruptionBudget, topology spread, anti-affinity, priorityClass/preemption, namespace-as-SOFT-boundary reality, hard multi-tenancy (vCluster/separate clusters/node isolation), Secrets in K8s (etcd encryption + ESO/sealed-secrets pointer) |
| rules/07-supply-chain-audit.md | Helm/Kustomize review before apply (helm template, RBAC-in-charts trap, privilege-granting values), OCI chart signing/provenance, only-signed-images enforcement, digest pinning at the manifest layer, K8s audit policy (what/levels), shipping audit logs, detection + runtime-security pointers |
When a task spans layers (most do), read every matching file. For a full cluster audit, read all seven plus the referenced sibling skills.
Top 10 non-negotiables
Violations are at minimum High in AUDIT mode and must never be introduced in BUILD mode:
anonymous-auth=falseon the API server AND kubelet; kubeletread-only-port=0, kubelet authz modeWebhook(notAlwaysAllow). Anonymous or unauthenticated access to either is cluster-game-over (01).- etcd encrypted at rest with a KMS v2 provider (envelope DEK/KEK), not the default
base64. Secrets are NOT encrypted out of the box (
01,06). - No wildcard RBAC. No
*verbs/resources in Roles/ClusterRoles bound to workloads or users; nocluster-adminbound to a ServiceAccount,system:authenticated, orsystem:unauthenticated(02). bind,escalate,impersonateare privilege-escalation verbs — grant only to a named, audited admin path, never to a workload or CI SA (02).automountServiceAccountToken: falseis the default; opt in only for pods that call the API. No long-lived Secret-based SA tokens — use bound/projected tokens (02).- Pod Security Admission
restrictedENFORCED on workload namespaces (labelpod-security.kubernetes.io/enforce: restricted), not merelywarn/audit. PSA is stable; PSP is gone (03). Pod-level field internals → sota-sandboxing. - Admission policies enforce, with a documented AUDIT→ENFORCE rollout. A policy
parked in
Audit/warnindefinitely is a real, recurring finding (03). - GitOps is the only cluster write path; the controller and its AppProjects are
tightly scoped. Never
clusterResourceWhitelist: [{group:'*',kind:'*'}]with broad project access; pin Argo CD/Flux versions and patch known CVEs (04). - Operators get least-privilege RBAC; admission webhooks have correct
failurePolicy/timeoutSeconds/namespaceSelectorand TLS — a webhook is both a control and a single point of failure/attack (05). - API server audit logging enabled (RequestResponse for sensitive verbs/Secrets),
shipped to tamper-resistant storage; etcd backups taken AND restore-tested (
07,01). Detection content on the stream → sota-detection-engineering.
If the user asks for something that violates a non-negotiable, implement the secure alternative and explain the delta; comply only after they acknowledge the risk explicitly.
Files (sota-skills)
-
rules
-
01-control-plane-etcd.md 14.8 KB
# 01 — Control Plane, etcd & Node/Distro Hardening Scope: the cluster's brain — HA control plane, API server and kubelet hardening, etcd encryption/backup/restore, node and immutable-distro hardening (Talos, k3s/k0s), version skew and CVE response. Frame against the **CIS Kubernetes Benchmark** (v1.12.0 at writing, matched to your minor version) and the **NSA/CISA Kubernetes Hardening Guidance** (v1.2, Aug 2022 — still the current edition; verify before citing). On managed clusters (EKS/GKE/AKS) the provider owns the control plane and etcd — you cannot set these flags; audit the *managed equivalents* (provider security posture, control-plane logging, secrets-encryption setting) and focus your effort on RBAC, admission, and nodes. --- ## 1. HA control plane - **Three (or five) control-plane nodes** across failure domains; etcd quorum needs an odd count. Two nodes is *worse* than one (split-brain risk, no quorum gain). - **API server behind a load balancer**; control-plane components talk to it via a stable endpoint, not a single node IP. - **etcd: stacked vs external.** Stacked (etcd co-located with control plane) is simpler and fine for most; external etcd isolates blast radius and is preferred for large or high-security clusters. Either way etcd is the crown jewel — everything below applies. - Control-plane nodes run **no workloads** (taint `node-role.kubernetes.io/control-plane: NoSchedule`). A tenant pod on a control-plane node is a node-escape away from etcd. ## 2. API server hardening These are kube-apiserver flags (or managed-cluster equivalents). Each is a CIS control. | Setting | Required value | Why | |---|---|---| | `--anonymous-auth` | `false` | Anonymous requests hit RBAC as `system:anonymous`/`system:unauthenticated`; combined with any loose binding = unauthenticated access. **Critical if true.** | | `--authorization-mode` | `Node,RBAC` (never includes `AlwaysAllow`) | `AlwaysAllow` disables authz entirely. `Node` authorizer + `RBAC` is the baseline. | | `--enable-admission-plugins` | includes `NodeRestriction` | Stops a compromised kubelet from editing other nodes/pods or escalating via node labels. | | `--audit-policy-file` / `--audit-log-path` | set (see `rules/07`) | No audit log = no forensics, no detection. | | `--encryption-provider-config` | set with KMS v2 (see §4) | Secrets at rest. | | `--service-account-lookup` | `true` | Revoked SA tokens stop working. | | `--tls-min-version` | `VersionTLS12`+ ; strong cipher suites | Defense in depth on the API. | | `--profiling` | `false` in prod | `/debug/pprof` leaks data and is a DoS vector. | | `--request-timeout`, `--max-requests-inflight` | tuned | API server DoS resistance. | | `--enable-priority-and-fairness` | `true` (the default) | API Priority & Fairness — never set `false`. | **API Priority & Fairness (APF)** is stable since v1.29 and **on by default**; it supersedes the raw `--max-requests-inflight` limits by splitting that total concurrency across priority levels via `FlowSchema` + `PriorityLevelConfiguration`, with fair queuing so one runaway controller can't starve `leader-election` or node heartbeats. Audit: confirm it isn't disabled (`--enable-priority-and-fairness=false` is a finding), give a dedicated `PriorityLevelConfiguration` to high-value controllers, and alert on `apiserver_flowcontrol_rejected_requests_total`. A catch-all FlowSchema that lumps everything into one level re-creates the global-limit DoS APF exists to prevent. **Authentication**: prefer **OIDC** for human users (RBAC-design and SSO are `sota-identity-access` territory — wire it there). Never distribute the cluster-admin kubeconfig / client cert as the day-to-day human credential; client certs cannot be revoked short of CA rotation. Service-to-API auth uses ServiceAccount tokens (`rules/02`). ```yaml # BAD — kube-apiserver manifest fragments that are Critical/High findings - --anonymous-auth=true - --authorization-mode=AlwaysAllow - --insecure-port=8080 # removed in modern K8s; if present, you're ancient - --token-auth-file=/etc/tokens # static token files are unrevocable plaintext creds ``` ## 3. Kubelet hardening The kubelet is a root-capable agent on every node with its own API. Harden it explicitly — defaults have historically been loose. | Setting | Required value | Why | |---|---|---| | `anonymous.enabled` (`--anonymous-auth`) | `false` | An open kubelet API = run any pod, read any Secret mounted on the node, exec into containers. **Critical if anonymous.** | | `authorization.mode` (`--authorization-mode`) | `Webhook` | `AlwaysAllow` lets any authenticated client drive the kubelet. | | `readOnlyPort` (`--read-only-port`) | `0` | The read-only port (10255) exposes pod/node metadata unauthenticated. | | `--rotate-certificates`, `serverTLSBootstrap` | `true` | Short-lived, rotated kubelet certs. | | `protectKernelDefaults` | `true` | Kubelet refuses unsafe sysctls. | | `streamingConnectionIdleTimeout` | non-zero | Reaps idle exec/attach streams. | | `makeIPTablesUtilChains` | `true` | Expected networking baseline. | Hunt: `curl -sk https://NODE:10250/pods` returning data without a token is a Critical finding (anonymous kubelet). `curl http://NODE:10255/pods` returning data means the read-only port is open. ## 4. etcd security — "Secrets are base64, not encrypted" **The reality:** by default Kubernetes stores Secret objects in etcd as **base64-encoded plaintext**. Anyone who reads etcd (etcd client access, an etcd backup file, a disk image, a node with the data dir) reads every Secret in the cluster. Base64 is encoding, not encryption. **Encrypt at rest with a KMS v2 provider** (GA since Kubernetes v1.29; KMS v1 deprecated v1.28, disabled-by-default v1.29 — do not build on v1). KMS v2 does envelope encryption: a local DEK encrypts data, a remote KMS KEK wraps the DEK, and v2 derives single-use DEKs from a rotated seed for performance. ```yaml # EncryptionConfiguration — KMS v2 first, so Secrets get the strong provider apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: ["secrets"] providers: - kms: apiVersion: v2 name: cloud-kms endpoint: unix:///run/kmsplugin/socket.sock - identity: {} # fallback for reads of not-yet-rewritten data; never first ``` ```yaml # BAD — identity first means everything is written in plaintext providers: - identity: {} - aescbc: { keys: [...] } # also: aescbc is weaker than aesgcm/KMS; local key on disk ``` - The **provider listed first encrypts** new writes. `identity` must never be first. - After enabling/rotating, **rewrite all Secrets** so existing data is re-encrypted: `kubectl get secrets -A -o json | kubectl replace -f -`. - On managed clusters, enable the provider's secrets-encryption feature (e.g. EKS KMS envelope encryption, GKE Application-layer Secrets Encryption, AKS KMS etcd encryption). - **etcd transport**: peer and client TLS with cert auth (`--client-cert-auth=true`, `--peer-client-cert-auth=true`); etcd reachable only from control-plane nodes, never exposed to the pod network or internet. ### etcd backup, defrag, restore — TESTED - **Scheduled `etcdctl snapshot save`** (or managed backup), stored **off-cluster** in a separate trust domain, encrypted, immutable, not deletable by cluster credentials. - A backup of an *encrypted* etcd needs the KMS KEK to restore — back up your key access / document the KEK recovery, or the snapshot is unrestorable. - **Periodic defrag** (`etcdctl defrag`) — fragmented etcd causes `mvcc: database space exceeded` and a hard API-server outage. Alert on etcd DB size approaching `--quota- backend-bytes`. - **Restore drills**: an untested backup is a hope. Drill restore-into-a-throwaway-cluster on a schedule; record RTO. Cross-link DR posture with `sota-cloud-infrastructure` rules/07. ## 5. Node hardening - **Minimal, hardened OS**; CIS-bench the nodes; auto-patch or immutable-rebuild. - **No SSH to nodes as a routine workflow** — node access is break-glass, audited. - Container runtime hardened (containerd with sane defaults); no Docker socket on nodes. - Pod-level isolation (seccomp/AppArmor/securityContext, runtime class gVisor/Kata) is **`sota-sandboxing`** (rules/03) — enforce it via admission (`rules/03`), don't re-spec it here. - Protect node metadata endpoints: block pod access to the cloud metadata service (169.254.169.254) unless via workload identity — see `sota-cloud-infrastructure` rules/02 and `sota-network-security`. ## 6. Immutable distros (e.g. Talos, including on ARM) **Talos Linux** — API-driven, immutable, minimal Linux purpose-built for K8s: - **No SSH, no shell, no package manager, no interactive login.** The entire node is managed via the gRPC `talosctl` API (mTLS-authenticated). This removes the single biggest node-attack surface: there is no interactive foothold to gain. - **Machine config is the security boundary.** Treat `machineconfig` like a Secret: it holds the cluster CA and join material. Store it encrypted, deliver via a trusted channel, and scope `talosconfig` credentials (the client cert) tightly — it is root-equivalent on the node. - **SecureBoot + TPM disk encryption.** Modern Talos (systemd-boot + Unified Kernel Image is the default for new UEFI installs since v1.10) supports SecureBoot; combine with **LUKS2 disk encryption keyed to the TPM** (`machine.systemDiskEncryption`) for measured boot and at-rest disk protection. On ARM, confirm board/firmware SecureBoot + TPM 2.0 support before relying on it; where TPM is unavailable, use a `nodeID` or KMS key source and document the weaker guarantee. - **KubePrism / API access**: restrict the Talos API and Kubernetes API endpoints to management networks; the Talos API at :50000 is as sensitive as the kube-apiserver. - Apply config changes via versioned, reviewed `talosctl apply-config` from git — Talos config is GitOps-able; treat it like the rest of `rules/04`. ```yaml # Talos machine config fragment — TPM-bound disk encryption (verify slot/keys per version) machine: systemDiskEncryption: state: { provider: luks2, keys: [{ tpm: {}, slot: 0 }] } ephemeral: { provider: luks2, keys: [{ tpm: {}, slot: 0 }] } ``` **k3s / k0s** (lightweight self-hosted): - k3s ships SQLite by default for single-server; use **embedded etcd (HA)** or an external datastore for multi-server, and apply the same etcd encryption discipline (§4) — k3s supports `--secrets-encryption` to enable at-rest encryption. - k3s bundles components; pin the version, track its CVE feed, and disable bundled add-ons you don't use (`--disable traefik,servicelb` etc.) to shrink surface. - k0s separates controller/worker cleanly; harden the same API-server/kubelet flags (§2, §3) via its config. Both still need RBAC/admission/audit from the other rules files. ## 7. Version skew, EOL & CVE response - **Supported window**: the project maintains the **latest three minor releases**, each with ~1 year of patch support. Run a supported minor; an EOL control plane gets no CVE fixes. (Verify current numbers at kubernetes.io/releases — at writing latest is the 1.36 line; 1.34/1.35/1.36 supported.) - **Version skew policy** (since 1.28): the **control plane may be up to 3 minor versions ahead of kubelets**; kube-apiserver instances within ≤1 minor of each other; kubectl within ±1 of the API server. Upgrade control plane first, then nodes — never the reverse. - **Upgrade cadence**: minor releases ~3×/year. Plan a rolling upgrade every 1–2 minors; don't fall to EOL. On managed clusters, stay on a supported channel and don't defer past the provider's forced-upgrade date. - **CVE response runbook**: subscribe to the `kubernetes-announce` list / CVE feed; for a control-plane RCE or auth-bypass, patch on the emergency track, not the quarterly one. Track CVEs for *every* control too: Argo CD (`rules/04`), Kyverno/Gatekeeper (`rules/03`), operators (`rules/05`), **ingress controllers and CSI drivers** (the official feed's 2026 wave hit both — ingress-nginx config-injection/auth-bypass/DoS CVE-2026-24512/-24513/ -24514/-1580/-3288/-4342; CSI SMB/NFS subDir path traversal CVE-2026-3864/-3865), and the distro (Talos/k3s/k0s). - **ingress-nginx is EOL** (retired by SIG Network/SRC; maintenance ended March 2026, repo read-only). The 2026 CVE wave *was* patched in the final releases (>=1.13.9/1.14.5/1.15.1); the standing risk is that any CVE found after EOL gets **no** fix. A deployed ingress-nginx is a standing High finding: migrate to Gateway API or an actively maintained ingress controller (ingress/edge config depth is `sota-network-security`). ## Audit checklist - [ ] API server: `--anonymous-auth=false`, `--authorization-mode` includes RBAC and not `AlwaysAllow`, `NodeRestriction` enabled, profiling off, audit configured? (`grep -E 'anonymous-auth|authorization-mode|NodeRestriction|profiling' /etc/kubernetes/manifests/kube-apiserver.yaml`; managed → check provider posture) - [ ] API Priority & Fairness left enabled (no `--enable-priority-and-fairness=false`), high-value controllers on a dedicated `PriorityLevelConfiguration`, `apiserver_flowcontrol_rejected_requests_total` alerted? - [ ] Kubernetes Dashboard: not deployed unless required; if deployed, NOT exposed publicly (no LoadBalancer/Ingress to it), reached only via `kubectl proxy`/authenticating proxy, and its ServiceAccount is least-privilege (never `cluster-admin`) — a privileged, exposed Dashboard is a one-click takeover (historic Tesla cryptojacking). Talos does not ship it; keep it that way. - [ ] Kubelet: anonymous-auth off, authz `Webhook`, `read-only-port=0`? (`curl -sk https://NODE:10250/pods` should 401; `curl http://NODE:10255/pods` should refuse) - [ ] etcd encrypted at rest with KMS v2, `identity` not first, all existing Secrets rewritten? (`kubectl get secret -A -o json | head` against an etcd dump; check `EncryptionConfiguration`) - [ ] etcd reachable only from control plane, client/peer TLS cert-auth on? (`etcdctl` from a worker should fail) - [ ] etcd backups scheduled, off-cluster, immutable, restore-DRILLED, KEK recoverable? (when was the last restore drill?) - [ ] etcd defrag scheduled and DB-size alerting wired? - [ ] Control-plane nodes tainted to run no workloads? - [ ] Nodes: minimal/hardened OS, kube-bench passing, no routine SSH? Talos: SecureBoot + TPM/LUKS2 disk encryption, machineconfig/talosconfig stored as secrets and scoped? k3s/k0s: `--secrets-encryption` on, unused add-ons disabled? - [ ] Running a SUPPORTED minor (not EOL)? Skew within policy (control plane ≥ nodes, ≤3 minors)? (`kubectl version`, `kubectl get nodes -o wide`) - [ ] CVE-response runbook exists and covers control plane + every add-on/controller + ingress/CSI + distro? - [ ] No EOL ingress-nginx still deployed (retired March 2026 — final releases patched the 2026 CVE wave, but any post-EOL CVE is unpatched — High; migration to Gateway API or a maintained controller done or dated)? -
02-rbac-serviceaccounts.md 11.5 KB
# 02 — RBAC & ServiceAccounts Scope: the K8s authorization graph and workload identity — least-privilege Roles/ ClusterRoles, the privilege-escalation traps, ServiceAccount hygiene and token model, and how to audit who-can-do-what. RBAC *role-design methodology and SSO* is `sota-identity-access`; this file owns the **K8s RBAC mechanics and the escalation traps**. Workload identity to cloud and secret backends is `sota-secrets-management` (rules/01). RBAC is the cluster's authorization fabric: `Role`/`ClusterRole` (a set of allowed verbs×resources) bound to subjects (users, groups, ServiceAccounts) by `RoleBinding` (namespaced) / `ClusterRoleBinding` (cluster-wide). Authorization is additive and allow-only — there is no deny rule. So the only lever is **granting less**. --- ## 1. Least privilege — the shape of a good Role - **Enumerate verbs and resources explicitly.** Name the API groups, resources, and verbs the workload actually uses. Default to `get`/`list`/`watch`; add `create`/`update`/ `patch`/`delete` only where proven. - **Namespaced `Role` over `ClusterRole`** unless the resource is cluster-scoped or the subject genuinely needs all namespaces. A `ClusterRoleBinding` is cluster-wide blast radius. - **`resourceNames`** to scope to specific objects where the API supports it (e.g. read one named ConfigMap, not all ConfigMaps). - **Separate read from write**; separate by namespace/team. One mega-role bound everywhere is the distributed `cluster-admin`. ```yaml # GOOD — narrow, namespaced, explicit kind: Role metadata: { name: orders-reader, namespace: orders } rules: - apiGroups: [""] resources: ["configmaps"] resourceNames: ["orders-config"] verbs: ["get", "watch"] ``` ## 2. The escalation traps (each a High/Critical finding) ### 2.1 Wildcards `verbs: ["*"]`, `resources: ["*"]`, or `apiGroups: ["*"]` grant **everything that exists now and everything added in future API versions**. A `*/*` ClusterRole is effectively cluster-admin. Never on a workload, CI, or tenant subject. ```yaml # BAD — this is cluster-admin with extra steps rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] ``` ### 2.2 The escalation verbs: `bind`, `escalate`, `impersonate` These are not ordinary write verbs — they let a subject **grant itself more than it has**: - **`escalate`** (on `roles`/`clusterroles`): create/edit a role with *more* permissions than you hold. Normally RBAC stops you from authoring a role above your own privileges; `escalate` removes that guard. → self-grant cluster-admin. - **`bind`** (on `roles`/`clusterroles`): bind an existing powerful role to yourself. `bind` on `cluster-admin` = `bind` yourself to cluster-admin. - **`impersonate`** (on `users`/`groups`/`serviceaccounts`): act as another principal — impersonate a cluster-admin user/group, or `system:masters`. Total bypass. Grant these only to a **named, audited human-admin path**, never to a workload or automation SA. `impersonate` on `groups` for `system:masters` is an instant Critical. ### 2.3 ClusterRoleBinding to `cluster-admin` (or to broad subjects) - A ClusterRoleBinding of the built-in `cluster-admin` ClusterRole to a **ServiceAccount** means owning that pod owns the cluster. Critical. - Binding a role to the group **`system:authenticated`** grants it to every authenticated principal; to **`system:unauthenticated`** grants it to unauthenticated callers. **There is no bridge between them**: an anonymous request gets the username `system:anonymous` and the group `system:unauthenticated`, and `system:authenticated` is assigned only when authentication *succeeds* ([K8s authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/), verified 2026-09-16). - **Rate the verbs and resources, not the group name.** Kubernetes itself binds `system:public-info-viewer` to *"system:authenticated and system:unauthenticated groups"* by default, so "bound to a broad group" cannot be Critical on its own — that would flag the shipped defaults. Critical is a *broad grant* to a broad group: write/exec/secrets-read, or wildcard verbs on wildcard resources. Compare against the default bindings (`kubectl get clusterrolebinding -o wide`) before filing. - **Know which defaults changed at v1.14**, because an old cluster inverts this check. `system:public-info-viewer` was *introduced* in v1.14; before it existed, **`system:basic-user` and `system:discovery` were themselves bound to `system:unauthenticated`** and were unbound from it at that release. So on a pre-1.14 cluster those two bindings are the shipped default, and on a current one they are a finding. Read the table rather than recalling it — a summary of this page got `system:discovery` wrong in exactly this way (default bindings table, verified 2026-09-16: [K8s RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)). ### 2.4 Aggregated ClusterRoles ClusterRoles with `aggregationRule` automatically absorb the rules of any ClusterRole matching the label selector. A new operator/chart that ships a ClusterRole with the aggregation label (e.g. `rbac.authorization.k8s.io/aggregate-to-admin: "true"`) silently **widens the built-in `admin`/`edit`/`view` roles cluster-wide**. Audit what aggregates into the powerful roles; a hostile or careless chart uses this to escalate quietly. ### 2.5 The secret-reader → privilege chain `get`/`list` on `secrets` is rarely "just read a config value." Secrets hold ServiceAccount tokens, TLS keys, kubeconfigs, cloud creds. **Read on secrets in a namespace ≈ the union of every identity whose token/cred lives there.** Specifically: - Read SA token Secrets → authenticate as those SAs → inherit their RBAC. - `create` on `serviceaccounts/token` (TokenRequest) or `create pods` with a privileged SA → mint/borrow a stronger identity. - `create`/`update` on `pods` lets you schedule a pod that *mounts a more-privileged SA* or mounts host paths/Secrets — pod-create is a classic lateral/escalation primitive. Treat broad `secrets` read, `pods` create, and `serviceaccounts/token` create as near-Secret-equivalent and near-impersonate-equivalent; scope them hard. Policy-engine CRDs belong in the same class: create/update on Kyverno (namespaced) policies runs attacker logic inside the admission-controller pod (`rules/03` §2, CVE-2026-4789). ### 2.6 The Helm-chart-grants-cluster-admin trap Charts and operators bundle their own RBAC. A values toggle like `rbac.clusterAdministrator: true`, `rbac.create: true` with a `*/*` ClusterRole, or a default ServiceAccount bound to `cluster-admin` hands the workload the cluster. **Always `helm template | grep -A20 -iE 'ClusterRole|ClusterRoleBinding'` before install** and read the rules (`rules/07` covers chart review). A logging agent does not need cluster-admin. ## 3. ServiceAccount hygiene & the token model - **`automountServiceAccountToken: false` by default** — on the ServiceAccount and/or pod spec. A pod that never calls the Kubernetes API should not carry a credential to it. Opt in only for pods that talk to the API. A mounted token + an SSRF/RCE in the pod = the attacker holds that SA's RBAC. ```yaml # GOOD — SA carries no token unless a pod explicitly needs it apiVersion: v1 kind: ServiceAccount metadata: { name: web, namespace: shop } automountServiceAccountToken: false --- apiVersion: v1 kind: Pod spec: serviceAccountName: web automountServiceAccountToken: true # explicit, only because this pod calls the API ``` - **Bound, projected, short-lived tokens are the model** (bound-SA-token volumes GA since 1.22). Pods get an auto-rotating, time-bound token (default ~1h) via a projected volume, audience-scoped, tied to the pod's lifetime. This is the default mount mechanism — good. - **No long-lived Secret-based SA tokens.** Since 1.24 the API server **no longer auto-creates** a forever-token Secret per ServiceAccount (GA 1.26). Do not manually create `kubernetes.io/service-account-token` Secrets for routine use — they are non-expiring, non-rotating bearer credentials that leak into logs/backups/etcd. If an external system needs an SA token, mint a **short-lived audience-scoped token** via the TokenRequest API (`kubectl create token sa --audience=... --duration=...`) and refresh it. - **Audience-scoped tokens**: a token minted for audience `vault` is rejected by the API server and by any verifier expecting a different audience — limits replay if leaked. - **One ServiceAccount per workload**, never the namespace `default` SA for real workloads, never shared across apps. The `default` SA should have an empty token mount and no bindings. ## 4. Auditing RBAC RBAC is allow-only and additive, so the real question is always "what is the transitive closure of what subject X can do, and can it escalate?" Tools: - **`kubectl auth can-i`** — point checks, including as another subject: ```bash kubectl auth can-i '*' '*' --as=system:serviceaccount:ci:deployer # cluster-admin? kubectl auth can-i create clusterrolebindings --as=... kubectl auth can-i list secrets -A --as=... ``` - **`kubectl auth whoami`** — confirm your own identity/groups. - **rbac-tool** (insights/aquasecurity), **krane**, **kubectl-who-can** (aquasecurity) — build the reverse index: *who* can `escalate`, `bind`, `impersonate`, read secrets, create pods, mint tokens. Run who-can on every escalation primitive: ```bash kubectl who-can create pods -A kubectl who-can '*' '*' kubectl who-can impersonate users ``` - **Hunt patterns** across the RBAC manifests in git / `kubectl get clusterroles -o yaml`: ```bash # wildcards in cluster roles kubectl get clusterroles -o json | jq -r '.items[]|select(.rules[]?|(.verbs[]?=="*") or (.resources[]?=="*"))|.metadata.name' # escalation verbs anywhere grep -rE 'escalate|impersonate|"bind"' rbac/ # cluster-admin bound to a ServiceAccount or broad group kubectl get clusterrolebindings -o json | jq -r '.items[]|select(.roleRef.name=="cluster-admin")|{name:.metadata.name,subjects:.subjects}' ``` - **Find dead subjects**: bindings to departed users / deleted SAs accrue silently — Low, but they're attack surface and noise. Reconcile against the identity source. ## Audit checklist - [ ] No wildcard `*` verbs/resources/apiGroups in any Role/ClusterRole bound to a workload, CI, or tenant? (`kubectl get clusterroles,roles -A -o json | jq` for `"*"`) - [ ] `escalate`/`bind`/`impersonate` granted only to a named, audited admin path — never automation? (`kubectl who-can escalate clusterroles`, etc.) - [ ] No `cluster-admin` (or any broad role) bound to a ServiceAccount, `system:authenticated`, or `system:unauthenticated`? (`kubectl get clusterrolebindings -o json | jq '...roleRef.name=="cluster-admin"'`) - [ ] Aggregated ClusterRoles reviewed — nothing unexpected aggregates into `admin`/`edit`/`view`? - [ ] Broad `secrets` read, `pods` create, `serviceaccounts/token` create, and policy-engine CRD writes (`rules/03`) scoped tightly (treated as escalation primitives)? - [ ] `automountServiceAccountToken: false` is the default; only API-calling pods opt in? (`grep -rL automountServiceAccountToken` deployments; check SA spec) - [ ] No manually-created long-lived `service-account-token` Secrets; external consumers use short-lived audience-scoped TokenRequest tokens? - [ ] One SA per workload, `default` SA unused/unbound, no SA shared across apps? - [ ] who-can run on every escalation primitive and the transitive closure for high-value SAs reviewed? - [ ] Bindings reconciled against the identity source — no dead users/SAs? -
03-admission-policy.md 11.1 KB
# 03 — Admission Control & Policy-as-Code Scope: the gate between "a manifest was submitted" and "the object exists in etcd." Pod Security Admission, the policy engines (Kyverno, Gatekeeper/OPA, ValidatingAdmission Policy/MutatingAdmissionPolicy), the AUDIT→ENFORCE rollout discipline, image verification at admission, and PolicyException hygiene. **Pod-level securityContext/seccomp field internals are `sota-sandboxing` (rules/03);** this file owns *enforcing them at admission*. Image *signing/provenance production* is `sota-devsecops` (rules/02); this file owns *verifying signatures at admission*. Admission runs after authn/authz, on the object: **validating** webhooks/policies accept or reject; **mutating** ones modify (inject sidecars, set defaults). Admission is where policy becomes enforcement — RBAC says *who*, admission says *what's allowed to exist*. --- ## 1. Pod Security Admission (PSA) — the built-in floor PSA is the in-tree replacement for the removed PodSecurityPolicy (PSA stable since K8s v1.25; PSP gone since 1.25). It enforces the three **Pod Security Standards** by namespace **label**, in three **modes**: - **Standards**: `privileged` (no restrictions), `baseline` (blocks known escalations: hostNetwork/PID/IPC, privileged, hostPath, most added caps), `restricted` (hardened: non-root, no privilege escalation, seccomp RuntimeDefault, drop ALL caps, etc.). - **Modes**: `enforce` (reject), `audit` (allow + audit-log annotation), `warn` (allow + client warning). You can set all three independently and pin a `-version`. ```yaml # GOOD — enforce restricted; audit/warn at the same level catch drift in subresources apiVersion: v1 kind: Namespace metadata: name: payments labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latest # or pin to your minor pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted ``` **Enforce `restricted` on workload namespaces.** `baseline` is a transitional floor, not a destination. **PSA's limits** (this is why you also need an engine, §2): - PSA is **per-namespace and standard-only** — it cannot express "images must come from our registry," "no `:latest` tag," "every pod has resource limits," or "the SA isn't `default`." It only checks the built-in pod-security fields. - It does **not mutate** — it won't add a missing `securityContext`, only reject. - Privileged-namespace exemptions (kube-system, some operators) are broad. Don't let a workload land in an exempt/privileged namespace to dodge PSA. The *meaning* of `runAsNonRoot`, `seccompProfile`, capability drops → `sota-sandboxing`. ## 2. Policy engines — choosing one You need a general engine for everything PSA can't express. The 2026 options: | Option | What it is | Use when | |---|---|---| | **ValidatingAdmissionPolicy (VAP)** | In-tree, CEL-based validating policies. GA since K8s **v1.30**. No external webhook/pod — runs in the API server. | The check is *validation only* and expressible in CEL. Lowest operational risk (no webhook to fail, no extra pod). Prefer for simple invariants. | | **MutatingAdmissionPolicy (MAP)** | In-tree, CEL-based *mutating* policies. GA in K8s **v1.36** (beta 1.34, feature-gated). | Mutation you'd otherwise run a mutating webhook for, once you're on a GA-supporting version. Verify your cluster's version before relying on it. | | **Kyverno** | YAML-native policy engine (validate/mutate/generate/verifyImages/cleanup). Current line ~v1.18 (verify). Runs as an admission webhook + controllers. | You want readable, K8s-native policies, image verification, resource generation, and don't want to write Rego. Most teams' default. | | **Gatekeeper (OPA)** | OPA/Rego policies via ConstraintTemplates + Constraints. Current line ~v3.22 (verify). Webhook + audit controller; can also generate VAPs. | You already use Rego/OPA org-wide, or need very expressive logic. Heavier; Rego learning curve. | Guidance: **use VAP for what it cleanly covers** (it's the lowest-risk, in-tree option), and one of **Kyverno / Gatekeeper** for the rest (image verification, mutation, generation, cross-resource logic). Don't run all three doing overlapping things. Whatever you pick: policies live in git, are GitOps-deployed (`rules/04`), and have **CI-tested deny cases** (kyverno CLI `kyverno test`, gatekeeper-library tests, VAP unit tests) so an "improvement" can't silently stop blocking. **Policy objects are an escalation primitive — never delegate their creation to tenants.** Policies execute inside the engine's privileged pod: CVE-2026-4789 (CVSS 9.8, disclosed 2026-03) let any author of a Kyverno `NamespacedValidatingPolicy` use the CEL `http.Get()`/`http.Post()` library to make arbitrary HTTP requests *from the Kyverno admission-controller pod* — reaching cloud metadata (169.254.169.254) and internal services, bypassing RBAC; it was one of a 2026 wave of Kyverno SSRF/apiCall advisories. So: treat create/update on (namespaced) policy CRDs like `bind`/`escalate` (`rules/02` §2.5); put an egress NetworkPolicy on policy-engine pods (block the metadata endpoint and all unneeded egress); and confirm the running line includes the CVE-2026-4789 fix (merged 2026-04; patched releases per the project's security advisories). ```yaml # Kyverno — require resource limits AND block :latest (PSA cannot do either) apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: { name: workload-baseline } spec: validationFailureAction: Enforce # NOT Audit (see §3) background: true rules: - name: require-limits match: { any: [{ resources: { kinds: ["Pod"] } }] } validate: message: "containers must set cpu/memory limits" pattern: spec: { containers: [{ resources: { limits: { memory: "?*", cpu: "?*" } } }] } - name: disallow-latest match: { any: [{ resources: { kinds: ["Pod"] } }] } validate: message: ":latest tag is not allowed; pin a digest" pattern: spec: { containers: [{ image: "!*:latest" }] } ``` ## 3. The AUDIT→ENFORCE rollout discipline (and the trap) New policies break workloads if you enforce blind. The discipline: 1. **Deploy in audit/warn** (`validationFailureAction: Audit` / PSA `audit`+`warn`, Gatekeeper `enforcementAction: dryrun`). 2. **Watch the audit signal** — collect every would-be violation, fix the workloads (or add a *scoped* exception, §5). 3. **Flip to enforce** once the violation rate is zero/known. Then enforce stays on. **The trap (a real, recurring audit finding):** policies left in **Audit/dryrun forever**. Audit-only is not a control — it produces a dashboard nobody reads while violations sail through. Treat "policy exists but never enforces" as **High**: the org believes it's protected and isn't. For each Audit-mode policy, demand a flip-date and an owner, or downgrade it to honest "we don't enforce this." Hunt: `grep -rE 'validationFailureAction:\s*Audit|enforcementAction:\s*dryrun' policies/` and `kubectl get ns -o json | jq '.items[].metadata.labels | select(.["pod-security. kubernetes.io/enforce"]==null)'` (namespaces with warn/audit but no enforce). ## 4. Image verification at admission Signing an image (cosign, `sota-devsecops` rules/02) does nothing unless **admission verifies the signature and rejects unsigned/unattested images**. Use Kyverno `verifyImages` or the sigstore **policy-controller**: ```yaml # Kyverno verifyImages — Enforce, keyless, exact identity, mutate tag→digest apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: { name: verify-signed-images } spec: validationFailureAction: Enforce rules: - name: verify match: { any: [{ resources: { kinds: ["Pod"] } }] } verifyImages: - imageReferences: ["registry.example.com/*"] mutateDigest: true # pin the verified digest into the spec required: true attestors: - entries: - keyless: issuer: "https://token.actions.githubusercontent.com" subject: "https://github.com/org/repo/.github/workflows/release.yml@refs/heads/main" ``` Traps: - **"Signing exists but admission only Audits it."** Same as §3 — unsigned images still run. High. Enforce, or it's theater. - **Prereq ordering**: enforce *can block scheduling* if your build/sign path isn't complete for every in-use image (base images, third-party charts, kube-system). Inventory what's deployed, get everything signed/allowlisted, *then* enforce — or you'll wedge the cluster. Scope `imageReferences` to registries you control and allowlist the rest explicitly (with expiry) rather than disabling enforcement. - Verify the exact **issuer + subject identity**, not just "a signature exists" — an attacker's valid signature from their own identity passes a check that doesn't pin who. - Require **provenance attestations** (SLSA) too, not only signatures, for high-value workloads (`sota-devsecops` rules/02). ## 5. PolicyException / exemption discipline Every engine has an escape hatch (Kyverno `PolicyException`, Gatekeeper Constraint `excludedNamespaces`/`match`, VAP `matchConditions`, PSA namespace exemptions). Without discipline these become permanent holes: - **Scoped**: exact namespace + resource + rule, never cluster-wide "skip this policy." - **Owned and time-bound**: an owner annotation and an expiry; CI/cron fails or alerts on expired exceptions. - **PR-reviewed and inventoried**: exceptions live in git, are reviewed like code, and a query lists all live ones. An untracked, unexpiring exception is how "we enforce restricted" quietly becomes "except in these 30 namespaces." ## Audit checklist - [ ] PSA `enforce: restricted` (or justified `baseline`) on every workload namespace, not just `warn`/`audit`? (`kubectl get ns -L pod-security.kubernetes.io/enforce`) - [ ] No workloads parked in privileged/exempt namespaces (kube-system, operator ns) to dodge PSA? - [ ] A policy engine covers what PSA can't (registry allowlist, no `:latest`, required limits, non-default SA, host-path/host-namespace bans)? - [ ] Engine choice sane (VAP for simple validation; Kyverno/Gatekeeper for the rest; not three overlapping)? Version current/supported? - [ ] Policy CRD writes (incl. Kyverno namespaced policies) never delegated to tenants; policy-engine pods behind an egress NetworkPolicy (metadata endpoint blocked); running line includes the CVE-2026-4789 fix? - [ ] All security policies in `Enforce`, not parked in `Audit`/`dryrun` indefinitely? (`grep -rE 'Audit|dryrun' policies/`) Each Audit-mode policy has a flip-date + owner? - [ ] Image verification ENFORCED for controlled registries: exact signer issuer+subject, provenance required, tag→digest mutation, full image inventory covered before enforce? (not Audit-only) - [ ] Policies in git, GitOps-deployed, with CI-tested deny cases (`kyverno test` / gatekeeper tests / VAP units)? - [ ] Exceptions scoped, owned, time-bound, PR-reviewed, inventoried, expiry-alerted? (`kubectl get polex -A`; list Gatekeeper excludedNamespaces) - [ ] Mutating policies/webhooks reviewed for what they inject (a hostile mutation adds a sidecar/hostPath) — see `rules/05`? -
04-gitops-controllers.md 14.3 KB
# 04 — GitOps Controller Security (Argo CD, Flux) Scope: the controller that reconciles git into the cluster — Argo CD and Flux. Its own privileges and self-management risk, project/tenant scoping (the `clusterResourceWhitelist` escalation trap), SSO/RBAC, repo/credential scoping, ApplicationSet/templating injection, sync strategy, drift, recent CVEs, and promotion/rollback as git operations. CI/CD *pipeline* and deployment *strategy* are `sota-devsecops` (rules/06); this file owns the in-cluster controller's security model. SSO/OIDC *design* is `sota-identity-access`. **Core principle:** GitOps is the *only* write path to the cluster. Humans don't `kubectl apply` to prod — they open a PR; the controller reconciles the merged state. This makes git the audit log and review gate, and makes the **controller a cluster-admin-class component** whose compromise = cluster compromise. Scope it accordingly. --- ## 1. The controller is privileged — treat it as crown jewels Argo CD's application-controller / Flux's controllers apply arbitrary manifests, so they hold broad (often cluster-admin-equivalent) RBAC by design. Therefore: - **The git repo + the controller's RBAC + who can change either = your real attack surface.** Anyone who can merge to the watched branch, or who can edit the controller's config, can deploy anything the controller can apply. - **Branch protection + required review + signed commits** on GitOps repos are not optional (`sota-devsecops` rules/01). A self-approved merge to `main` is a deploy. - **Self-management risk**: if the controller manages *its own* manifests (RBAC, config, the App that defines other Apps — "app-of-apps"), a malicious PR can widen its own privileges or repoint it at a hostile repo. Put the controller's own bootstrap under extra-tight review, or manage it out-of-band. - Run the controller in its own namespace; restrict who has RBAC *to* Argo CD/Flux resources (Applications, AppProjects, GitRepositories, Kustomizations) — editing an Application is editing a deployment. ## 2. Argo CD AppProject scoping — the `clusterResourceWhitelist` trap `AppProject` is Argo CD's tenancy/blast-radius boundary. It constrains which repos, which destination clusters/namespaces, and which **resource kinds** an Application may deploy. The traps: ```yaml # BAD — a project that can deploy anything, anywhere, from any repo apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: { name: team-a, namespace: argocd } spec: sourceRepos: ["*"] destinations: [{ server: "*", namespace: "*" }] clusterResourceWhitelist: [{ group: "*", kind: "*" }] # can create ClusterRoles, etc. ``` - **`clusterResourceWhitelist: [{group:'*',kind:'*'}]`** lets the project create *cluster- scoped* resources of any kind — including `ClusterRole`/`ClusterRoleBinding`. A tenant who can commit to that project's repo can bind themselves cluster-admin via a manifest. **Combined with broad SSO/RBAC into the project, this is Critical.** Allow only the specific cluster-scoped kinds the tenant legitimately needs (often *none* — leave it empty/unset so cluster resources are denied). - **`sourceRepos: ["*"]`** lets the project deploy from any repo, defeating provenance. Pin to the exact repos. - **`destinations` with `*`** lets one project write to every cluster/namespace. Pin to the tenant's clusters and namespaces. - The **`default` AppProject is wide open** — don't run production Apps in it; lock it down (empty sourceRepos/destinations) and use named projects. - Use `namespaceResourceBlacklist` / `clusterResourceBlacklist` to forbid dangerous kinds (e.g. `ResourceQuota`, `LimitRange`, RBAC) even within an otherwise-scoped project. ```yaml # GOOD — scoped project; no cluster-scoped kinds, pinned repo + destination spec: sourceRepos: ["https://github.com/org/team-a-config.git"] destinations: [{ server: "https://kubernetes.default.svc", namespace: "team-a-*" }] clusterResourceWhitelist: [] # cluster-scoped resources denied namespaceResourceBlacklist: [{ group: "rbac.authorization.k8s.io", kind: "*" }] ``` ## 3. SSO, RBAC, and credentials - **SSO over local accounts.** Disable the built-in `admin` account in prod (`admin. enabled: false`) after bootstrap, or scope it to break-glass; rotate its secret. Wire OIDC/SSO (design → `sota-identity-access`). - **Argo CD RBAC** (`policy.csv`): map SSO groups to roles; default policy `role:''` (no access). Scope roles to specific projects/Applications — not `role:admin` for everyone. The `g, <group>, role:admin` line is the equivalent of a cluster-admin binding. - **Repository credentials are secrets**: scope read-only deploy keys / fine-grained PATs per-repo, store them as Argo CD repo Secrets (or via ESO — `sota-secrets-management`), never embed write creds. A controller with org-write git creds can poison source. - **Cluster credentials**: when one Argo CD manages multiple clusters, each cluster credential is a foothold; scope and rotate them. ## 4. ApplicationSet & templating injection ApplicationSet generates Applications from generators (git directories, PR/SCM, lists, clusters). Generated fields are **templated** — untrusted input in a generator can inject into the Application spec: - **PR/SCM generators reading fork PRs** can let an external contributor influence generated Applications (repo URL, path, namespace) — treat like the `pull_request_target` problem (`sota-devsecops` rules/01). Restrict generators to trusted repos/branches; don't template attacker-controlled fields into `project`, `destination`, or `source`. - Keep ApplicationSet `templatePatch`/Go-template values constrained; an injected `project:` field can move an App into a more-privileged AppProject. ## 5. Sync strategy & drift - **Auto-sync vs sync-with-approval**: auto-sync makes a merge an immediate deploy — good for low-risk envs, paired with strong PR review. For prod, gate with manual sync / **sync windows** / approval so a merge doesn't instantly hit prod without a release step. - **`selfHeal: true`** reverts out-of-band `kubectl` changes back to git — this is the point of GitOps (it kills drift and unaudited changes). Pair with **`prune: true`** carefully (prune deletes resources removed from git; a bad git change can delete prod). - **Drift detection is a security signal**: an OutOfSync resource nobody changed in git means someone wrote to the cluster directly (or a controller is fighting). Alert on unexpected drift; investigate it as a possible intrusion, not just noise. - **Promotion and rollback are git operations**: promote by moving a verified digest through environment overlays/branches (build-once-promote-many, `sota-devsecops` rules/06); roll back by reverting the git commit, not by hand-editing the cluster. ## 6. Patch the controller — recent Argo CD CVEs The GitOps controller is high-value and has a live CVE stream; track and patch it like the control plane (`rules/01` §7). Recent examples (verify against the Argo CD security advisories before citing exact IDs/versions): - **GHSA-3v3m-wc6v-x4x3 (CVE-2026-42880), ~May 2026, Critical** — Kubernetes Secret extraction via the ServerSideDiff feature: a low-privilege/read-only user could obtain plaintext Secret data. Patched in the then-current 3.2.x/3.3.x patch releases; upgrade. - **GHSA-786q-9hcg-v9ff (CVE-2025-55190), Sep 2025, Critical (CVSS ~10)** — project API tokens with even `get` permission could retrieve repository credentials. Patched in 2.13.9 / 2.14.16 / 3.0.14 / 3.1.2. - Plus assorted webhook-parser DoS and stored-XSS (annotation) advisories. Current Argo CD line is ~3.4.x (verify). Stay on a supported minor, watch the advisories feed, and patch Critical auth/secret-exposure issues on the emergency track. Flux likewise publishes advisories — track its controllers' CVEs. ## 7. Write-back controllers — the success log is not the commit §1–§6 cover what the controller may do *inward*: git → cluster. A controller that also writes **outward** — an image-update controller, a PR bot, config sync committing rendered manifests back — is an instrument reporting on itself, so `sota-code-security` rules/15 §2 applies to it in full. Its log describes the update it *decided* to make, not the write landing. **R7.1 — Verify at the remote, never from the controller's own counters.** `git ls-remote` the branch, read the file on `origin`, or diff the rendered manifest. A line like `images_considered=2 images_skipped=0 images_updated=1 errors=0` reports decisions taken in memory, and is emitted independently of any commit. Measured (2026-08-18, a GitOps image-update controller on a private cluster). A second image alias was added to one custom resource so that two images built from the same commit would move in lockstep. The controller then logged `Committing 1 parameter update(s)` and `Successfully updated the live application spec` **every reconcile cycle for ~15 minutes across ~7 cycles**, while making no `git commit` or `git push` at all — nothing in its own log, and the remote branch never moved. Net effect: the primary application silently stopped auto-deploying and the two images skewed apart, which is the exact invariant the second alias existed to guarantee. Removing the alias, with the primary image deliberately left on the old tag so the next reconcile was a clean experiment, produced a commit within **84 seconds** — that is what established cause rather than correlation. The controller-side root cause was never found and did not need to be: the operator has to check the remote, not explain the bug. **R7.2 — A per-object counter counts decisions; an aggregate hides the failure.** Both traps are in that incident. `updated=1` counts what the reconciler resolved to do. And where a controller manages N objects, one object's success masks another's silent failure in any rolled-up total — the write kept working for one image and stopped for the other while the aggregate stayed healthy. **Assert on all N, not on the sum.** Alert on the **age of the last observed remote change**, which goes stale by itself when writes stop; a counter of successes never will. **R7.3 — This is a liveness gap, not an integrity one.** Nothing was corrupted and no alarm was wrong: a write that should have happened did not, and every signal reporting on it was a signal about intent. Freshness is the only observable that degrades when an outbound write stops — the same distinction `sota-code-security` rules/04 §8 draws between integrity and completeness for audit ledgers. ## 7a. What you wrote is not what lands — verify a rendered object by rendering An overlay, a transformer or a templating layer sits between the file you edited and the object the controller applies, and **the field you wrote can be overridden without a warning**. The common one: a manifest declares `namespace: monitoring`, the kustomization sets `namespace: argo-workflows`, and the object lands in the second. The tell is a diagnostic that reads like a *different* failure — `kubectl get <kind> -n monitoring` returns nothing, which reads as **"it was never applied"** rather than **"you are looking in the wrong place"**, and the next hour goes into re-applying something that already exists. - **Render, then grep the render** — `kustomize build .`, `helm template`, the controller's own diff/preview — and confirm the field's *final* value before you use it to look something up. Same discipline as reproducing a gate's exact invocation (`sota-devsecops` rules/09 §5): what you believe you ran and what ran are two questions. - **Ask the cluster where the object actually is** (`kubectl get <kind> -A`) before concluding it is absent. An empty namespaced query is an absence claim, and it carries the burden of one (`sota/rules/03` §2). - **Deleting a GitOps-managed object means removing it from the rendered set**, not deleting its file. The controller prunes what it no longer renders, so dropping the reference converges the cluster immediately and the file tidy-up can follow under normal review. Useful when file deletion is blocked; it also means an object can be *functionally* gone while its manifest is still in the repository. ## Audit checklist - [ ] **Is any conclusion drawn from a field an overlay can override?** (§7a) Namespace, name prefixes, labels and images are transformer-owned — render (`kustomize build`, `helm template`) and grep the render before treating an empty namespaced lookup as "not applied"; `kubectl get <kind> -A` before any absence claim. - [ ] GitOps is the only write path to prod (no routine human `kubectl apply`); GitOps repos have branch protection + required review + signed commits? - [ ] Controller runs least-privilege where possible; RBAC *to* Argo CD/Flux resources (Applications/AppProjects/Kustomizations) is restricted; self-management/bootstrap under extra review? - [ ] No AppProject with `clusterResourceWhitelist: [{group:'*',kind:'*'}]` (or it's justified + tightly access-controlled)? (`kubectl get appprojects -A -o yaml | grep -A3 clusterResourceWhitelist`) - [ ] AppProjects pin `sourceRepos` and `destinations` (no `*`); `default` project locked down; RBAC-creating kinds blacklisted where not needed? - [ ] Argo CD SSO wired, built-in admin disabled/break-glass, RBAC `policy.csv` scoped per project (no blanket `role:admin`)? - [ ] Repo/cluster credentials scoped (read-only deploy keys, per-repo), stored as secrets/ESO, rotated — no org-write creds in the controller? - [ ] ApplicationSet generators restricted to trusted repos/branches; no untrusted input templated into `project`/`destination`/`source`? - [ ] Prod sync gated (approval/sync windows), not blind auto-sync; `prune`/`selfHeal` reviewed; drift alerted and triaged as a security signal? - [ ] Promotion = move verified digest through git; rollback = git revert (not hand-edits)? - [ ] Argo CD/Flux on a supported version, CVE feed tracked, recent Critical advisories (Secret-extraction, cred-exposure) patched? (`argocd version`) - [ ] Any controller that writes **outward** (image updater, PR bot, config sync committing back) is verified **at the remote** — `git ls-remote` / the file on `origin` / a rendered-manifest diff — not from its own `updated=N errors=0`; freshness of the last remote change is alerted, and a controller managing N objects asserts on all N rather than an aggregate? (§7) -
05-operators-crds-webhooks.md 8.4 KB
# 05 — Operators, CRDs & Admission Webhooks Scope: extension components that run with cluster privilege — operators (controllers reconciling CRDs), the CRDs themselves, and admission webhooks (validating/mutating). These are the third-party code you grant standing power inside the cluster; each is both a control and an attack surface. Trusted-controller examples: ExternalSecrets Operator (ESO), cert-manager. Secret *backends/ESO usage* is `sota-secrets-management` (rules/02); this file owns *vetting the operator's cluster privilege and webhook surface*. --- ## 1. The operator privilege problem An operator is a long-running controller with a ServiceAccount that, by design, holds broad RBAC: it watches and mutates the resources it manages, often cluster-wide, often including **Secrets, RBAC objects, or pods**. That makes every installed operator a standing, privileged principal — and **CRD-mediated escalation** the recurring risk: - The operator reads a CRD (a custom resource you or a tenant created) and *acts* with its own (broad) privileges. So **anyone who can create/edit that CRD can often induce the operator to do something privileged on their behalf** — a confused-deputy. Example: a CRD field that becomes a pod's `serviceAccountName`, `hostPath` mount, image, or RBAC binding lets a CRD-author escalate via the operator's hands. - An operator with `clusterroles`/`clusterrolebindings` write, or `secrets` cluster-wide, or `pods` create with arbitrary SA, is effectively cluster-admin-adjacent. Owning the operator pod (its image, its dependencies, an RCE) = owning that power. **Mitigations:** - **Vet the operator's RBAC before install.** `helm template` / read the bundled ClusterRole (`rules/07`, `rules/02`). Reject or scope wildcard verbs, cluster-wide `secrets:*`, and RBAC-write unless the operator genuinely needs them. Prefer **namespaced-scoped operators** (watch one/few namespaces via OLM `OperatorGroup` or the operator's `--namespace` flag) over cluster-wide where the operator supports it. - **Limit who can create the operator's CRs.** RBAC on the CRD's API group is an escalation primitive — treat `create`/`update` on a CRD whose fields drive pods/RBAC/ secrets like `pods` create (`rules/02` §2.5). Don't hand tenants CR-create on a CRD the operator turns into privileged actions. - **Pin and patch the operator** like any privileged software (image digest, CVE feed). - **Watch what the operator mutates**: an operator that injects sidecars or modifies pod specs is doing admission-time mutation — audit it like a mutating webhook (§3). Trusted, widely-used operators still need scoping: **ESO** holds creds to your secret backend and writes Secrets — scope its `SecretStore`/`ClusterSecretStore` and the namespaces it serves; **cert-manager** can issue certs and holds issuer credentials — scope `ClusterIssuer` usage and ACME/CA access. "Popular" is not "harmless when over-privileged." ## 2. CRD validation & security CRDs extend the API; a sloppy CRD is an injection and DoS vector into the operator. - **Structural schema with strict validation.** Define `openAPIV3Schema` with types, enums, patterns, and `x-kubernetes-validations` (CEL) for cross-field rules. A CRD that accepts arbitrary fields (`x-kubernetes-preserve-unknown-fields: true`) lets attackers smuggle data the operator may mishandle — avoid except where genuinely needed. - **Validate fields that become privileged.** If a CR field becomes a container image, an SA name, a host path, a command, or an RBAC subject, the CRD schema (and/or an admission policy) must constrain it — don't trust the operator to sanitize. - **CRD scope**: cluster-scoped CRDs are visible/creatable per cluster-wide RBAC; namespaced CRDs are easier to delegate safely. Prefer namespaced unless the resource is genuinely global. - **Conversion webhooks** (for multi-version CRDs) are admission-webhook-class surface — same hardening as §3. - Removing/replacing a CRD with `deletionPolicy` implications can cascade-delete CRs and the operator-managed resources — review before applying CRD changes via GitOps. ## 3. Admission webhooks as attack surface AND single point of failure Validating/mutating webhooks are external HTTPS endpoints the API server calls on every matching request. They are powerful (they can reject or rewrite any object) and fragile (if they're down or misconfigured, they can wedge the cluster). Configure deliberately: ```yaml # Knobs that matter on a (Validating|Mutating)WebhookConfiguration webhooks: - name: policy.example.com failurePolicy: Fail # Fail = secure (reject if webhook down) for SECURITY gates # Ignore = available (allow if down) — only for non-security mutators timeoutSeconds: 5 # short; a slow/hung webhook stalls every API write namespaceSelector: # EXCLUDE kube-system / the webhook's own ns to avoid deadlock matchExpressions: - { key: kubernetes.io/metadata.name, operator: NotIn, values: ["kube-system","webhook-system"] } matchPolicy: Equivalent sideEffects: None admissionReviewVersions: ["v1"] clientConfig: caBundle: <pinned CA> # TLS verified; rotate before expiry ``` The hard tradeoffs and traps: - **`failurePolicy`**: `Fail` is correct for **security** webhooks (an unavailable policy engine must not mean "allow everything") — but a `Fail` webhook that covers its own namespace or kube-system can **deadlock the cluster** (you can't restart the webhook because admission needs the webhook). Always set a `namespaceSelector`/`objectSelector` that **excludes the webhook's own namespace and kube-system**. Run the webhook HA (multiple replicas, PDB — `rules/06`) so `Fail` doesn't take you down on a single pod restart. Non-security convenience mutators may use `Ignore`, but know that means fail-open. - **`timeoutSeconds`**: keep it small (≤5–10s). Every matching API write blocks on the webhook; a hung webhook is a cluster-wide write outage. - **TLS**: the `caBundle` must be valid and **rotated before expiry** — an expired webhook cert with `failurePolicy: Fail` is a self-inflicted outage; with `Ignore` it silently disables your control. Use cert-manager (and its CA-injector) to manage and rotate webhook certs; alert on cert expiry. - **Scope (`rules`/`namespaceSelector`/`objectSelector`)**: match only what you must. An over-broad webhook intercepting every resource is latency + blast radius. A *malicious or compromised* webhook scoped to `pods`/`*` can read every spec (data exfil) and a mutating one can inject sidecars, hostPath mounts, or credentials into every pod — so **who can create/edit WebhookConfigurations is an escalation primitive** (it's cluster-wide RBAC; guard `admissionregistration.k8s.io` writes like RBAC writes, `rules/02`). - Prefer **in-tree CEL policies (ValidatingAdmissionPolicy)** over a webhook where the logic fits (`rules/03` §2) — no external endpoint means no webhook outage/attack surface. ## Audit checklist - [ ] Every installed operator's RBAC reviewed; no unjustified wildcard verbs, cluster-wide `secrets:*`, RBAC-write, or `pods` create with arbitrary SA? (`kubectl get clusterrole -o yaml` for each operator SA; trace its ClusterRoleBinding) - [ ] Operators run namespaced-scoped where supported, not cluster-wide by default? - [ ] RBAC to create/edit operator CRDs is restricted (treated as an escalation primitive when the CRD drives pods/RBAC/secrets)? - [ ] CRDs have structural schemas with validation (`x-kubernetes-validations`), no needless `preserve-unknown-fields`; privileged-becoming fields constrained? - [ ] Trusted controllers (ESO, cert-manager) scoped — SecretStore/Issuer usage and served namespaces limited, backend creds tight? - [ ] Operators pinned by digest and on a tracked CVE/patch cadence? - [ ] Security webhooks `failurePolicy: Fail` BUT with `namespaceSelector` excluding kube-system and the webhook's own ns; webhook runs HA + PDB? - [ ] Webhook `timeoutSeconds` small; webhook scope (`rules`/selectors) minimal, not `*`? - [ ] Webhook TLS `caBundle` valid, managed by cert-manager, rotated before expiry, expiry-alerted? - [ ] Writes to `admissionregistration.k8s.io` (WebhookConfigurations) guarded like RBAC writes? (`kubectl who-can create validatingwebhookconfigurations`) - [ ] Logic that fits CEL moved to ValidatingAdmissionPolicy instead of a webhook where practical? -
06-workloads-tenancy.md 8 KB
# 06 — Workloads, Scheduling Availability & Multi-Tenancy Scope: the platform-level properties of workloads that affect **availability and tenant isolation** — resource requests/limits as QoS (not isolation), PodDisruptionBudgets, topology spread / anti-affinity, priority/preemption, the namespace-as-soft-boundary reality and when you need hard multi-tenancy, and where Secrets live. **Pod isolation mechanics (securityContext, seccomp, runtime classes) are `sota-sandboxing` (rules/03);** this file references the *isolation decision* but does not re-spec the pod fields. Secret *backends* are `sota-secrets-management`; this file states the etcd-encryption requirement and points there. --- ## 1. Resource requests/limits — availability + QoS, NOT isolation Requests/limits govern scheduling and the cgroup budget; they are an **availability and fairness** control, not a security boundary (a shared kernel means a noisy/hostile neighbor can still affect others — real isolation is `sota-sandboxing`'s node/runtime story). - **Set both `requests` and `limits`** for CPU and memory on every container. - `requests` drive scheduling (the scheduler packs to requested capacity). - **Memory `limit` is a hard kill**: exceed it → OOMKill. Set it from real usage + headroom; too low = crashloop, too high = node memory overcommit and node-level OOM that evicts *other* pods. - **CPU `limit` throttles** (CFS), it doesn't kill — and aggressive CPU limits cause latency cliffs. Many shops set CPU *requests* and omit CPU *limits* (or set generous ones) to avoid throttling while keeping scheduling fairness; decide deliberately. - **QoS classes** follow from this: **Guaranteed** (requests == limits for all containers) is evicted last; **Burstable** (requests < limits) next; **BestEffort** (none set) is evicted first under node pressure. Critical workloads should be Guaranteed. - **A pod with no requests is a scheduling and eviction hazard** — it can be packed onto a full node and is first to be evicted. Enforce limits/requests at admission (`rules/03`) and set namespace **`LimitRange`** (defaults) + **`ResourceQuota`** (caps) so a tenant can't starve the cluster or schedule unbounded pods. ## 2. Disruption, spread & priority — staying up - **PodDisruptionBudget (PDB)** on every multi-replica critical workload: caps how many pods *voluntary* disruptions (node drain/upgrade) may take down at once (`minAvailable`/`maxUnavailable`). Without a PDB, a node drain during an upgrade can take all replicas at once → outage. (PDBs don't protect against involuntary disruptions like node crashes — that's replicas + spread.) ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget spec: minAvailable: 2 # or maxUnavailable: 1 selector: { matchLabels: { app: api } } ``` - **Topology spread constraints** spread replicas across zones/nodes so one failure domain loss doesn't kill the service: ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: { matchLabels: { app: api } } ``` - **Pod anti-affinity** (`podAntiAffinity`) keeps replicas off the same node — use `requiredDuringScheduling` for hard, `preferred` for soft. Topology spread is usually the cleaner modern tool; don't double-specify conflicting rules. - **PriorityClass & preemption**: assign higher `priorityClassName` to platform-critical pods so they preempt lower-priority workloads under contention. Beware: a too-high default priority on tenant workloads lets a tenant preempt platform pods — reserve top priorities for the platform, and consider `preemptionPolicy: Never` for non-urgent batch. ## 3. Multi-tenancy — namespace is a SOFT boundary The most common dangerous assumption: **a Namespace is not a security boundary.** It scopes names and RBAC, but tenants in different namespaces still share **the same nodes, kernel, control plane, CNI, and CRDs**. A container escape, a kernel CVE, a node-level secret read, or a cluster-scoped resource crosses namespaces. So: - **Soft multi-tenancy (namespace-per-tenant)** is appropriate only for **mutually trusted or low-sensitivity** tenants (teams in one org). Even then, enforce per-namespace: RBAC isolation (`rules/02`), `ResourceQuota`/`LimitRange`, PSA `restricted` (`rules/03`), and **default-deny NetworkPolicy** — note the requirement here, but NetworkPolicy/CNI depth is `sota-network-security`. - **When you need HARD multi-tenancy** (untrusted tenants, strong compliance/regulatory separation, hostile-neighbor model), namespaces are insufficient. Escalate the boundary: - **Separate clusters per tenant** — the strongest, simplest-to-reason-about boundary; no shared control plane/etcd. Default for genuinely untrusted tenants. - **vCluster** (virtual clusters) — each tenant gets its own API server/control plane syncing into a host namespace; far stronger control-plane isolation than a namespace, cheaper than separate clusters. Good middle ground; the *data plane* still shares host nodes unless combined with node isolation. - **Node isolation** — dedicate node pools per tenant (taints/tolerations + node affinity) so workloads don't co-reside; combine with a **stronger runtime boundary (gVisor/Kata)** for untrusted code — this is `sota-sandboxing` (rules/01 boundary choice, rules/03 runtime classes). Co-residency of untrusted tenants on a shared kernel is the thing node isolation + microVM/gVisor exists to fix. State the tenancy model and its boundary explicitly in design; "we use namespaces for isolation" against an untrusted tenant is a **High** finding. ## 4. Secrets in Kubernetes (platform view) - **etcd encryption-at-rest with KMS v2 is mandatory** — K8s Secrets are base64, not encrypted, by default (full treatment in `rules/01` §4). - **Don't inject secrets as plain env vars** where avoidable — env is readable via `/proc`, crash dumps, `kubectl describe`, and child processes; prefer **mounted files** (projected/volume) and tools that support file-based secrets. (App-level secret handling is `sota-secrets-management`.) - **Use a real secrets workflow**: **ExternalSecrets Operator** (sync from Vault/cloud secret managers — `sota-secrets-management` rules/02, and scope it per `rules/05`), **sealed-secrets** (encrypt secrets safe to commit to git for GitOps), or the **Secrets Store CSI driver** (mount from an external store, optionally never landing in etcd). Pick per your GitOps model; the goal is that the source of truth is an encrypted store, not a plaintext manifest in git. - **Restrict who can read Secrets** (`rules/02` §2.5) — `get`/`list secrets` is near-credential-equivalent. Audit it. ## Audit checklist - [ ] Every container sets memory requests+limits and CPU requests (CPU limits a deliberate decision)? Critical workloads Guaranteed QoS? (`kubectl get pods -A -o json | jq '..|.resources? // empty'`) - [ ] Namespaces have `ResourceQuota` + `LimitRange` so a tenant can't starve the cluster or schedule unbounded pods? (`kubectl get resourcequota,limitrange -A`) - [ ] Multi-replica critical workloads have a PDB; PDB values won't block legitimate drains or allow full takedown? (`kubectl get pdb -A`) - [ ] Topology spread / anti-affinity spreads replicas across zones/nodes (no single-node single-zone critical service)? - [ ] PriorityClasses reserve top priorities for the platform; tenants can't preempt platform pods? - [ ] Tenancy model stated explicitly: is "namespace" being relied on as a security boundary against untrusted tenants? (that's a High finding — escalate to vCluster / separate clusters / node isolation + gVisor/Kata per sota-sandboxing) - [ ] Soft-tenant namespaces enforce RBAC isolation, quota, PSA restricted, and default-deny NetworkPolicy (depth → sota-network-security)? - [ ] etcd Secrets encrypted with KMS v2 (rules/01); secrets mounted as files not env where possible; ESO/sealed-secrets/CSI used over plaintext-in-git (sota-secrets-management)? - [ ] Secret read RBAC scoped and audited (rules/02)? -
07-supply-chain-audit.md 8.7 KB
# 07 — Helm/Kustomize Supply Chain & K8s Audit Logging Scope: two platform concerns the other files lean on — (a) reviewing what a chart/overlay actually applies before it hits the cluster, plus chart/image provenance enforced **at admission**; and (b) producing and shipping the Kubernetes API audit log. Image *signing production* and *CI provenance* are `sota-devsecops` (rules/02, rules/04); admission *verification mechanics* are `rules/03`; **detection content on the audit stream and runtime detection (Falco/Tetragon) are `sota-detection-engineering`** — this file owns *generating and shipping* the stream, not writing the detections. --- ## 1. `helm template` review before apply A Helm chart is arbitrary templated YAML that becomes live cluster objects. Treat installing a third-party chart like running third-party code with the controller's privileges. **Always render and review before apply** — never `helm install` an unread chart into a privileged cluster: ```bash helm template rel chart/ -f values.yaml > rendered.yaml # what RBAC does this chart grant? grep -nE 'kind:\s*(ClusterRole|ClusterRoleBinding|Role|RoleBinding)' rendered.yaml # privilege-granting values / specs grep -nE 'privileged:\s*true|hostPID|hostNetwork|hostPath|cluster-admin|automountServiceAccountToken:\s*true|"\*"' rendered.yaml # images and tags (are they pinned? from your registry?) grep -nE 'image:' rendered.yaml ``` Then diff against the running state (`helm diff upgrade`, or render-and-`kubectl diff`) so an upgrade can't silently widen privileges. In GitOps (`rules/04`), the rendered output is what the controller applies — review the chart version bump like code. ## 2. The RBAC-in-charts / privilege-values trap (Cross-ref `rules/02` §2.6.) Charts ship their own RBAC and expose values that grant privilege. The recurring traps: - **`rbac.create: true` paired with a `*/*` ClusterRole**, or a values toggle like `rbac.clusterAdministrator: true` / `clusterRole.rules: [{apiGroups:["*"],...}]` — the chart hands its ServiceAccount the cluster. A logging/monitoring agent that "needs to see everything" rarely needs *write* on everything; scope it. - **`securityContext`/`podSecurityContext` values defaulting to root/privileged**, or `hostNetwork`/`hostPID`/`hostPath` toggles. Override to non-root, no host namespaces; admission (`rules/03`) should reject these anyway — if the chart can't deploy under PSA `restricted`, that's a signal. - **`serviceAccount.create` + a powerful binding** — confirm the SA it creates isn't bound to a built-in `admin`/`cluster-admin` role. - **Operators bundled in charts** — apply `rules/05` (vet the operator RBAC and CRDs). Don't disable your own admission policies to make a chart install. If a vendor chart demands cluster-admin or privileged pods, push back, scope it, or sandbox it — vendor convenience is not a risk acceptance. ## 3. Chart & image provenance — enforced at admission - **Sign and verify Helm charts.** OCI-registry charts can be **cosign-signed**; verify the signature/provenance before install (`cosign verify <oci-chart-ref>`), and pin the chart by **digest**, not a mutable version tag, in your GitOps source. An unsigned chart pulled by floating version is mutable supply-chain surface. - **Only-signed-images enforcement** is the admission control from `rules/03` §4 — the manifests a chart produces must reference images that pass signature/provenance verification (cosign/Kyverno verifyImages / policy-controller). Signing without admission enforcement is theater (`sota-devsecops` rules/02 produces the signatures; this is where they're checked). - **Digest-pin images at the manifest layer.** Deploy manifests reference `image@sha256:<digest>` (or a signed tag your admission policy mutates to a verified digest), never `:latest` or a floating tag — build-once-promote-many (`sota-devsecops` rules/06). Mutable tags admit a different artifact than you reviewed. - **Kustomize**: pin `images:` with `digest:`; review `patches`/`patchesStrategicMerge` for privilege escalation the same way (a patch can add `privileged: true` or a hostPath); beware remote bases (`resources:` pointing at a URL) — pin and review them like remote charts. ## 4. Kubernetes API audit logging — produce the stream The API server audit log is the authoritative record of *who did what to the cluster* — the basis for forensics and the detections in `sota-detection-engineering`. Without it, a cluster compromise is uninvestigable. (On managed clusters, enable the provider's control-plane audit logging — you configure the policy via the provider, e.g. EKS audit logs to CloudWatch, GKE audit logs to Cloud Logging, AKS to the diagnostic settings.) **Audit levels** (per rule, least→most): `None` → `Metadata` (who/what/when, no bodies) → `Request` (+ request body) → `RequestResponse` (+ response body). Tune per sensitivity — log everything at `Metadata`, escalate sensitive verbs/resources to `RequestResponse`, and drop noise (read-only system loops) to `None`. ```yaml # audit-policy.yaml — log sensitive things richly, drop noise, default to Metadata apiVersion: audit.k8s.io/v1 kind: Policy omitStages: ["RequestReceived"] rules: # Secrets/configmaps: log requests but DON'T capture bodies (would log secret values) - level: Metadata resources: [{ group: "", resources: ["secrets","configmaps"] }] # RBAC and admission changes: full bodies — these are escalation actions - level: RequestResponse resources: - { group: "rbac.authorization.k8s.io", resources: ["*"] } - { group: "admissionregistration.k8s.io", resources: ["*"] } # exec/attach/portforward into pods: high-signal, log them - level: Request resources: [{ group: "", resources: ["pods/exec","pods/attach","pods/portforward"] }] # drop noisy authenticated read loops from system components - level: None users: ["system:kube-scheduler","system:kube-controller-manager"] verbs: ["get","list","watch"] - level: Metadata # default for everything else ``` Note: **don't log Secret/ConfigMap bodies** (`Request`/`RequestResponse` on `secrets`) or you write secret values into the audit log — keep them at `Metadata`. ## 5. Ship the stream (tamper-resistant, retained) - **Forward audit logs off the control-plane node** to tamper-resistant, append-only storage in a **separate trust domain** — credentials that can compromise the cluster must not be able to delete its audit trail. Retention ≥1 year (align with compliance — `sota-privacy-compliance`). - Pipe to your SIEM / log platform; the **plumbing** is `sota-observability` territory, the **detections** (privilege escalation, anonymous access, exec into prod, token abuse, policy disable) are `sota-detection-engineering`. - **Pipeline/controller identities are alertable principals**: an action by the Argo CD SA outside a sync, or by a CI SA at an odd time, is a signal. ## 6. Runtime security pointer Audit logs capture the *API* surface; they don't see in-container behavior (a process that never calls the API). For runtime detection — anomalous syscalls, unexpected egress, process execution, container drift — deploy **Falco** or **Tetragon (eBPF)**. Their deployment-as-privileged-DaemonSet hardening overlaps `sota-sandboxing`, and the **detection rules/content are `sota-detection-engineering`**. This skill's job is to ensure the audit-log source exists and is shipped; pair it with runtime detection there. ## Audit checklist - [ ] Third-party charts rendered (`helm template`) and reviewed for RBAC, privileged/host-* specs, and images BEFORE install; upgrades diffed so privilege can't silently widen? - [ ] No chart values granting cluster-admin / `*/*` ClusterRole / root-privileged pods accepted unscoped (RBAC-in-charts trap)? Admission policies not disabled to force an install? - [ ] Charts signed (cosign) and pinned by digest; remote Kustomize bases pinned + reviewed? - [ ] Images digest-pinned at the manifest layer (no `:latest`/floating tags); only-signed-images enforced at admission (rules/03)? - [ ] API server audit logging enabled with a tuned policy: Metadata default, RequestResponse on RBAC/admission changes, Request on exec/attach, secrets/configmaps NOT capturing bodies, system read-loops dropped? (`grep -E 'audit-policy-file|audit-log' kube-apiserver.yaml`; managed → provider audit logging on) - [ ] Audit logs shipped off-node to tamper-resistant, append-only storage in a separate trust domain, retained ≥1y, not deletable by cluster-compromising creds? - [ ] Audit stream wired to SIEM; controller/CI/automation SAs treated as alertable principals (detections → sota-detection-engineering)? - [ ] Runtime detection (Falco/Tetragon) deployed to cover in-container behavior the audit log can't see (content → sota-detection-engineering)?
-
-
SKILL.md 12.5 KB
--- name: sota-kubernetes description: >- State-of-the-art Kubernetes platform security and operations (2026) for cloud-managed (EKS/GKE/AKS) and self-hosted clusters (kubeadm, k3s/k0s, Talos). Use when building, operating, hardening, or auditing the cluster PLATFORM layer: control plane and etcd, API server/kubelet, RBAC, admission control and policy-as-code, GitOps controllers, operators/CRDs/admission webhooks, Helm/Kustomize supply chain, multi-tenancy, cluster lifecycle/upgrades/CVE response, and audit logging. Trigger keywords: Kubernetes, k8s, cluster, control plane, etcd, kube-apiserver, kubelet, RBAC, ClusterRole, ServiceAccount, admission controller, Pod Security Admission, PSA, Kyverno, Gatekeeper, OPA, ValidatingAdmissionPolicy, Argo CD, Flux, GitOps, operator, CRD, admission webhook, Helm, Kustomize, multi-tenancy, vCluster, Talos, k3s, k0s, kubeadm, EKS, GKE, AKS, CIS benchmark, kube-bench, audit log, version skew. NOT pod-level securityContext/seccomp (sota-sandboxing) or NetworkPolicy/CNI depth (sota-network-security). --- # SOTA Kubernetes Platform Security & Operations ## Purpose Engineer and audit the Kubernetes **platform** so that a compromised workload, a hostile chart, a leaked token, or a malicious controller cannot pivot to cluster-admin, read every Secret, or take the cluster down. This skill owns the layer above the pod: control plane and etcd, API server and kubelet hardening, RBAC and ServiceAccount identity, admission control and policy-as-code, GitOps controllers, operators/CRDs and admission webhooks, Helm/Kustomize at admission, multi-tenancy boundaries, cluster lifecycle, and K8s audit logging. It does **not** re-teach pod isolation mechanics. Boundaries it defers: - **Pod `securityContext`, seccomp, AppArmor, capabilities, PSA pod-level fields** → `sota-sandboxing` (rules/03 containers & microVMs; rules/01 boundaries). This skill owns the *admission-time enforcement* of those fields, not their internals. - **NetworkPolicy semantics, CNI choice, service mesh, mTLS** → `sota-network-security`. This skill states only the *requirement* (default-deny per namespace, enforced at admission). - **OIDC/SSO and RBAC-role *design* methodology** → `sota-identity-access`. This skill owns the K8s RBAC *mechanics and escalation traps*. - **Runtime/audit-log detection content (Falco/Tetragon rules, detections)** → `sota-detection-engineering`. This skill owns *producing and shipping* the audit stream. - **Workload identity, Secret storage backends (ESO, sealed-secrets, CSI)** → `sota-secrets-management` (rules/01, rules/02). This skill owns etcd encryption-at-rest. - **Cloud IAM, managed-K8s selection, DR** → `sota-cloud-infrastructure` (rules/02, rules/04, rules/07). - **CI/CD provenance, image signing, IaC/GitOps pipeline, runtime ops** → `sota-devsecops` (rules/02, rules/04, rules/06, rules/07). Two modes. Pick one explicitly at the start of the task. --- ## BUILD mode Use when provisioning a cluster, writing RBAC/policies/manifests, configuring a GitOps controller, or installing an operator/chart. 1. **Name the cluster topology first**: managed (EKS/GKE/AKS) vs self-hosted (kubeadm, k3s/k0s, Talos), single- vs multi-tenant, who the threat actors are (workload → control plane? tenant → tenant? compromised CI → cluster?). Read the matching rules. 2. **Least privilege by default**: no wildcard RBAC verbs/resources, `automountService AccountToken: false` unless the pod calls the API, scoped AppProjects, operators get the narrowest RBAC that works. Loosen only with a comment stating why. 3. **Encrypt etcd at rest with a KMS provider** (`rules/01`) — Secrets are base64, not encrypted, by default. This is the single most common "we thought we were covered" gap. 4. **Admission is fail-closed and ENFORCING, not auditing.** A policy left in `audit`/ `warn` forever is documentation, not a control (`rules/03`). Ship the AUDIT→ENFORCE rollout plan with the policy. 5. **GitOps is the only write path to the cluster.** Humans propose via PR; the controller reconciles. Scope the controller's own privileges and AppProjects tightly (`rules/04`). 6. **Plan the upgrade before you build**: version skew, EOL date, CVE-response runbook, tested etcd restore (`rules/01`, `rules/07`). Deliverables: topology + threat statement, the concrete manifests/configs, the AUDIT→ENFORCE rollout, and the residual-risk/assumptions list (cloud-managed control plane internals you cannot see, org IAM, CNI behavior). ## AUDIT mode Use when reviewing an existing cluster, its RBAC, policies, GitOps config, or operators. Procedure: inventory the cluster surface (control plane flags or managed equivalent, RBAC graph, admission policies, GitOps controllers, operators/CRDs, namespaces/tenancy); for each, walk the relevant rules-file Audit checklist; verify empirically with `kubectl`, `kubectl auth can-i`, rbac-tool/krane, kube-bench, and `helm template` where possible; report findings in the format below. Do not report style nits as security findings. ### Severity conventions | Severity | Meaning | Examples | |---|---|---| | **Critical** | Cluster-admin, all-Secrets read, or cluster takedown reachable now | `anonymous-auth` enabled on API server/kubelet; etcd unencrypted AND reachable; ClusterRoleBinding granting `cluster-admin` to a workload SA or `system:authenticated`; wildcard `*/*` ClusterRole bound broadly; Argo CD AppProject `clusterResourceWhitelist: [{group: '*', kind: '*'}]` with broad SSO; unpatched control plane on a known-RCE CVE | | **High** | Escalation/secret-read by an in-cluster or contributor principal, or a single event from it | `escalate`/`bind`/`impersonate` verbs granted; secret-reader → token-mint → privilege chain; admission policy in `audit` mode for a control that should enforce; image-verification policy only `Audit`s signatures; operator with cluster-wide `secrets:*`; kubelet `read-only-port` open; no etcd backup or untested restore | | **Medium** | Weakens defense in depth or detection | PSA not enforced (only `warn`/`audit`); `automountServiceAccountToken` defaulted on for non-API pods; no default-deny NetworkPolicy (requirement-level; depth → network-security); audit policy missing or `None`/Metadata-only for Secret access; aggregated ClusterRole accreting verbs; no PDB on critical workloads | | **Low** | Hygiene, hardening headroom | RBAC subjects for departed users; unused ClusterRoles; namespaces without resource quotas; `:latest` image tags admitted; missing PolicyException expiry | Severity is judged by **reachability** (anonymous > workload/tenant > contributor > admin) × **yield** (cluster-admin/all-Secrets > namespace compromise > info leak > availability). ### Finding format ``` file:line | rule | severity | effort | fix ``` Where `effort` is one of trivial / small / medium / large, `rule` is the rules-file section (e.g. `02 §RBAC-wildcards`), and `fix` is the concrete change. Expand below the line with: what is wrong, who exploits it and how (concrete attack path), and a snippet when short. Example: ``` clusterrole-ci.yaml:14 | 02 §escalate-verbs | Critical | small | remove the bind/escalate grant; scope to the 3 named ClusterRoles the controller actually applies Issue: ClusterRole bound to the CI ServiceAccount grants rbac.authorization.k8s.io/{bind,escalate} on clusterroles. Attack path: anyone who can run a job as this SA (any merged PR) can bind themselves cluster-admin. ``` End every audit with: counts per severity, top 3 fixes by risk-reduction-per-effort, and an explicit OUT-OF-SCOPE list (managed control-plane internals, org IAM, CNI/mesh, runtime detection content). ## Rules index | File | Read this when... | |---|---| | [rules/01-control-plane-etcd.md](rules/01-control-plane-etcd.md) | HA control plane, API server flags (anonymous-auth, authz modes, audit), etcd encryption-at-rest with KMS v2 + backup/defrag/restore, kubelet hardening, node + immutable-distro hardening (Talos no-SSH/machine-config/SecureBoot+TPM, k3s/k0s), CIS benchmark + kube-bench | | [rules/02-rbac-serviceaccounts.md](rules/02-rbac-serviceaccounts.md) | Roles/ClusterRoles least-privilege, the escalation traps (wildcards, `bind`/`escalate`/`impersonate`, cluster-admin bindings, aggregated roles, secret-reader chains), ServiceAccount hygiene (automount off, bound/projected/audience-scoped tokens, no long-lived token Secrets), RBAC auditing (`auth can-i`, rbac-tool/krane/who-can) | | [rules/03-admission-policy.md](rules/03-admission-policy.md) | Pod Security Admission (restricted/baseline/privileged, enforce/audit/warn) and its limits, Kyverno vs Gatekeeper/OPA vs ValidatingAdmissionPolicy/MutatingAdmissionPolicy, the AUDIT→ENFORCE rollout discipline, image verification at admission (cosign/Kyverno verifyImages), PolicyException discipline | | [rules/04-gitops-controllers.md](rules/04-gitops-controllers.md) | Argo CD / Flux security: AppProject scoping (the `clusterResourceWhitelist:[{*,*}]` trap), project/RBAC/SSO, the controller's own privileges and self-management, repo/SSH creds, ApplicationSet injection, auto-sync vs approval, drift, recent Argo CD CVEs, promotion/rollback as git ops | | [rules/05-operators-crds-webhooks.md](rules/05-operators-crds-webhooks.md) | The operator privilege problem (broad RBAC → CRD-mediated escalation), vetting operator RBAC, CRD validation/security, trusted controllers (ESO, cert-manager), admission webhooks as attack surface (failurePolicy, timeout, TLS, namespaceSelector) | | [rules/06-workloads-tenancy.md](rules/06-workloads-tenancy.md) | Resource requests/limits as availability+QoS, PodDisruptionBudget, topology spread, anti-affinity, priorityClass/preemption, namespace-as-SOFT-boundary reality, hard multi-tenancy (vCluster/separate clusters/node isolation), Secrets in K8s (etcd encryption + ESO/sealed-secrets pointer) | | [rules/07-supply-chain-audit.md](rules/07-supply-chain-audit.md) | Helm/Kustomize review before apply (`helm template`, RBAC-in-charts trap, privilege-granting values), OCI chart signing/provenance, only-signed-images enforcement, digest pinning at the manifest layer, K8s audit policy (what/levels), shipping audit logs, detection + runtime-security pointers | When a task spans layers (most do), read every matching file. For a full cluster audit, read all seven plus the referenced sibling skills. ## Top 10 non-negotiables Violations are at minimum **High** in AUDIT mode and must never be introduced in BUILD mode: 1. **`anonymous-auth=false` on the API server AND kubelet; kubelet `read-only-port=0`, kubelet authz mode `Webhook` (not `AlwaysAllow`).** Anonymous or unauthenticated access to either is cluster-game-over (`01`). 2. **etcd encrypted at rest with a KMS v2 provider** (envelope DEK/KEK), not the default base64. Secrets are NOT encrypted out of the box (`01`, `06`). 3. **No wildcard RBAC.** No `*` verbs/resources in Roles/ClusterRoles bound to workloads or users; no `cluster-admin` bound to a ServiceAccount, `system:authenticated`, or `system:unauthenticated` (`02`). 4. **`bind`, `escalate`, `impersonate` are privilege-escalation verbs** — grant only to a named, audited admin path, never to a workload or CI SA (`02`). 5. **`automountServiceAccountToken: false`** is the default; opt in only for pods that call the API. No long-lived Secret-based SA tokens — use bound/projected tokens (`02`). 6. **Pod Security Admission `restricted` ENFORCED** on workload namespaces (label `pod-security.kubernetes.io/enforce: restricted`), not merely `warn`/`audit`. PSA is stable; PSP is gone (`03`). Pod-level field internals → sota-sandboxing. 7. **Admission policies enforce, with a documented AUDIT→ENFORCE rollout.** A policy parked in `Audit`/`warn` indefinitely is a real, recurring finding (`03`). 8. **GitOps is the only cluster write path; the controller and its AppProjects are tightly scoped.** Never `clusterResourceWhitelist: [{group:'*',kind:'*'}]` with broad project access; pin Argo CD/Flux versions and patch known CVEs (`04`). 9. **Operators get least-privilege RBAC; admission webhooks have correct `failurePolicy`/`timeoutSeconds`/`namespaceSelector` and TLS** — a webhook is both a control and a single point of failure/attack (`05`). 10. **API server audit logging enabled** (RequestResponse for sensitive verbs/Secrets), shipped to tamper-resistant storage; etcd backups taken AND restore-tested (`07`,`01`). Detection content on the stream → sota-detection-engineering. If the user asks for something that violates a non-negotiable, implement the secure alternative and explain the delta; comply only after they acknowledge the risk explicitly.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.