Claude Skill

kubernetes

Kubernetes operations: debugging, security, RBAC, and infrastructure tooling.

LLM Mart · 0 points · 0 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download notque-vexjoy-agent-skills_infrastructure_kubernetes-8ad6845.zip · 16 KB

notque/vexjoy-agent 421 44 forks MIT Updated 5d ago

Part of notque/vexjoy-agent — 69 skills

Install

skills CLI npx skills add https://github.com/notque/vexjoy-agent/tree/main/skills/infrastructure/kubernetes

Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install notque-vexjoy-agent@llmmart

Git git clone https://github.com/notque/vexjoy-agent.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole notque/vexjoy-agent collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Kubernetes Skill

Three domains: debugging (pod triage, networking, resources), security (RBAC, pod hardening, network isolation, supply chain), and cobaltcore (KVM exporter, hypervisor metrics). Select by request signal, then follow the phases below.

Always specify -n <namespace> in every kubectl command. Use read-only commands to gather evidence before proposing changes.

Domain Selection

Signal	Domain
CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending	Debugging
Service unreachable, DNS failure, port-forward	Debugging (network)
CPU throttling, memory limit, disk pressure	Debugging (resources)
RBAC, permissions, roles, ServiceAccount	Security (access)
Pod hardening, container security, PodSecurity	Security (pods)
NetworkPolicy, default-deny, namespace isolation	Security (network)
Image signing, secrets, admission control	Security (supply chain)
KVM exporter, cobaltcore, hypervisor metrics	Cobaltcore

Phase 1: TRIAGE

Debugging Triage Flow

Follow this sequence for every pod or workload issue. Do not skip steps -- many failures are only visible in events and describe output, not in logs.

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
kubectl exec -it <pod-name> -n <namespace> -c <container-name> -- /bin/sh

Always check --previous logs for crashed containers before current logs -- restarting destroys them permanently.

Diagnosis routing:

Symptom	Action
CrashLoopBackOff, ImagePullBackOff, Pending, FailedScheduling	Check describe output for events, previous logs, image pull errors
Service unreachable, DNS failure	Check service endpoints, CoreDNS, NetworkPolicy below
CPU throttling, OOMKill, disk pressure	Check resource limits, requests vs actual, node capacity
"no endpoints available for service"	Compare svc selector with pod labels

Network debugging:

# Verify service has endpoints
kubectl get endpoints <service-name> -n <namespace>
# DNS lookup from inside cluster
kubectl run dns-debug --rm -it --restart=Never --image=busybox:1.36 -n <namespace> -- \
  nslookup <service-name>.<namespace>.svc.cluster.local
# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Port-forward for local testing
kubectl port-forward svc/<service-name> -n <namespace> 8080:80

Security Domain Selection

For security requests, provide concrete YAML manifests from the patterns below. Reference-backed specifics, not generic advice.

RBAC patterns: Prefer namespace-scoped Roles over ClusterRoles. Write exact verbs and resources. Create dedicated ServiceAccounts per workload. Set automountServiceAccountToken: false on pods that need no API access.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-team
  name: deployment-reader
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

Pod security: Enforce PodSecurity labels at namespace level. All containers: runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities: drop: ["ALL"]. Use distroless base images. Pin image digests.

labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/warn: restricted

Network policies: Start with default-deny for ingress and egress. Add allow-list rules per service. Always allow DNS egress (UDP/TCP 53).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Cobaltcore Domain

Components: KVM Exporter (cobaltcore-dev/kvm-exporter). Load references/cobalt-kvm-exporter.md for architecture, metric catalogs, configuration, and deployment. Pair with go-patterns for code, prometheus-grafana-engineer for metrics.

Phase 2: DIAGNOSE / RESPOND

Debugging: Follow the triage flow. Gather evidence with read-only commands before proposing changes.

Security: Provide copy-paste-ready YAML using the patterns in Phase 1.

Cobaltcore: Use component-specific references for architecture, metrics, concurrency patterns, and testing.

Phase 3: VERIFY

Debugging: Confirm the fix resolves the symptom with the same triage commands.
Security: Validate against the PodSecurity standards and RBAC least-privilege patterns above.
Cobaltcore: Verify against component test patterns in references/cobalt-testing-patterns.md.

Deep References

Signal	Reference	Content
KVM exporter architecture, metrics, config	`references/cobalt-kvm-exporter.md`	Full component reference (463 lines)
Cobaltcore concurrency, goroutine, semaphore	`references/cobalt-concurrency-patterns.md`	Go concurrency patterns (268 lines)
Cobaltcore testing, mock, Kind cluster	`references/cobalt-testing-patterns.md`	Testing strategies (271 lines)

Files (vexjoy-agent)

references

cobalt-concurrency-patterns.md 9.4 KB

# Cobalt Core — Concurrency Patterns

> **Scope**: Goroutine management, synchronization primitives, and scrape-safe concurrency as used in kvm-exporter. Does not cover general Go concurrency theory.
> **Version range**: Go 1.21+ (uses `sync.Mutex.TryLock`, generics-ready patterns)
> **Generated**: 2026-04-16 — verify against `internal/libvirt/` source

---

## Overview

kvm-exporter collects metrics from 50–500+ KVM domains per scrape, each requiring libvirt RPC calls, /proc reads, and cgroup stat lookups. The concurrency model is built around: (1) goroutine-per-domain with a semaphore cap to prevent socket exhaustion, (2) `TryLock` to serialize overlapping Prometheus scrapes, and (3) tiered caching with `sync.Map` for cross-scrape state. Violating these patterns causes libvirt socket exhaustion, metric duplication, or stale data.

---

## Pattern Table

| Pattern | Version | Use When | Avoid When |
|---------|---------|----------|------------|
| `sync.Mutex.TryLock()` | Go 1.18+ | Scrape serialization — skip if already collecting | Long-held business locks |
| Buffered channel semaphore | All | Cap goroutines to protect shared resource (libvirt socket) | Global rate limits (use `golang.org/x/sync/semaphore`) |
| `sync.Map` | Go 1.9+ | Per-domain delta state that survives scrapes | Simple per-request scratch space (use plain map + mutex) |
| `atomic.Value` | All | Single-value timestamp for readiness probe | Multi-field structs (use mutex instead) |
| `context.WithTimeout` | All | Bound collection to 40s scrape deadline | Calling `context.Background()` directly in collection |

---

## Correct Patterns

### Semaphore-Limited Goroutines (the kvm-exporter pattern)

Cap concurrent domain collections to prevent libvirt socket exhaustion. The semaphore is a buffered channel of empty structs.

```go
sem := make(chan struct{}, 50) // max 50 concurrent domain goroutines

var wg sync.WaitGroup
for _, domain := range domains {
    wg.Add(1) // Add BEFORE go func to avoid race with wg.Wait()
    go func(d libvirt.Domain) {
        defer wg.Done()
        sem <- struct{}{}         // acquire slot
        defer func() { <-sem }() // release slot
        collectDomain(ctx, d, ch)
    }(domain)
}
wg.Wait()
```

**Why**: libvirt's Unix socket is a single RPC endpoint. Sending 500 simultaneous calls causes libvirt to drop connections. The semaphore keeps in-flight calls bounded without serializing all collection.

---

### TryLock for Scrape Serialization

Use `TryLock` to skip a scrape when the previous one is still running, rather than blocking Prometheus.

```go
type ServiceImpl struct {
    mu sync.Mutex
}

func (s *ServiceImpl) Collect(ch chan<- prometheus.Metric) {
    if !s.mu.TryLock() {
        // Previous scrape still running — emit nothing, Prometheus retries
        return
    }
    defer s.mu.Unlock()
    s.retrieveMetrics(ctx, ch)
}
```

**Why**: If collection takes 35s and Prometheus scrapes every 30s, blocking stacks goroutines until OOM. `TryLock` sacrifices one scrape rather than accumulating blocked callers.

**Version note**: `sync.Mutex.TryLock()` added in Go 1.18.

---

### sync.Map for Cross-Scrape Delta State

Steal time requires comparing current CPU time to the previous scrape's value. `sync.Map` provides concurrent-safe access without a mutex wrapping the domain loop.

```go
// Key: "domainName-PID"
var stealTimeHistory sync.Map

func calculateStealTime(domain, pid string, current uint64) float64 {
    key := domain + "-" + pid
    if prev, ok := stealTimeHistory.Load(key); ok {
        delta := current - prev.(uint64)
        stealTimeHistory.Store(key, current)
        return float64(delta)
    }
    stealTimeHistory.Store(key, current)
    return 0 // first scrape — no delta available
}
```

**Why**: The domain loop runs concurrently. A plain `map[string]uint64` requires a global mutex serializing all delta lookups. `sync.Map` is optimized for read-heavy workloads with stable key sets (domain IDs don't churn rapidly).

---

### Context Cancellation in Collection Loops

Check `ctx.Err()` before queuing each domain to exit early when the scrape timeout fires.

```go
for _, domain := range domains {
    if ctx.Err() != nil {
        return // timeout fired before we could queue this domain
    }
    wg.Add(1)
    go func(d libvirt.Domain) {
        defer wg.Done()
        sem <- struct{}{}
        defer func() { <-sem }()
        if ctx.Err() != nil {
            return // re-check inside goroutine before doing any work
        }
        collectDomain(ctx, d, ch)
    }(domain)
}
```

**Why**: Without the pre-queue check, all 500 domains could be queued before the timeout fires, then all goroutines start and run over deadline together.

---

## Pattern Catalog

### Cap Goroutines with a Semaphore

**Detection**:
```bash
# Find goroutine launches — review each for semaphore acquire
rg 'go func\(' --type go internal/
grep -rn 'go func(' --include="*.go" internal/
```

**Signal**:
```go
for _, domain := range domains {
    go func(d libvirt.Domain) {
        collectDomain(ctx, d, ch) // no semaphore
    }(domain)
}
```

**Why this matters**: On a hypervisor with 500 VMs, this launches 500 goroutines simultaneously. Each goroutine makes libvirt RPC calls over the same Unix socket. libvirt queues drop, connections time out, and the exporter logs hundreds of "connection refused" per scrape.

**Preferred action**: Add buffered channel semaphore with cap ≤ 50 as shown in Correct Patterns above.

---

### Use TryLock in Prometheus Collect Path

**Detection**:
```bash
rg '\.Lock\(\)' --type go internal/libvirt/
grep -n "\.Lock()" internal/libvirt/*.go
```
Review if `Lock()` appears in any `Collect()` or `Describe()` method.

**Signal**:
```go
func (s *ServiceImpl) Collect(ch chan<- prometheus.Metric) {
    s.mu.Lock()         // blocks if previous scrape running
    defer s.mu.Unlock()
    s.retrieveMetrics(ctx, ch)
}
```

**Why this matters**: Prometheus's default scrape timeout is 10s. If collection takes 35s and Prometheus fires every 30s, the second `Collect()` call blocks, then the third — goroutines stack until the scrape target appears hung and alerts fire.

**Preferred action**: Replace `s.mu.Lock()` with `if !s.mu.TryLock() { return }`.

**Version note**: `TryLock` added in Go 1.18. For earlier versions, use `sync/atomic` swap-based implementation.

---

### Defer ClearScrapeCache() at Function Start

**Detection**:
```bash
rg 'ClearScrapeCache' --type go
grep -rn 'ClearScrapeCache' --include="*.go" .
```
Confirm it is called in `defer` or in all exit paths of `retrieveMetrics`.

**Signal**:
```go
func (s *ServiceImpl) retrieveMetrics(ctx context.Context, ch chan<- prometheus.Metric) {
    if err := s.connectLibvirt(); err != nil {
        log.Errorf("connect: %v", err)
        return // scrape cache NOT cleared — stale data persists
    }
    // ...
    s.ch.ClearScrapeCache() // only reached on success
}
```

**Why this matters**: Per-scrape caches (PID lookup, VM counters) accumulate stale entries. On the next successful scrape, the exporter reads stale PIDs for domains restarted since last scrape, emitting metrics for dead processes.

**Preferred action**: `defer s.ch.ClearScrapeCache()` at the top of `retrieveMetrics`, before any error paths.

---

### Use sync.Map for Concurrent Domain State

**Detection**:
```bash
rg 'map\[string\]' --type go internal/libvirt/
grep -n "map\[string\]" internal/libvirt/*.go
```
Any `map[string]` accessed from goroutines without a wrapping mutex is a data race.

**Signal**:
```go
var history = map[string]uint64{} // shared, no mutex

func updateHistory(key string, val uint64) uint64 {
    prev := history[key]  // concurrent read — data race
    history[key] = val    // concurrent write — data race
    return val - prev
}
```

**Why this matters**: Go's race detector (`go test -race`) catches this. In production without `-race`, it causes silent map corruption or `panic: concurrent map read and map write`.

**Preferred action**: Use `sync.Map` for stable read-heavy key sets (domain IDs), or a `sync.RWMutex`-protected map if you need `range` iteration.

---

## Error-Fix Mappings

| Error | Root Cause | Fix |
|-------|-----------|-----|
| `panic: concurrent map read and map write` | Goroutine-per-domain accessing plain `map` | Replace with `sync.Map` or add `sync.RWMutex` |
| `libvirt: connection refused` (mass, same scrape) | Too many concurrent goroutines saturating libvirt socket | Verify semaphore is applied; reduce cap if needed |
| `context deadline exceeded` every scrape | Collection regularly exceeds 40s timeout | Enable pprof (`ENABLE_PPROF=true`); common culprit is blocked libvirt RPC without per-call timeout |
| Metrics missing for some domains each scrape | `wg.Add(1)` called inside goroutine rather than before launch | Move `wg.Add(1)` to before `go func(...)` |

---

## Detection Commands Reference

```bash
# Find unbounded goroutine launches (check each for semaphore)
rg 'go func\(' --type go internal/

# Find blocking Lock() in Prometheus Collect path
rg '\.Lock\(\)' --type go internal/libvirt/

# Find plain map in concurrent code
rg 'map\[string\]' --type go internal/libvirt/

# Find missing or misplaced ClearScrapeCache calls
rg 'ClearScrapeCache' --type go

# Run race detector across all packages
go test -race ./internal/...
```

---

## See Also

- `references/kvm-exporter.md` — Concurrency Model and Cache Tiers sections for architecture overview
- `references/testing-patterns.md` — How to write race-detector-clean tests with moq

cobalt-kvm-exporter.md 17.2 KB

# kvm-exporter Reference

Repository: [cobaltcore-dev/kvm-exporter](https://github.com/cobaltcore-dev/kvm-exporter)
Language: Go 1.23+ (module `github.com/cobaltcore-dev/kvm-exporter`)
License: Apache-2.0
Current version: 1.3.0 (Helm chart 0.3.8)

Prometheus metrics exporter collecting hypervisor-level statistics from KVM hosts running QEMU or Cloud Hypervisor (CH). Gathers data via libvirt, cgroups v2, `/proc`, and CH HTTP socket API. Designed for SAP Converged Cloud OpenStack Nova compute nodes.

---

## Architecture

### Directory Structure

```
cmd/
  server/main.go          # Entry point — HTTP server with chi router
  playground/main.go      # Development playground
internal/
  config/                 # Environment-based configuration (envconfig)
  globals/                # HTTP endpoint constants
  libvirt/                # Core: all collectors, CH integration, metrics
  log/                    # Structured logging (logrus wrapper)
  middlewares/            # HTTP readiness probe middleware
  numareader/             # NUMA topology reader (/proc/pid/numa_maps, sysfs numastat)
  procreader/             # Process reader (/proc: schedstat, cmdline)
chart/                    # Helm chart (DaemonSet, ServiceMonitor, alerts, dashboards)
config/                   # Kustomize deployment configs
test/                     # E2E infrastructure (Kind cluster with VMs)
```

### Data Flow

```
                              Prometheus scrape
                                    |
                               GET /metrics
                                    |
                            promhttp.Handler()
                                    |
                          ServiceImpl.Collect()
                                    |
                          RetrieveMetrics(ctx)
                                    |
          +-----------+-------------+----------+-----------+
          |           |             |          |           |
      libvirt     cgroups v2    /proc fs    CH socket   sysfs
      (RPC)       (cgroup2)    (procreader) (HTTP API)  (numa)
          |           |             |          |           |
          +-----------+-------------+----------+-----------+
                                    |
                          chan<- prometheus.Metric
```

### Entry Point (`cmd/server/main.go`)

1. Reads config via `config.ReadConfig()` (env vars + optional libvirt.conf)
2. Sets GC to 50%, memory limit to 450 MiB (tuned for memory-constrained pods)
3. Creates `libvirt.ServiceImpl` (the Prometheus collector)
4. Registers with `prometheus.MustRegister()`
5. Sets up chi router: `/health`, `/ready`, `/metrics`, `/debug/pprof/*` (optional)
6. Starts HTTP server (default port 8080)
7. Graceful shutdown on SIGINT/SIGTERM

### Collector Architecture

`ServiceImpl` implements `prometheus.Collector` (Describe/Collect). On each scrape:

1. **Connect** to libvirt via Unix socket if not connected
2. **List all domains** (active + inactive) via `ConnectListAllDomains`
3. **Parallel /proc scan** — `UpdateAllProcesses()` finds VM processes
4. **Bulk domain stats** — single `ConnectGetAllDomainStats` RPC (QEMU only)
5. **Concurrent domain collection** — goroutines per domain (semaphore-limited to 50):
   domain info, vCPU, block, network, memory, uptime, steal time, hugepages, NUMA
6. **Node-level collection** — storage pools, libvirt version, NIC bonding, CPU SMT, host NUMA
7. **Stale cleanup** — remove cached data for inactive domains
8. **Aggregation** — weighted average steal time across all domains

### Performance Optimizations

| Technique | Purpose |
|-----------|---------|
| Bulk `ConnectGetAllDomainStats` | Replace N per-domain RPCs with one call |
| XML cache (5-min TTL) | Eliminate XML desc RPCs per scrape |
| Block I/O tune cache (5-min TTL) | Reduce I/O tune lookups |
| Concurrency semaphore (50 goroutines) | Prevent libvirt socket exhaustion |
| Collection timeout (40s) | Context cancellation on slow scrapes |
| `sync.Mutex.TryLock()` | Prevent overlapping scrapes |
| `debug.FreeOSMemory()` | Explicit memory return after collection |

---

## Configuration

All via environment variables (`kelseyhightower/envconfig`):

| Env Variable | Type | Default | Purpose |
|---|---|---|---|
| `SERVICE_NAME` | string | `kvm-exporter` | Service identifier in logs |
| `APP` | string | `kvm-exporter` | Application name |
| `ENV` | string | `LOCAL` | Environment (LOCAL = debug text, else JSON info) |
| `PORT` | int | `8080` | HTTP listen port |
| `LIBVIRT_SOCKET` | string | `/run/libvirt/libvirt-sock-ro` | Libvirt Unix socket path |
| `LIBVIRT_URI` | string | `ch:///system` | Libvirt connection URI |
| `LIBVIRT_CONF_PATH` | string | `/etc/libvirt/libvirt.conf` | Optional libvirt config |
| `CLOUDHYPERVISOR_SOCKET_PATH` | string | `/run/libvirt/ch` | CH socket base directory |
| `EXTERNAL_MOUNT_PATHS` | []string | `/var/lib/nova/mnt,/var/lib/nova/instances` | Disk usage mount paths |
| `DISABLED_COLLECTORS` | []string | (empty) | Collectors to disable |
| `HOSTNAME` | string | (from env) | Node hostname |
| `ENABLE_PPROF` | string | (unset) | `true` enables pprof endpoints |

### Valid Collector Names for `DISABLED_COLLECTORS`

`domain_info`, `vcpu`, `network`, `memory`, `block`, `block_limits`, `storage_pools`, `uptime`, `steal_time`, `hugepages`, `version`, `nic_bonding`, `cpu_smt`, `domain_numa`, `node_numa`

### Hypervisor Detection

`isCloudHypervisor()` checks if `LIBVIRT_URI` starts with `ch://`. Drives conditional logic throughout all collectors.

---

## Metric Catalog

### Domain Info (`domain_info`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_info` | Gauge | node, domain, uuid, instance_name, flavor, user_name, user_uuid, project_name, project_uuid, root_type, root_uuid |
| `kvm_domain_libvirt_info_maximum_memory_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_info_memory_usage_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_info_virtual_cpus` | Gauge | node, domain |
| `kvm_domain_libvirt_info_cpu_time_seconds_total` | Counter | node, domain |
| `kvm_domain_libvirt_info_vstate` | Gauge | node, domain |

vstate values: 0=no state, 1=running, 2=blocked, 3=paused, 4=shutting down, 5=shut off, 6=crashed, 7=suspended

### vCPU (`vcpu`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_vcpu_delay_nanoseconds` | Counter | node, domain, vcpu, cpu_index |
| `kvm_domain_libvirt_vcpu_spent_nanoseconds` | Counter | node, domain, vcpu, cpu_index |
| `kvm_domain_libvirt_vcpu_timeslices_total` | Counter | node, domain, vcpu, cpu_index |
| `kvm_domain_libvirt_vcpu_time_seconds_sum` | Counter | node, domain |
| `kvm_domain_libvirt_vcpu_online_count` | Gauge | node, domain |
| `kvm_domain_libvirt_vcpu_blocked_count` | Gauge | node, domain |

### Network (`network`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_interface_stats_receive_bytes_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_receive_packets_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_receive_errors_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_receive_drops_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_transmit_bytes_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_transmit_packets_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_transmit_errors_total` | Counter | node, domain, target_device |
| `kvm_domain_libvirt_interface_stats_transmit_drops_total` | Counter | node, domain, target_device |

### Memory (`memory`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_memory_stats_major_fault_total` | Counter | node, domain |
| `kvm_domain_libvirt_memory_stats_minor_fault_total` | Counter | node, domain |
| `kvm_domain_libvirt_memory_stats_unused_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_available_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_actual_balloon_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_rss_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_usable_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_disk_cache_bytes` | Gauge | node, domain |
| `kvm_domain_libvirt_memory_stats_used_percent` | Gauge | node, domain |

Memory stats fallback chain: libvirt API → CH cgroup stats → /proc/PID/status

### Block Device (`block`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_block_device_info` | Gauge | node, domain, device_name, path, allocation, capacity, physical |

### Block Limits (`block_limits`)

19 Gauge metrics with labels `node, domain, target_device` covering I/O throttling:
- `kvm_domain_libvirt_block_stats_limit_{total,write,read}_bytes`
- `kvm_domain_libvirt_block_stats_limit_{total,write,read}_requests`
- `kvm_domain_libvirt_block_stats_limit_size_iops_bytes`
- `kvm_domain_libvirt_block_stats_limit_burst_{total,write,read}_bytes`
- `kvm_domain_libvirt_block_stats_limit_burst_{total,write,read}_requests`
- `kvm_domain_libvirt_block_stats_limit_burst_{total,write,read}_bytes_length_seconds`
- `kvm_domain_libvirt_block_stats_limit_burst_length_{total,write,read}_requests_seconds`

### Storage Pools (`storage_pools`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_pool_info_capacity_bytes` | Gauge | pool |
| `kvm_domain_libvirt_pool_info_allocation_bytes` | Gauge | pool |
| `kvm_domain_libvirt_pool_info_available_bytes` | Gauge | pool |

### Uptime (`uptime`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_uptime_seconds` | Gauge | node, domain |

### Steal Time (`steal_time`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_libvirt_steal_time` | Gauge | node, domain |
| `kvm_node_hypervisor_steal_time` | Gauge | node |

Per-domain steal time is percentage-based (delta between scrapes). Node-level is weighted average by vCPU count.

### Version (`version`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_node_libvirt_versions_info` | Gauge | node, libvirtd_running, libvirt_library, libvirt_uri |

### NIC Bonding (`nic_bonding`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_node_nic_bonding_info` | Gauge | node, master, slave_interface, status |

status: 1=up, 0=down

### CPU SMT (`cpu_smt`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_node_cpu_smt_info` | Gauge | node, smt_active, smt_control |

### Hugepages (`hugepages`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_hugepages_bytes` | Gauge | node, domain |

From /proc/PID/smaps `Private_Hugetlb`.

### NUMA (`domain_numa`, `node_numa`)

| Metric | Type | Labels |
|---|---|---|
| `kvm_domain_numa_memory_bytes` | Gauge | node, domain, numa_node |
| `kvm_node_numa_hit_total` | Counter | node, numa_node |
| `kvm_node_numa_miss_total` | Counter | node, numa_node |
| `kvm_node_numa_foreign_total` | Counter | node, numa_node |
| `kvm_node_numa_local_total` | Counter | node, numa_node |
| `kvm_node_numa_other_total` | Counter | node, numa_node |
| `kvm_node_numa_interleave_hit_total` | Counter | node, numa_node |

### Internal Operational Metrics

| Metric | Type | Description |
|---|---|---|
| `functions_processed` | Counter | Processed function calls |
| `jobs_errors` | Counter | Error counter |
| `methods_called_seconds` | Histogram | Per-method latency (labels: source, method, node) |

---

## Cloud Hypervisor Integration

`CloudhypervisorWrapper` provides an alternative data path for CH-based VMs.

### Connection

Per-domain Unix socket at `{socketPath}/{domainName}-socket`. HTTP client with 15s timeout.

### Cgroup Stats

Reads cgroup v2 via `containerd/cgroups/v3`. Includes workaround for containerd library bug where `Stat()` fails to read `cpu.stat` for leaf cgroups — reads the file directly as fallback.

### PID Lookup Strategies

1. Cgroup `cgroup.procs` file
2. Domstatus XML at `/run/libvirt/ch/{domain}.xml`
3. Process name matching via /proc

### Key Differences from QEMU

| Aspect | QEMU | Cloud Hypervisor |
|--------|------|------------------|
| Stats source | libvirt RPC | HTTP API (vm.info, vm.counters) |
| Cgroup path | `machine.slice` | `machine` |
| CPU time | libvirt stats | Cgroup-based |
| Memory stats | libvirt API | Cgroup + /proc fallback |
| Socket | Single libvirt socket | Per-domain socket |

### Caching

- **Per-scrape caches**: PID, VM counters — cleared via `ClearScrapeCache()`
- **Cross-scrape caches**: config, qemu-img info, df source — 5-minute TTL

---

## Deployment

### Container Image

- Base: `gcr.io/distroless/cc:nonroot`
- CGO_ENABLED=1 (required for libvirt)
- Bundled tools: `ps`, `df`, `qemu-img`, `nsenter`, `cat`, `ls`
- UID/GID: `42438:42438` (kvm-node-agent user)
- Registry: `ghcr.io/cobaltcore-dev/kvm-exporter`

### Kubernetes DaemonSet

```yaml
hostPID: true        # Read /proc of host VM processes
hostNetwork: true    # Host network access
securityContext:
  runAsUser: 0
  readOnlyRootFilesystem: true
  capabilities:
    add: [SYS_PTRACE]   # /proc reading
    drop: [ALL]
appArmorProfile:
  type: Unconfined       # Required for libvirt operations
```

Volume mounts: `/run/libvirt` (ro), `/sys/fs/cgroup` (ro), libvirt.conf, Nova mount paths.

### Health Probes

| Probe | Endpoint | Behavior |
|-------|----------|----------|
| Liveness | `GET /health` | Chi heartbeat (200 if alive) |
| Readiness | `GET /ready` | 503 if last collection > 5 minutes ago (`atomic.Value`) |

---

## Alerts

Built-in alerts in `chart/alerts/exporter.yaml`:

| Alert | Severity | Condition |
|---|---|---|
| `KvmNodeLibvirtMetricsMissing` | warning | `up{job="kvm-exporter"} == 0` for 15m |
| `KvmNodeMissingKvmExporter` | critical | Ready node has no kvm-exporter job for 5m |
| `KvmExporterScrapeDurationHigh` | warning | `scrape_duration_seconds > 30` for 15m |
| `KvmNodeLibvirtNotResponding` | critical | GardenLinux node without version_info for 10m |

---

## Dependencies

### Runtime

| Dependency | Purpose |
|---|---|
| `digitalocean/go-libvirt` | Pure Go libvirt RPC client (no CGO bindings) |
| `prometheus/client_golang` | Prometheus metrics |
| `Tinkoff/libvirt-exporter` | Libvirt XML schema types |
| `containerd/cgroups/v3` | Cgroup v2 reader |
| `go-chi/chi/v5` | HTTP router |
| `kelseyhightower/envconfig` | Env-based config |
| `sirupsen/logrus` | Structured logging |
| `shirou/gopsutil/v3` | System info (disk usage) |
| `tklauser/go-sysconf` | Sysconf for clock ticks |
| `coreos/go-systemd/v22` | Systemd unit name escaping |
| `prometheus/procfs` | /proc filesystem helpers |

### Development

| Tool | Version | Purpose |
|---|---|---|
| `golangci-lint` | 1.57.2 | 18 linters enabled |
| `moq` | 0.5.0 | Interface mock generation |
| `kustomize` | 5.4.1 | K8s manifest management |
| `controller-gen` | 0.15.0 | CRD generation |
| `setup-envtest` | release-0.18 | Test environment setup |

---

## Code Patterns

### Interface-Based Design

`LibVirt` and `Cloudhypervisor` interfaces allow mock injection for testing. Mocks generated with `moq`.

### Metric Naming Convention

- Domain-scoped libvirt: `kvm_domain_libvirt_` prefix (via `newLibvirtDesc()`)
- Non-libvirt domain: `kvm_domain_` prefix (via `newRawDesc()`)
- Node-level: `kvm_node_` prefix (via `newRawDesc()`)

### Error Handling

Errors logged with structured fields (runID, domain, collector name). Collection continues for other domains — no panics in collection path.

### Context Cancellation

`ctx.Err()` checked at multiple points in collection for early exit when scrape timeout approaches.

### Delta Calculations

Steal time uses `sync.Map` keyed by `"domainName-PID"`, calculating deltas between scrapes.

### Cache Tiers

| Tier | Lifetime | Examples |
|------|----------|----------|
| Per-scrape | Cleared via `ClearScrapeCache()` | PID, VM counters |
| Cross-scrape (TTL) | 5-minute TTL | XML desc, block I/O tune |
| Persistent | `sync.Once` | Boot time |

### Concurrency Model

| Mechanism | Purpose |
|-----------|---------|
| `sync.Map` | Shared state (steal time history, caches) |
| `sync.Mutex.TryLock()` | Scrape serialization |
| Buffered channel semaphore | Goroutine limiting (50 max) |
| `atomic.Value` | Thread-safe timestamp (readiness probe) |

---

## Testing

### Unit Tests

- Standard Go testing with `testify/assert`
- Mock-based via `moq`-generated `InterfaceMock` (`interface_mock_gen.go`)
- Pattern: create `ServiceImpl` with mocked `virt`, inject metric channel, call collector, verify

### E2E Tests

- Custom Kind cluster image with libvirt + Cloud Hypervisor (`test/kind/Dockerfile`)
- Test VMs created on worker nodes
- `make test-all` — build image, set up cluster, deploy DaemonSets, create VMs, validate metrics
- `test/test-metrics.sh` — validates expected metrics in HTTP response
- Separate `test-qemu` and `test-ch` targets

---

## CI/CD Workflows

| Workflow | Purpose |
|---|---|
| `app-test.yaml` | Go tests |
| `app-push.yaml` | Build and push container image |
| `app-test-image-build.yaml` | Build test Kind node image |
| `chart-push.yaml` / `chart-validate.yaml` | Helm chart CI |
| `validate-pr.yaml` | PR validation |
| `validate-prometheus-alerts.yaml` | Alert rule validation |
| `bump.yaml` | Automated version bumping |
| `release.yaml` | Release creation |
| `reuse.yaml` | REUSE compliance |
| `stale.yaml` | Stale issue cleanup |

### Versioning

SemVer in `VERSION` file. PR title markers for automated bumps:
- `[BUGFIX]` — patch bump
- `[FEATURE]` — minor bump
- `[BREAKING_CHANGE]` — major bump

cobalt-testing-patterns.md 9.4 KB

# Cobalt Core — Testing Patterns

> **Scope**: Unit testing with moq-generated mocks and E2E testing with Kind clusters, as used in kvm-exporter. Does not cover general Go testing theory.
> **Version range**: Go 1.21+, moq 0.5.0, Kind 0.23+
> **Generated**: 2026-04-16 — verify against `test/` and `internal/libvirt/*_test.go`

---

## Overview

kvm-exporter has two test layers: unit tests using `moq`-generated interface mocks (fast, no libvirt), and E2E tests using a custom Kind cluster image with actual libvirt + Cloud Hypervisor running (slow, requires Docker). The unit test pattern is interface injection — `ServiceImpl` accepts mock `LibVirt` and `Cloudhypervisor` dependencies. The E2E layer validates that expected metrics actually appear in the HTTP response. Skipping the race detector on unit tests is the most common error that lets concurrency bugs through.

---

## Pattern Table

| Tool | Version | Use When | Avoid When |
|------|---------|----------|------------|
| `moq` | 0.5.0 | Generate mocks for `LibVirt`, `Cloudhypervisor` interfaces | Hand-rolling mocks (use `moq` for regeneration) |
| `testify/assert` | All | Readable assertion failures | `t.Fatalf` for non-fatal checks |
| `go test -race` | All | All concurrent code; CI always | Benchmarks (adds overhead) |
| Kind E2E | 0.23+ | Validating actual metric output end-to-end | Unit-testable logic (too slow) |
| `test/test-metrics.sh` | — | HTTP response metric validation in E2E | Go test for HTTP parsing |

---

## Correct Patterns

### Unit Test: Interface Injection with moq

Create `ServiceImpl` with mocked interfaces, inject a metric channel, call the collector, verify emitted metrics.

```go
func TestVcpuCollector(t *testing.T) {
    virt := &LibVirtMock{
        ConnectGetAllDomainStatsFunc: func(...) ([]libvirt.DomainStats, error) {
            return []libvirt.DomainStats{
                {
                    Domain: libvirt.Domain{Name: "test-domain"},
                    Vcpu:   []libvirt.DomainStatsVcpu{{State: 1, Time: 1000}},
                },
            }, nil
        },
    }

    svc := &ServiceImpl{virt: virt}
    ch := make(chan prometheus.Metric, 10)

    svc.collectVcpu(context.Background(), ch)
    close(ch)

    var metrics []prometheus.Metric
    for m := range ch {
        metrics = append(metrics, m)
    }
    assert.NotEmpty(t, metrics, "expected at least one vcpu metric")
}
```

**Why**: No libvirt socket required. The mock records call arguments so you can assert the RPC was called with correct parameters. Regenerate with `make generate` after interface changes.

---

### Regenerating Mocks After Interface Change

After modifying `LibVirt` or `Cloudhypervisor` interfaces, regenerate mocks:

```bash
# In repo root
make generate
# or directly:
moq -out internal/libvirt/interface_mock_gen.go \
    internal/libvirt LibVirt Cloudhypervisor
```

The generated file is committed — never edit `interface_mock_gen.go` by hand.

**Why**: Hand-edited mocks drift from the interface. `moq` regeneration guarantees the mock matches the interface exactly, and CI will fail if the generated file is out of sync.

---

### E2E Test: Kind Cluster + metric validation

The E2E setup creates a Kind cluster with a custom node image containing libvirt and Cloud Hypervisor. VMs are created on worker nodes, the exporter DaemonSet is deployed, and metrics are validated via HTTP.

```bash
# Full E2E (QEMU + CH paths):
make test-all

# QEMU path only:
make test-qemu

# CH path only:
make test-ch

# Manual metric check against running exporter:
curl -s http://NODE_IP:8080/metrics | grep 'kvm_domain_libvirt_vcpu'
```

Metric validation script (`test/test-metrics.sh`) uses grep patterns against the HTTP response — add expected metrics there when adding new collectors.

```bash
# Add to test/test-metrics.sh for a new metric:
check_metric "kvm_domain_libvirt_new_metric_name"
```

**Why**: The unit tests mock libvirt responses. The E2E layer validates that the actual libvirt RPC, cgroup reads, and /proc parsing produce real metrics. It's the only layer that catches libvirt version incompatibilities.

---

### Race-Detector-Clean Tests

Always run the race detector in CI. For local development, add `-race` to catch issues before push.

```bash
# Run with race detector
go test -race ./internal/...

# Run specific test with race detector
go test -race -run TestCollectVcpu ./internal/libvirt/

# Run with verbose output and race detector
go test -race -v ./internal/libvirt/
```

When using `sync.Map` or channels in tests, the race detector validates the access patterns are safe.

---

## Pattern Catalog

### Regenerate Mocks via make generate

**Detection**:
```bash
git log --oneline internal/libvirt/interface_mock_gen.go
grep -n "hand" internal/libvirt/interface_mock_gen.go
rg 'interface_mock_gen' --type go
```
Check if the file has commits that don't come from `make generate`.

**Signal**:
```go
// In interface_mock_gen.go — manually added method:
func (m *LibVirtMock) NewMethod(ctx context.Context) error {
    // hand-written — not from moq
    return nil
}
```

**Why this matters**: The next `make generate` overwrites the file. The hand-written method disappears silently. Tests that relied on it compile but the mock no longer has the behavior.

**Preferred action**: Extend the actual `LibVirt` interface, then run `make generate`. The mock is always derived from the interface.

---

### Run Race Detector on All Concurrent Tests

**Detection**:
```bash
rg 'go test' Makefile
grep -n "go test" Makefile
```
Check that all `go test` invocations in Makefile include `-race`.

**Signal**:
```makefile
test:
    go test ./internal/...   # no -race
```

**Why this matters**: kvm-exporter's domain collection is concurrent. Tests that exercise collection without `-race` can pass even with data races — the race detector is not enabled by default. Race conditions surface only under load in production.

**Preferred action**:
```makefile
test:
    go test -race ./internal/...
```

**Version note**: `-race` requires CGO on Linux. Since kvm-exporter already requires `CGO_ENABLED=1` for libvirt, this is always available.

---

### Assert Metric Presence, Not Exact Count

**Detection**:
```bash
rg 'assert\.Len\(t, metrics' --type go internal/
grep -n "assert.Len" internal/libvirt/*_test.go
```

**Signal**:
```go
assert.Len(t, metrics, 3, "expected exactly 3 metrics")
```

**Why this matters**: Adding a label to an existing metric (e.g., adding `numa_node` label to steal time) changes the cardinality. The exact-count assertion breaks, but the metric is still correct. Tests become a maintenance burden.

**Preferred action**: Assert metric presence and specific label values rather than total count:
```go
assert.NotEmpty(t, metrics)
// Find the specific metric you care about:
found := false
for _, m := range metrics {
    if strings.Contains(m.Desc().String(), "kvm_domain_libvirt_steal_time") {
        found = true
    }
}
assert.True(t, found, "steal time metric must be present")
```

---

### Add test-metrics.sh Coverage for Every New Collector

**Detection**:
```bash
grep -n "check_metric" test/test-metrics.sh
# Compare against collector list in DISABLED_COLLECTORS docs
```
If a new collector name does not appear in `test-metrics.sh`, its E2E coverage is missing.

**Signal**:
New collector `hugepages` is added to `internal/libvirt/hugepages.go` but `test/test-metrics.sh` has no `check_metric "kvm_domain_hugepages_bytes"` line.

**Why this matters**: Unit tests cover the mock path. The E2E cluster deploys the exporter against real VMs. If hugepages collection silently returns no metrics (e.g., smaps not readable), no test catches it.

**Preferred action**: For every new collector, add at minimum one `check_metric "kvm_..."` line to `test/test-metrics.sh` targeting the primary metric the collector emits.

---

## Error-Fix Mappings

| Error | Root Cause | Fix |
|-------|-----------|-----|
| `undefined: LibVirtMock` | `interface_mock_gen.go` not regenerated after interface change | Run `make generate` |
| `DATA RACE` in `go test -race` | Goroutine-per-domain accessing shared state without sync | Check `sync.Map` usage; add mutex if using plain map |
| `no such file or directory: /run/libvirt/libvirt-sock-ro` (unit test) | Test code directly instantiating real libvirt connection | Inject mock via interface; never connect to real libvirt in unit tests |
| E2E `make test-all` fails at Kind node image build | Custom Kind Dockerfile outdated after Ubuntu/libvirt version bump | Rebuild base image: `make build-test-image` |
| `check_metric` fails in `test-metrics.sh` | New collector not returning metrics in E2E cluster | Check collector enabled/disabled status; add debug logging; validate cgroup paths on Kind node |

---

## Detection Commands Reference

```bash
# Check if race detector is in Makefile test targets
grep -n "go test" Makefile

# Find any test that skips -race
rg 'go test [^-]' Makefile

# Find hand-edits to generated mock file
git log --oneline internal/libvirt/interface_mock_gen.go

# Find exact-count metric assertions (fragile)
rg 'assert\.Len\(t, metrics' --type go internal/

# Find uncovered collectors in E2E script
grep "check_metric" test/test-metrics.sh

# Run unit tests with race detector
go test -race -v ./internal/...
```

---

## See Also

- `references/kvm-exporter.md` — Testing section for E2E infrastructure setup and `make test-all` targets
- `references/concurrency-patterns.md` — Concurrency patterns that need race-detector-clean tests

SKILL.md 6 KB

---
name: kubernetes
description: "Kubernetes operations: debugging, security, RBAC, and infrastructure tooling."
user-invocable: false
context: fork
agent: kubernetes-helm-engineer
routing:
  triggers:
    # from kubernetes-debugging
    - "kubernetes debug"
    - "pod failure"
    - "pod crashloop"
    - "kubectl logs"
    - "OOMKilled"
    - "pod pending"
    # from kubernetes-security
    - "kubernetes security"
    - "k8s RBAC"
    - "RBAC setup"
    - "pod security policy"
    - "network policy"
    # from cobalt-core
    - "cobalt core"
    - "cobaltcore"
    - "kvm-exporter"
    - "kvm exporter"
    - "hypervisor metrics"
    - "libvirt exporter"
    - "cloud hypervisor"
  category: kubernetes
  pairs_with:
    - assessment
    - programming
    - prometheus-grafana-engineer
---

# Kubernetes Skill

Three domains: **debugging** (pod triage, networking, resources), **security** (RBAC, pod hardening, network isolation, supply chain), and **cobaltcore** (KVM exporter, hypervisor metrics). Select by request signal, then follow the phases below.

Always specify `-n <namespace>` in every kubectl command. Use read-only commands to gather evidence before proposing changes.

---

## Domain Selection

| Signal | Domain |
|--------|--------|
| CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending | Debugging |
| Service unreachable, DNS failure, port-forward | Debugging (network) |
| CPU throttling, memory limit, disk pressure | Debugging (resources) |
| RBAC, permissions, roles, ServiceAccount | Security (access) |
| Pod hardening, container security, PodSecurity | Security (pods) |
| NetworkPolicy, default-deny, namespace isolation | Security (network) |
| Image signing, secrets, admission control | Security (supply chain) |
| KVM exporter, cobaltcore, hypervisor metrics | Cobaltcore |

---

## Phase 1: TRIAGE

### Debugging Triage Flow

Follow this sequence for every pod or workload issue. Do not skip steps -- many failures are only visible in events and describe output, not in logs.

```bash
kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
kubectl exec -it <pod-name> -n <namespace> -c <container-name> -- /bin/sh
```

Always check `--previous` logs for crashed containers before current logs -- restarting destroys them permanently.

**Diagnosis routing**:

| Symptom | Action |
|---------|--------|
| CrashLoopBackOff, ImagePullBackOff, Pending, FailedScheduling | Check describe output for events, previous logs, image pull errors |
| Service unreachable, DNS failure | Check service endpoints, CoreDNS, NetworkPolicy below |
| CPU throttling, OOMKill, disk pressure | Check resource limits, requests vs actual, node capacity |
| "no endpoints available for service" | Compare svc selector with pod labels |

**Network debugging**:

```bash
# Verify service has endpoints
kubectl get endpoints <service-name> -n <namespace>
# DNS lookup from inside cluster
kubectl run dns-debug --rm -it --restart=Never --image=busybox:1.36 -n <namespace> -- \
  nslookup <service-name>.<namespace>.svc.cluster.local
# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Port-forward for local testing
kubectl port-forward svc/<service-name> -n <namespace> 8080:80
```

### Security Domain Selection

For security requests, provide concrete YAML manifests from the patterns below. Reference-backed specifics, not generic advice.

**RBAC patterns**: Prefer namespace-scoped Roles over ClusterRoles. Write exact verbs and resources. Create dedicated ServiceAccounts per workload. Set `automountServiceAccountToken: false` on pods that need no API access.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-team
  name: deployment-reader
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]
```

**Pod security**: Enforce PodSecurity labels at namespace level. All containers: `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, `capabilities: drop: ["ALL"]`. Use distroless base images. Pin image digests.

```yaml
labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/warn: restricted
```

**Network policies**: Start with default-deny for ingress and egress. Add allow-list rules per service. Always allow DNS egress (UDP/TCP 53).

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
```

### Cobaltcore Domain

Components: KVM Exporter (`cobaltcore-dev/kvm-exporter`). Load `references/cobalt-kvm-exporter.md` for architecture, metric catalogs, configuration, and deployment. Pair with `go-patterns` for code, `prometheus-grafana-engineer` for metrics.

---

## Phase 2: DIAGNOSE / RESPOND

**Debugging**: Follow the triage flow. Gather evidence with read-only commands before proposing changes.

**Security**: Provide copy-paste-ready YAML using the patterns in Phase 1.

**Cobaltcore**: Use component-specific references for architecture, metrics, concurrency patterns, and testing.

---

## Phase 3: VERIFY

- **Debugging**: Confirm the fix resolves the symptom with the same triage commands.
- **Security**: Validate against the PodSecurity standards and RBAC least-privilege patterns above.
- **Cobaltcore**: Verify against component test patterns in `references/cobalt-testing-patterns.md`.

---

## Deep References

| Signal | Reference | Content |
|--------|-----------|---------|
| KVM exporter architecture, metrics, config | `references/cobalt-kvm-exporter.md` | Full component reference (463 lines) |
| Cobaltcore concurrency, goroutine, semaphore | `references/cobalt-concurrency-patterns.md` | Go concurrency patterns (268 lines) |
| Cobaltcore testing, mock, Kind cluster | `references/cobalt-testing-patterns.md` | Testing strategies (271 lines) |