{
  "schema_version": 1,
  "skill_name": "platform-engineering",
  "evals": [
    {
      "id": "ci-cd-pipeline-design",
      "prompt": "We are migrating a monolith to microservices and I need to design the CI/CD pipeline for one of the new services. Requirements: every merge to main must produce a deployable artifact, staging should look like production, and we want to move to GitOps-based promotion instead of pushing from a laptop. What should the pipeline look like and where do the approval gates belong?",
      "expected_output": "A CI/CD pipeline design with distinct stages: build and test on every pull request, artifact production and registry push on merge to main, environment promotion via GitOps (Argo CD or Flux) where the cluster reconciles to Git as the source of truth, and approval gates placed at the environment boundary (staging to production) rather than inside build. The design covers OIDC-based cloud authentication instead of static keys, image signing and provenance, semantic versioning of artifacts, and a rollback path that reverts the Git manifest rather than redeploying a build.",
      "assertions": [
        "The response defines a multi-stage pipeline where build and test run on pull requests and artifact production runs on merge to main",
        "The response uses GitOps promotion with Git as the source of truth for environment state",
        "The response places manual or protected approval gates at the staging-to-production boundary, not inside the build stage",
        "The response replaces static cloud credentials with OIDC workload identity for pipeline authentication",
        "The response covers artifact signing, provenance, and a Git-revert-based rollback path"
      ]
    },
    {
      "id": "iac-review-module-structure",
      "prompt": "A new platform team member wrote Terraform modules for our AWS landing zone and I need to review them before they are used across the organization. The modules combine VPC, EKS, and IAM in one root module, store state locally, and hardcode account IDs in the code. What should the review focus on and what structural changes should I request?",
      "expected_output": "An infrastructure-as-code review record that flags the root module as too broad and requests composition from smaller, single-purpose modules; requires a remote state backend with locking (e.g., S3 with DynamoDB or Terraform Cloud) instead of local state; removes hardcoded account IDs in favor of variables, data sources, or provider-level configuration; checks for state exposure of secrets and requires secret material to come from a secret manager or dynamic credentials rather than plaintext variables; validates that for_each and module composition are used instead of copy-pasted resource blocks, and that outputs expose the minimal surface area consumers need.",
      "assertions": [
        "The review flags monolith root modules and recommends composing smaller single-purpose modules",
        "The review requires a remote state backend with locking instead of local state",
        "The review flags hardcoded account IDs and requires parameterization via variables or data sources",
        "The review addresses secrets in state and requires secret material to come from a secret manager or dynamic credentials",
        "The review recommends for_each-driven composition and a minimal output surface area"
      ]
    },
    {
      "id": "observability-strategy-design",
      "prompt": "Our new checkout service goes live next month and we currently have no monitoring, no dashboards, and no alerting. I want an observability strategy that tells us when the service is broken and lets us debug latency regressions after release. Where do I start and what should be the default contract for every service we build?",
      "expected_output": "An observability strategy organized around the three signals: metrics (RED and USE patterns via Prometheus counters, gauges, histograms), logs (structured JSON shipped to a Loki-style store), and traces (OpenTelemetry spans with W3C trace context propagated across service boundaries). The strategy defines an observability contract for every service: a /metrics endpoint, structured logging, at least one RED dashboard, recording rules for latency and error rate, burn-rate-based alerting tied to an error budget rather than threshold guessing, and a golden signal dashboard as code in Git so dashboards are reviewable and reproducible. It also specifies the cardinality and label hygiene rules that keep the metrics usable at scale.",
      "assertions": [
        "The response structures the strategy around metrics, logs, and traces with named tooling for each signal",
        "The response defines a default observability contract every service must meet, including a metrics endpoint and structured logging",
        "The response uses RED and USE patterns and burn-rate alerting tied to an error budget",
        "The response requires dashboards-as-code stored in Git so they are reviewable and reproducible",
        "The response covers OpenTelemetry tracing with W3C context propagation and metrics cardinality hygiene"
      ]
    },
    {
      "id": "secret-management-design",
      "prompt": "We run Kubernetes with GitOps and currently store database credentials in plaintext Kubernetes Secrets committed to a private Git repository. I want a secret-management design that stops putting credentials in Git, handles rotation, and works for both static config secrets and dynamically generated credentials. Which approach fits and what is the migration path?",
      "expected_output": "A secret-management design that separates the problem into static configuration secrets and dynamic credentials. For static secrets in GitOps, the design recommends SOPS-encrypted files with age or KMS keys for small configs, or Sealed Secrets for cluster-bound secrets, with External Secrets Operator to sync from a central store such as HashiCorp Vault or a cloud secret manager. For dynamic credentials (database passwords, cloud access keys), it recommends Vault dynamic secrets with short TTLs and automatic revocation rather than long-lived static secrets. The design covers the migration path: inventory current plaintext secrets, encrypt at rest in Git, rotate existing credentials during the cutover, and wire renewal/rotation into the workload lifecycle. It also calls out that SOPS and Sealed Secrets are static-only and do not provide rotation or audit, which is why a dynamic store is needed for anything that changes.",
      "assertions": [
        "The response separates static configuration secrets from dynamically generated credentials and recommends a different mechanism for each",
        "The response recommends SOPS or Sealed Secrets for static secrets in Git and External Secrets Operator for syncing from a central store",
        "The response recommends Vault dynamic secrets with short TTLs and revocation for database and cloud credentials",
        "The response includes a migration path that inventories plaintext secrets and rotates credentials at cutover",
        "The response identifies that SOPS and Sealed Secrets lack rotation and audit, motivating a dynamic store"
      ]
    },
    {
      "id": "cloud-architecture-assessment",
      "prompt": "Our leadership wants to move the entire platform to a second cloud provider in parallel with AWS to reduce vendor lock-in and cut costs. I have been asked to assess whether this is a good idea before we commit. What factors should the assessment weigh, and what should the recommendation look like?",
      "expected_output": "A cloud architecture assessment that evaluates the multi-cloud proposal against decision criteria rather than assuming multi-cloud is inherently beneficial. It weighs regulatory data-residency requirements, the small set of services where lock-in actually matters (object storage, Kubernetes), operational cost of duplicated IAM, networking, and skills across providers, and calls out that multi-cloud is not a cost-savings strategy because egress charges and duplicated operational overhead usually outweigh rate differences. The assessment recommends single-cloud for the default path with provider-agnostic abstractions (Terraform/OpenTofu providers, Kubernetes) used as escape hatches rather than abstraction layers for everything, and it covers cost governance: tagging, budget alerts, right-sizing, and committed-use discounts applied to the primary provider before adding a second one.",
      "assertions": [
        "The response evaluates multi-cloud against decision criteria including data residency and lock-in, not as a blanket strategy",
        "The response states that multi-cloud is not a cost-savings strategy and explains egress and duplicated-operations costs",
        "The response limits provider-agnostic abstraction to escape-hatch patterns such as Terraform providers and Kubernetes",
        "The response recommends applying cost governance (tagging, budgets, right-sizing, committed use) before expanding providers",
        "The response treats single-cloud as the default and multi-cloud as justified only by specific regulatory or availability requirements"
      ]
    },
    {
      "id": "golden-path-self-service-portal",
      "prompt": "Our developers keep opening tickets to get a database, a namespace, and a CI pipeline for each new service, and the platform team is the bottleneck. I want to design a golden path with a self-service portal so developers can provision their own stack. How do I scope the first golden path and what guardrails should the portal enforce?",
      "expected_output": "A golden-path and self-service portal design that treats the platform as a product with developers as customers. The first golden path is scoped to the most common request (a new service: repository, CI pipeline, namespace, database, observability defaults) and implemented as a repeatable template with Terraform modules, a pipeline template, and an API-first portal that calls those templates behind the scenes. The design enforces guardrails as policy rather than documentation: least-privilege IAM generated from the request, budget and quota limits, mandatory observability and security baselines, and an escape hatch that lets developers leave the golden path with an exception review instead of forking it. It prioritizes self-service over tickets, measures cognitive load and time-to-first-deploy as the primary success metrics, and keeps every scaffolded artifact in Git for review.",
      "assertions": [
        "The response scopes the first golden path to a single high-frequency request and implements it as a repeatable template",
        "The response uses an API-first self-service portal that provisions through templates rather than tickets",
        "The response enforces guardrails as policy, including least-privilege IAM, budget limits, and observability baselines",
        "The response includes an escape hatch with exception review so the golden path is a paved road, not a cage",
        "The response measures success by developer cognitive load and time-to-first-deploy and keeps scaffolded artifacts in Git"
      ]
    },
    {
      "id": "platform-applicability-and-stop-decision",
      "prompt": "A small company wants to build an internal developer platform because platform engineering is popular. It has four developers, little custom software, no measured delivery bottleneck, and a six-month budget. Assess the proposal and state what evidence would justify a small first slice or a stop decision.",
      "expected_output": "A bounded applicability assessment that tests strategic software dependence, internal user population, observed friction, opportunity cost, and time horizon before recommending investment. It proposes a minimum valuable slice only if a measurable workflow problem exists, names a baseline and adoption/value thresholds, and preserves a stop or reshape decision if evidence does not improve.",
      "assertions": [
        "The response does not assume a platform is justified merely because the practice is popular",
        "The response evaluates strategic software dependence, internal users, measured friction, and budget/time horizon",
        "The response defines a small first slice only when a measurable workflow problem is found",
        "The response includes baseline, adoption or value thresholds, and an explicit stop or reshape rule",
        "The response distinguishes evidence from assumptions and does not claim success from component delivery alone"
      ]
    },
    {
      "id": "self-service-contract-and-exception",
      "prompt": "Security requires every deployed service to have a signed image and a vulnerability scan, but application teams must retain control of their pipelines. Design the platform contract and the process for a team that needs a temporary exception for an approved legacy image.",
      "expected_output": "A developer-owned delivery design with reusable checks or policy interfaces, a documented input/decision/evidence flow, and a narrow exception record. The exception process identifies scope, authorization, risk, compensating controls, accountable owner, approver, expiry, review, and a path to improve the product or policy when exceptions recur.",
      "assertions": [
        "The response keeps application teams in control of their pipelines while providing reusable compliance checks",
        "The response defines policy inputs, versioned decision evidence, failure response, and ownership",
        "The response makes the legacy-image exception narrow, time-bounded, and explicitly authorized",
        "The response requires risk assessment, compensating controls, accountable owner, approver, expiry, and review",
        "The response treats recurring exceptions as product or policy backlog evidence rather than permanent bypasses"
      ]
    },
    {
      "id": "platform-observability-response-ownership",
      "prompt": "Our platform provides Prometheus and Grafana, but teams complain that alerts have no clear owner and dashboards are unused. Define an observability ownership and adoption workflow for a newly onboarded service.",
      "expected_output": "A workflow that separates service-owned instrumentation and symptom response from platform-owned collection, storage, access, and platform failures. It publishes SLIs/SLOs and a response matrix with alert recipient, diagnostic action, escalation, and retained evidence, then measures adoption and user friction and iterates based on feedback.",
      "assertions": [
        "The response assigns instrumentation and service-symptom response to the service owner while assigning collection/storage/platform failures to the platform owner",
        "The response requires published SLIs/SLOs and a response matrix with recipient, diagnostic action, escalation, and evidence",
        "The response includes an adoption workflow that measures dashboard usefulness, alert friction, and coverage",
        "The response provides a feedback loop for false positives, missing signals, and unused dashboards",
        "The response states an observable readiness gate and stops when ownership or failure routing is unknown"
      ]
    }
  ]
}

# Automation Languages — Reference

## Go

- **Use in platform engineering:** CLI tools (Cobra/Viper), Kubernetes operators (controller-runtime), Terraform providers (terraform-plugin-framework), ingress controllers, service mesh sidecars, infrastructure agents
- **Key patterns:** `os/exec` for running system commands, `os/signal` for graceful shutdown, `net/http` for API clients, `cobra.Command` for CLI structure, `viper` for config loading, `retry` patterns via backoff
- **Best practices:** Single binary deployment, cross-compilation (`GOOS=linux GOARCH=arm64`), no runtime dependencies, `go vet` + `staticcheck` in CI, readability over cleverness

## Python

- **Use in platform engineering:** Automation scripts, cloud SDK clients (boto3, google-cloud, azure-mgmt), CI/CD pipeline scripts, configuration validation, integration testing, internal tools
- **Key patterns:** `argparse`/`click` for CLI, `httpx`/`requests` for API calls, `pydantic` for config validation, `pyyaml` for config parsing, `rich`/`click` for CLI output formatting, `pathlib` for file operations
- **Best practices:** Type hints everywhere (mypy strict), `if __name__ == "__main__":` entry point, installable via `pip install` (entry_points in setup.cfg/pyproject.toml), dependency pinning for reproducibility, `--dry-run` flag on all mutating operations

## Bash / POSIX Shell

- **Use in platform engineering:** Bootstrap scripts, CI/CD glue, Dockerfile commands, container entrypoints, developer tool wrappers, provisioning one-shots
- **Key patterns:** `set -euo pipefail` for safety, argument parsing with `getopts` or `while case`, `mktemp` for temp files, `trap cleanup EXIT` for teardown, `${var:-default}` and `${var:?required}` patterns
- **Best practices:** ShellCheck in CI, prefer `[[ ]]` over `[ ]` in Bash, quote all variable expansions, use `printf` over `echo`, keep scripts short (beyond ~100 lines → Python or Go), `set -x` for debugging in development only

# CI/CD Pipelines — Reference

## Pipeline Platforms

### GitHub Actions

- **Workflow structure:** `.github/workflows/*.yml` — triggers, jobs, steps, matrix builds
- **Key patterns:** reusable workflows (`uses:` with `{owner}/{repo}/.github/workflows/{name}@{ref}`), composite actions, OIDC for cloud auth, artifacts/pages for delivery
- **Secrets:** GitHub Actions secrets, environment-level secrets, OIDC as an alternative to static keys
- **Matrix builds:** `strategy.matrix` for cross-platform/testing, `fail-fast` for early exit
- **Self-hosted runners:** scale sets, labels, network isolation, ephemeral runners

### GitLab CI

- **Pipeline structure:** `.gitlab-ci.yml` — stages, jobs, needs (DAG), artifacts, cache
- **Key patterns:** multi-project pipelines, parent-child pipelines, merge request pipelines, scheduled pipelines
- **Runners:** shared vs specific, Docker executor, Kubernetes executor, tags, concurrency limits
- **Registry:** GitLab Container Registry integration, dependency proxy

### Forgejo CI / Gitea Actions

- **Structure:** `.forgejo/workflows/*.yml` or `.gitea/workflows/*.yml` — compatible with GitHub Actions syntax
- **Runners:** Forgejo Runner (act-based), self-hosted, labels for platform targeting
- **Key differences from GitHub Actions:** Lighter ecosystem, smaller action marketplace, often need to self-host runners
- **Secrets:** Forgejo repository/organization secrets, no OIDC built-in (use manual token exchange)

### Jenkins

- **Pipeline structure:** `Jenkinsfile` — declarative (`pipeline { }`) vs scripted (`node { }`)
- **Key concepts:** agents, stages, steps, post-build actions, shared libraries, Blue Ocean
- **Cloud integration:** Jenkins X for Kubernetes, plugin ecosystem, custom agents via Docker

### CircleCI

- **Pipeline structure:** `.circleci/config.yml` — orbs, executors, jobs, workflows (DAG)
- **Key concepts:** contexts (env sharing), workspaces/persist-to-workspace, parallelism, test splitting
- **Orbs:** reusable config packages (official and community orbs for AWS, Slack, browsers, etc.)

## GitOps

### Argo CD

- **Core model:** Declarative GitOps — desired state in Git repository, Argo CD syncs to cluster
- **Key concepts:** Applications, Projects, Sync strategies (auto/manual), sync waves, prune policies, health checks
- **Multi-cluster:** Hub-and-spoke, cluster registration, RBAC per cluster
- **Progressive delivery:** Rollouts, canary deployments, blue-green, traffic mirroring (Argo Rollouts add-on)
- **Patterns:** App-of-apps, Kustomize/Helm integration, config management plugins (CMP), ApplicationSets for multi-env/deployment

### Flux

- **Core model:** GitOps toolkit — source → kustomize/helm → sync to cluster
- **Key components:** Source Controller, Kustomize Controller, Helm Controller, Notification Controller, Image Automation
- **Key concepts:** GitRepository/Bucket sources, Kustomization/HelmRelease, OCIRepository, ImagePolicy
- **Multi-tenancy:** Namespace isolation, cross-namespace references, access controls

## Release Automation

- **Semantic versioning:** `MAJOR.MINOR.PATCH` — breaking changes, features, fixes; pre-release suffixes, build metadata
- **Changelog generation:** Conventional Commits → automated changelog (git-cliff, standard-version, semantic-release)
- **Artifact provenance:** SLSA levels, attestation (in-toto), SBOM generation (Syft, Trivy), signing (Cosign)
- **Release gates:** Manual approvals (GitHub Environments, GitLab Deployments), automatic rollback on health check failure
- **Artifact registries:** Container registries (Docker Hub, GHCR, GitLab Registry, ECR, GAR), package registries (NPM, PyPI, Maven)

# Cloud Platforms — Reference

> **Last Updated:** 2026-08-03
> Patterns and decision guidance for cloud platform architecture. Operational configuration belongs to the tool skills (`terraform`, `kubernetes`, `docker-compose`, `traefik`); this file carries judgment frameworks.

## Provider Selection — Decision Guidance

### AWS

- **Core services:** VPC (subnets, route tables, NAT, security groups, NACLs, VPC peering, Transit Gateway), EC2 (instances, AMIs, auto-scaling, launch templates, spot), EKS (managed K8s, node groups, Fargate, IRSA), S3 (buckets, versioning, lifecycle, replication, presigned URLs), IAM (users, roles, policies, instance profiles, OIDC), Route53 (DNS, alias records, health checks, routing policies)
- **Common patterns:** Shared VPC (central networking team), multi-account (Control Tower, Organization, SCPs), IRSA for EKS pod IAM, S3 backend for Terraform state (bucket + DynamoDB lock), CodeBuild/CodePipeline for CI, CloudFront for CDN
- **When AWS fits:** Broadest service catalog, deepest managed-K8s and IAM maturity, most mature IaC ecosystem and third-party tooling. Strong default when the team already has AWS skills or needs services no other provider matches.

### GCP

- **Core services:** VPC (subnets, firewall rules, Cloud NAT, VPC peering, Shared VPC), GKE (K8s, node auto-repair/auto-upgrade, Workload Identity for pod IAM), Cloud Storage (buckets, nearline/archive, object lifecycle), IAM (roles, custom roles, service accounts, Workload Identity Federation), Cloud DNS (managed zones, DNS forwarding, policy-based routing)
- **Common patterns:** Shared VPC (host project + service projects), workload identity federation (no static keys), Artifact Registry, Cloud Build CI, Terraform state via Cloud Storage
- **When GCP fits:** Kubernetes-first workloads (GKE is the closest managed-K8s experience), data/ML platform strengths, most aggressive committed-use discounts, clean identity-federation story for keyless workloads.

### Azure

- **Core services:** VNet (subnets, NSGs, Azure Bastion, VPN Gateway, VNet peering), AKS (K8s, node pools, managed identity, Azure AD integration), Blob Storage (containers, tiers, lifecycle, Azure Files), RBAC (roles, custom roles, managed identities, service principals), DNS (public/private zones, alias records, Azure DNS Private Resolver)
- **Common patterns:** Hub-and-spoke networking (central firewall), managed identity for pod IAM (AKS with workload identity), Terraform state via Azure Storage, Azure DevOps pipelines
- **When Azure fits:** Windows/.NET/Active Directory shops, enterprise compliance and procurement (existing Microsoft agreements), hybrid on-prem connectivity, regulated industries where Azure's compliance footprint is a sales advantage.

## Multi-Cloud and Abstraction

- **Abstraction layers:** Terraform/OpenTofu providers — write once, target any cloud (with provider-specific variance). Pulumi similarly abstracts. Crossplane for K8s-native cloud resource provisioning
- **Governance cost:** State isolation per cloud, IAM duplication per provider, network egress charges (Free Tier per cloud but real cost at scale), skills distribution across cloud teams
- **When multi-cloud is worth it:** Regulatory (data residency), avoiding single-vendor lock-in for critical few services (object storage, K8s), acquisition integration. It is NOT a cost-savings strategy.
- **When multi-cloud is a trap:** Teams assume abstraction layers erase provider differences; they do not. Each provider's IAM model, quota semantics, and operational behavior leak through. A second provider doubles the platform surface for zero resilience unless workloads are actually replicated (active-active or active-passive with real failover testing).
- **Decision rule:** Start single-cloud. Add a second provider only for a named, measurable requirement (residency, availability, acquisition). If the goal is resilience, prove failover works before committing to the second provider.

## Cost Governance Patterns

- **Budget alerts** (each cloud): per-account/project budget with alert thresholds at 50/80/100%, billing exports to a data warehouse for cost analytics
- **Tagging policies:** `CostCenter`, `Environment`, `Owner`, `Service` — enforced at provisioning time (guardrails/Terraform validators), not retroactively
- **Right-sizing:** instance/container resource analysis against utilization, right-size before scaling out
- **Committed use:** reserved instances / committed use discounts / savings plans for steady-state baseline; spot/preemptible for batch and stateless workloads
- **Storage tier policies:** lifecycle rules moving cold data to archive tiers; know the retrieval cost before designing hot paths
- **Egress awareness:** egress charges dominate surprise bills; keep data transfer within a region/zone where possible, and route cross-provider traffic deliberately
- **FinOps cadence:** monthly cost review with owners, anomaly detection on the billing feed, unit-economics per service (see `capacity-and-cost-engineering` for the methodology)

## Security and Identity Patterns

- **Workload identity over static keys:** OIDC federation (IRSA, Workload Identity Federation, managed identity) so pods and CI never hold long-lived cloud keys
- **Multi-account/project structure as the security boundary:** control plane (org/root) separate from workload accounts, SCPs as policy guardrails, audit account for centralized logs
- **Shared responsibility model:** the provider secures the fabric; the platform team owns IAM, network boundaries, data encryption at rest/in transit, and image/artifact supply chain
- **Audit logging:** enable cloud trail/audit logs centrally with retention and alerting on privileged-role usage

## Sources and Dated References

- AWS Well-Architected Framework: https://aws.amazon.com/architecture/well-architected/ (accessed 2026-08-03)
- AWS Organizations multi-account best practices: https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html (accessed 2026-08-03)
- GCP resource hierarchy and IAM: https://cloud.google.com/docs/overview (accessed 2026-08-03)
- Azure cloud adoption framework / landing zones: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ (accessed 2026-08-03)
- FinOps framework and cost optimization: https://www.finops.org/framework/ (accessed 2026-08-03)

# Container Orchestration — Reference

## Kubernetes and k3s

- **Core objects:** Pod, Service, Deployment, StatefulSet, DaemonSet, Ingress, ConfigMap, Secret, PersistentVolume/PVC, Namespace, RBAC (Role/ClusterRole/RoleBinding/ClusterRoleBinding), NetworkPolicy
- **Lifecycle:** Rolling updates, recreate, canary via flags, blue-green via Services; readiness/liveness/startup probes, preStop hooks, pod disruption budgets
- **Scheduling:** nodeSelector, node affinity/anti-affinity, pod affinity/anti-affinity, taints and tolerations, topology spread constraints, resource limits/requests, QoS classes (Guaranteed/Burstable/BestEffort)
- **k3s specifics:** Lightweight Kubernetes (single binary), embedded etcd (or SQLite), Traefik as default ingress, servicelb, HelmController, local-path-provisioner, useful for edge/IoT/development

## Helm

- **Chart structure:** `Chart.yaml`, `values.yaml`, `templates/`, `charts/` (dependencies), `crds/`, `templates/NOTES.txt`, `templates/tests/`
- **Conventions:** helper templates (helpers.tpl), named templates, required values with `required`, global values for cross-chart values, conditionally-enabled subcharts
- **Lifecycle:** `helm create`, `helm lint`, `helm template`, `helm install --values`, `helm upgrade --install`, `helm rollback`, `helm uninstall`, `helm dependency update`
- **Repos:** ChartMuseum, OCI registries (GHCR, ECR, ACR), `helm repo add/update`
- **Best practices:** Pin dependency versions, set resource limits, use release namespace, separate env values files, enable/disable subcharts via `tags` or `condition`

## Kustomize

- **Core model:** Base + overlays — base sets common config, overlays apply env-specific patches
- **Key directives:** `resources`, `patchesStrategicMerge`, `patchesJson6902`, `configMapGenerator`, `secretGenerator`, `namePrefix`, `namespace`, `commonLabels`, `images`, `replicas`, `vars`
- **Patterns:** Multi-environment (dev/staging/prod), multi-cluster, component-based composition, in-line vs file-sourced patches
- **Integration:** `kustomize build` as input to kubectl or ArgoCD, `kustomize edit` for interactive modification

## Docker Compose (Production Patterns)

- **Production concerns:** Health checks (`healthcheck:`), restart policies (`unless-stopped/always`), resource limits (`deploy.resources.limits`), logging drivers, network isolation (custom networks), volume management (named volumes, bind mounts, tmpfs)
- **Multi-service patterns:** Depends-on with health check wait, init containers, sidecar containers (nginx reverse proxy, log shipper), environment file separation (`.env`, multiple `--env-file`)
- **Orchestration compatibility:** Compose file can be used directly (single host), converted to K8s via `kompose`, or used as a local dev environment matching production topology

# Infrastructure as Code — Reference

> **Last Updated:** 2026-08-03
> Patterns and decision guidance for infrastructure-as-code. Operational commands and runbooks belong to the tool skills (`terraform`, `kubernetes`, `docker-compose`); this file carries the judgment frameworks for choosing, structuring, and reviewing IaC.

## Tool Selection — Decision Guidance

### Terraform / OpenTofu

- **Core concepts:** Resources, data sources, providers, state (local, remote backends), modules, variables, outputs, lifecycle rules (`create_before_destroy`, `prevent_destroy`)
- **State management:** Remote backends (S3 + DynamoDB, GCS, Azure Storage, Terraform Cloud), state locking, state migration, workspaces for env separation, `terraform state` subcommands (mv, rm, pull, push)
- **Module design:** Composition (call smaller modules), version pinning, registry conventions (hashicorp/terraform-google-modules), output minimal surface area, internal vs published modules
- **Advanced patterns:** `for_each`/`count` for dynamic resources, `templatefile` for config injection, file/external data sources for bridge to external systems, provisioners as last resort (remote-exec/local-exec)
- **OpenTofu specifics:** Drop-in Terraform replacement, same HCL syntax, OSS license (no BSL change), `tofu` CLI, supports encryption at rest in state natively, enhanced provider signing
- **When to choose:** The default for cloud resource provisioning. Largest provider ecosystem, most transferable skills, works across all three major clouds. Choose OpenTofu when license/BSL or state encryption is a hard requirement.

### Pulumi

- **Core model:** Infrastructure as real code — Go, Python, TypeScript, .NET, Java, YAML
- **Key concepts:** Programs (stack definitions), stacks (env instances), resources, components (custom abstractions), providers (Pulumi-native, TF bridge), outputs, config/secret management
- **State:** Pulumi Cloud (managed), self-managed backends (S3, GCS, Azure Blob S3-compatible), state encryption
- **Automation API:** Embed Pulumi in applications (CI/CD, self-service platforms), inline updates, preview + deploy in code
- **Bridge to Terraform:** TF bridge adapter wraps existing TF providers as native Pulumi providers — convenient but adds a layer
- **When to choose:** Teams that need real programming-language logic (loops, conditionals, tests) inside the IaC layer, or who are building self-service automation and want the Automation API.

### Ansible

- **Core model:** Agentless — SSH/WinRM transport, push-based, YAML playbooks, Jinja2 templating
- **Key concepts:** Inventory (static, dynamic from cloud APIs), modules (idempotent operations), roles (reusable content packages), playbooks (execution order), variables and facts, handlers (notify-based triggers)
- **Best practices:** Role-based layout, vault for secrets, molecule for testing, ansible-lint, `--check --diff` for dry-run, `--limit` for targeted execution
- **Use case in platform engineering:** Day-2 configuration (post-provisioning), OS hardening, agent installation, but generally less suited than Terraform for cloud resource provisioning
- **When to choose:** Configuration management of existing servers and day-2 operations; not the right tool for the initial cloud resource graph.

### CloudFormation / CDK

- **CloudFormation:** Native AWS IaC — JSON/YAML templates, stacks, nested stacks, change sets, drift detection, stack sets (multi-account, multi-region)
- **CDK (Cloud Development Kit):** CloudFormation as real code (TypeScript, Python, Go, Java, C#) — constructs (L1/L2/L3 abstraction), `cdk synth` → CloudFormation template, `cdk deploy` / `cdk diff`, context, aspects, permissions boundaries
- **CDKTF (CDK for Terraform):** Bridge for Terraform providers in CDK languages — cross-platform between AWS and non-AWS providers
- **When to choose:** AWS-only shops that want native drift detection and change-set review, or teams already writing real code who prefer CDK's type safety over HCL.

## State and Drift Governance

- **State is a source of truth, not a database:** treat state as a serialized representation of the resource graph, never edit it directly; all changes go through `plan`/`apply` (or the equivalent)
- **Remote backend with locking is non-negotiable:** local state is a team-of-one anti-pattern; choose S3+DynamoDB, GCS, Azure Storage, or Terraform Cloud/Pulumi Cloud and make locking explicit
- **Drift detection cadence:** run periodic plans (`plan` on a schedule, `drift detect` in Terraform Cloud, or cloud-native drift tools) and review unintended diffs before they become incidents
- **Workspaces vs directories:** prefer directory-per-environment with shared modules over workspaces when environments differ materially; use workspaces only for near-identical instances
- **Import-before-manage:** adopt pre-existing resources with `terraform import`/state moves rather than `delete + recreate`; plan for state surgery (`state mv`, `state rm`) only with a locked state and a reviewed plan

## Secrets in IaC

- **Never commit secrets in plaintext:** secrets in HCL/JSON/YAML drift into state and logs; use provider-native secret references (`data "aws_secretsmanager_secret_version"`), Vault dynamic credentials, or SOPS/age for encrypted-at-rest config
- **Prefer dynamic credentials:** database and cloud keys should come from Vault dynamic secrets or managed identity (IRSA, Workload Identity) rather than static long-lived keys
- **State encryption:** OpenTofu encrypts state natively; on Terraform, encrypt the state backend at rest and restrict state read access (state holds secrets)
- **Mark sensitive outputs:** `sensitive = true` so values are redacted in logs and plan output; keep the full secret in the manager, only a reference in IaC

## Review Checklist Patterns

- **Composition over monoliths:** root modules should call child modules; a module that owns VPC + cluster + IAM + app is a candidate for splitting
- **Parameterize environment specifics:** no hardcoded account IDs, regions, or names; variables + data sources + consistent naming convention
- **Minimal outputs:** expose only what consumers need; every output is API surface
- **Plan review before apply:** review the plan for unintended replaces/deletes, not just additions; enforce a human gate on destructive changes
- **Tagging and cost attribution:** consistent tags (`CostCenter`, `Environment`, `Owner`, `Service`) enforced at plan time by guardrails/validators
- **Lifecycle rules:** `prevent_destroy` on irreplaceable resources (databases, state backends); `create_before_destroy` where downtime matters

## Sources and Dated References

- OpenTofu documentation and state encryption: https://opentofu.org/docs/ (accessed 2026-08-03)
- Terraform module composition best practices: https://developer.hashicorp.com/terraform/tutorials/modules (accessed 2026-08-03)
- Pulumi documentation (stacks, state, Automation API): https://www.pulumi.com/docs/ (accessed 2026-08-03)
- Ansible best practices (roles, vault, molecule): https://docs.ansible.com/ansible/latest/tips_tricks/sample_setup.html (accessed 2026-08-03)
- AWS CDK reference: https://docs.aws.amazon.com/cdk/v2/guide/home.html (accessed 2026-08-03)

# Observability — Reference

## Metrics: Prometheus

- **Core model:** Pull-based timeseries — scrape targets, service discovery, metric exposition format (`/metrics` endpoint)
- **Metric types:** Counter (cumulative, only increases), Gauge (up/down), Histogram (bucketed durations/sizes), Summary (quantile-based)
- **Recording rules:** Compute new timeseries from existing ones — `rate(...)[5m]` for per-second averages, `histogram_quantile()` for latency percentiles, aggregation via `sum() by ()` / `topk()`. Stored as new metric in Prometheus, faster than ad-hoc query
- **Alerting rules:** Vector → alert — `for:` duration eliminates flapping, severity labels (critical/warning/info), routing via Alertmanager to PagerDuty, Slack, email, etc.
- **Service discovery:** Kubernetes (pod annotations, kubelet), file_sd (JSON/YAML files), Consul, EC2, DNS
- **Best practices:** Use `rate()` not `irate()`, prefer histograms over summaries (aggregatable), label hygiene (cardinality limits, structured label naming)

## Dashboards: Grafana

- **Core model:** Data source abstraction — panel types (time series, bar, stat, table, gauge, logs), variables (interval, datasource filter), dashboard-as-code via JSON provisioning
- **Provisional dashboards:** JSON files in `provisioning/dashboards/` — auto-imported on Grafana startup. YAML datasource config in `provisioning/datasources/`. Version-controlled in Git alongside application config
- **Key patterns:** Template variables for environment switching, repeat panels per label value, annotations from Prometheus alerts, mixed data sources per panel, transformations (merge, group by, rename)
- **Best practices:** Single dashboard per service, row per concern (traffic, errors, latency, saturation), no more than 15 panels per row, dashboard links for navigation, `$__interval` for adaptive time range

## Logs: Loki

- **Core model:** Label-based log aggregation — indexes labels (not full text), stores compressed chunks in object storage. Promtail/Alloy/Fluent Bit for log shipping
- **LogQL:** `{label=~"value"} |= "error" \| json` for label matchers + content filters + pipeline stages. `rate()` for log error rates, `count_over_time()` for volume monitoring
- **Best practices:** CRI-O/Docker log format handling, structured logging (JSON), label cardinality limits, retention per storage tier (hot/warm/cold), multi-tenancy via label enforcement

## Tracing: OpenTelemetry

- **Core model:** Spans (operation units) → Traces (span DAG) → context propagation via W3C TraceContext headers
- **Signals:** Traces (request flow), Metrics (OTLP), Logs (via OTLP or file export) — unified in OpenTelemetry Collector
- **Components:** SDK (instrumentation libraries for Go, Python, JS, Java, etc.), Collector (receiver → processor → exporter pipeline), sampling (head-based, tail-based for storage cost management)
- **Instrumentation:** Auto-instrumentation (agent injection for Java/Python/.NET/Node), manual instrumentation (create spans, add attributes/events), existing library instrumentation (HTTP, gRPC, DB clients, messaging)

# Platform product evidence

Use this reference when deciding whether to start or expand an internal platform, shaping its first valuable slice, or reviewing whether it creates measurable value.

## Applicability and intake

Frame the opportunity before selecting tools:

1. Name the strategic software outcome and the internal teams who need it. If custom software is not material to the organization’s success, record why a platform still has a credible efficiency, security, or governance return.
2. Gather evidence of friction: queue time, repeated manual work, failure/rework, cognitive load, incident recovery, or duplicated cloud and staffing cost. Separate observed measurements from stakeholder assertions.
3. Identify the highest-frequency workflow and its grain (for example, “new service to first production deployment”). Define the minimum valuable slice, early adopters, dependencies, and an explicit escape hatch.
4. Name a technical product owner and the platform domain owners. Each owner needs a backlog, decision rights, customer feedback channel, service interface, and response responsibility.
5. State adoption, time-to-value, reliability, security, and cost hypotheses with a baseline, target, measurement window, and stop/escalation rule. Do not claim success from delivery of platform components alone.

Stop or reshape the investment when the internal-user population is too small, the workflow is not strategically important, the measured friction is trivial, or early adopters do not use the slice after a reasonable enablement pass. Preserve the evidence and the reason; a stopped platform experiment is a valid outcome.

## Outcome and self-service contracts

For every dependency owned by another team, capture either a self-service API or a testable outcome contract. A contract should state:

| Field | Required question |
|---|---|
| Consumer and owner | Who invokes it, who maintains it, and who responds to failure? |
| Outcome | What observable result is guaranteed? |
| Inputs and authorization | Which fields, identities, scopes, and quotas are accepted? |
| Idempotency and lifecycle | What happens on retry, duplicate request, update, and deletion? |
| Evidence | What audit, policy, cost, and status record is retained? |
| SLO and failure | What latency/availability target applies, and how is an actionable failure returned? |
| Escape hatch | What can a consumer do when the contract does not fit, and who reviews it? |

Prefer an API-first interface with a documented request/response shape. A portal or CLI may sit on top, but the interface must remain automatable and independently testable. Keep generated infrastructure and policy decisions in version control where practical.

## Adoption and value review

Review at a fixed interval using a small evidence set: active and repeat users, completion rate, time-to-first-use, lead time through the target workflow, support tickets or handoffs removed, failure/rework rate, incident recovery effect, developer sentiment/cognitive-load signal, security-policy coverage, and platform cost per consumer or workflow. Compare actuals to the original baseline and value model; revise the roadmap when the evidence disagrees.

Treat adoption as a product problem. Sample non-users, observe where they leave the path, fix the highest-friction step, and measure again. Do not force adoption by removing the escape hatch or by converting unresolved usability problems into policy violations.

## Domain and event boundaries

Keep platform domains independently accountable for their APIs and backlogs. Use events for cross-cutting adapters such as issue tracking, CI hooks, observability, configuration inventory, and audit collection. Define event schema, producer/consumer ownership, delivery semantics, replay behavior, and failure visibility before adding an adapter. Tool-specific cluster, IaC, proxy, or telemetry operations belong to their named tool skills.

# Policy and observability ownership

Use this reference when a platform must preserve developer control while meeting security, compliance, reliability, or audit requirements.

## Developer-owned delivery with guardrails

Keep the application team in control of its build and deployment workflow. Publish reusable checks, starter pipelines, and policy interfaces that teams can call from their own pipelines. Enforce requirements at the point where a change is admitted or completed, rather than inserting a centralized manual queue for every deployment.

For each check, document the input, policy version, decision, evidence location, failure response, exception route, and owner. A check may run before a change or immediately after it, but it must be deterministic enough to explain why a decision was made. Policy-as-code engines and admission controllers are implementation choices; this methodology owns the contract and evidence flow.

## Exception and escape-hatch workflow

1. Consumer states the requested deviation, affected resources, duration, reason, and compensating controls.
2. Platform owner checks scope, authorization, blast radius, expiry, and whether the golden path or policy should be improved instead.
3. Approver records the decision and accountable owner. High-risk exceptions need the relevant security, compliance, or service owner.
4. Automation applies the narrowest permitted scope and emits an auditable decision record.
5. Owner reviews the exception before expiry, closes it, renews it with fresh evidence, or turns the repeated exception into a product backlog item.

An exception is not a permanent bypass. Missing evidence, an expired exception, or an unowned violation is a release or access blocker according to the consuming release/security policy; this skill does not define that policy.

## Observability as a platform service

The platform should provide collection, default dashboards, published SLIs/SLOs, and a clear response route. For every signal, identify:

| Concern | Ownership to record |
|---|---|
| Instrumentation and collection | Service team supplies meaningful signals; platform supplies the supported path and availability of collection |
| Storage, retention, and access | Platform/telemetry owner, including tenancy and cost controls |
| SLI/SLO definition | Service owner proposes the indicator and target; platform helps publish and validate it |
| Alert interpretation and response | The service owner handles service symptoms; platform handles platform symptoms and routes ambiguous cases |
| Adoption and quality | Platform product owner measures coverage, usefulness, query/alert friction, and feedback |

Require a response matrix for each published SLO: symptom, query or alert, severity, recipient, first diagnostic action, escalation, and evidence retained. Review false positives, missing signals, and unused dashboards with consumers. The service-facing observability contract remains the technical baseline; this workflow adds product adoption and response ownership.

## Review gate

Before calling the platform capability ready, verify that the contract, policy decision evidence, exception record, SLO/response matrix, owners, and adoption measurement are present. If any owner or failure route is unknown, record the gap and stop at the appropriate readiness decision.

# Release Engineering for Infrastructure Artifacts

Release engineering for infrastructure artifacts differs from open source releases. Infrastructure artifacts — container images, Helm charts, Terraform modules, Compose stacks — have different publication patterns, versioning strategies, and verification requirements.

## Artifact Types and Publication Targets

| Artifact Type | Publication Target | Versioning | Verification |
|--------------|-------------------|------------|--------------|
| Container image | Container registry (GHCR, Docker Hub, ECR) | Git SHA + semver tags | Image scan, size check, smoke test |
| Helm chart | OCI registry, chart repo | Chart.yaml version, semver | `helm template` dry-run, install test |
| Terraform module | Git tag, registry | Semver tag | `terraform validate`, `tflint` |
| Docker Compose stack | Git tag | Semver or date tag | `docker compose config`, smoke test |
| Internal package | Private registry (PyPI, npm) | Semver + pre-release label | Build, install, import test |

## Container Image Lifecycle

```
Build → Tag → Scan → Sign → Push → Verify → Deploy
```

| Step | Tooling | Success Criteria |
|------|---------|-----------------|
| Build | `docker build`, `buildkit`, `ko` | Exit code 0, image ID captured |
| Tag | `docker tag` with SHA + semver + "latest" | Tags applied to build image |
| Scan | Trivy, Grype, Snyk | No critical/high CVEs, or exceptions documented |
| Sign | cosign | Signature attached to image |
| Push | `docker push`, `crane` | Registry confirms digest |
| Verify | `cosign verify`, digest comparison | Signature valid, digest matches |
| Deploy | Helm upgrade, kubectl apply, compose up | Pods healthy, endpoint responds |

## Versioning Strategies

| Strategy | When to Use | Example |
|----------|-------------|---------|
| Git SHA only | Development, CI-only artifacts | `sha-a1b2c3d` |
| Semver tags | Public-facing releases, API consumers | `v1.2.3` |
| Date-based | Internal rollups without consumer compatibility | `2026-06-01` |
| Semver + pre-release | Release candidates, staging deployments | `v1.2.3-rc.1` |
| Git SHA + semver | Production — both for traceability and semantics | `v1.2.3-sha-a1b2c3d` |

## Release Gate Checklist

Before promoting any artifact to production:

- [ ] All tests pass (unit, integration, E2E)
- [ ] Security scan passes with no unexcepted critical/high CVEs
- [ ] Image is signed (for containerized deployments)
- [ ] Changelog entry exists for the release
- [ ] Rollback plan documented (previous version, restore command)
- [ ] Smoke test passes against staging
- [ ] Release notes drafted for communication

# Secret Management — Reference

## HashiCorp Vault

- **Core model:** Dynamic secrets — generated on-demand (database credentials, cloud access keys, PKI certificates), short TTL, automatic revocation
- **Auth methods:** Token, Kubernetes (service account JWT → vault token), LDAP, OIDC, AppRole (machine-to-machine), AWS/GCP/Azure, GitHub
- **Secret engines:** KV (static, versioned), Database (dynamic DB creds), AWS/GCP/Azure (dynamic cloud creds), PKI (leaf certs), Transit (encryption as a service, data never leaves client)
- **Policies:** Path-based access control (`path "secret/data/app/*" { capabilities = ["read", "list"] }`), templating (`{{identity.entity.name}}`), fine-grained CRUD + deny + sudo
- **Patterns:** Sidecar injector (auto-auth, auto-renew), Vault Agent for caching/templating, kubernetes secrets via CSI provider, Terraform Vault provider, ACL templating for per-app secrets

## SOPS / age

- **Core model:** Encrypted files in Git — `sops --encrypt` (age or PGP), `sops --decrypt`, encrypted file is valid YAML/JSON with encrypted fields as `ENC[AES256_GCM,...]`
- **Encryption backends:** age (modern, key-based), PGP (traditional), AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault
- **Workflow:** `.sops.yaml` config — creation rules per file path, key list for multi-key encryption (dev team key + CI key). Secrets files committed alongside code, CI decrypts at deploy time
- **CI/CD integration:** `sops --decrypt` in pipeline (using CI system's key access), age key securely injected (not in repo), KMS-based for cloud-native CI
- **Limitations:** No secret rotation (re-encrypt manually), no access audit, suitable for static config secrets but not dynamic credentials

## External Secrets Operator (Kubernetes)

- **Core model:** CRD-based — ExternalSecret resource syncs from external API to Kubernetes Secret. One-time sync or polling
- **Backends:** AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, Akeyless, GitLab, SOPS-encrypted files
- **Patterns:** `refreshInterval` for periodic sync, `target` to control created secret name/type, `data` for static key mapping, `dataFrom` for bulk (all keys from remote), `remoteRef` strategies (property, version)
- **Best practices:** Namespace-scoped `ClusterSecretStore` vs `SecretStore`, push secret reconciliation errors to monitoring, avoid over-polling (set realistic `refreshInterval`)

## Sealed Secrets

- **Core model:** Encrypt secrets client-side — `SealedSecret` CRD (controller decrypts, creates regular Secret in cluster)
- **Workflow:** Developer creates `SealedSecret` YAML with `kubeseal` using cluster's public cert. Committed to Git. Controller on cluster decrypts and materializes `Secret` when applied.
- **Best for:** GitOps workflows where secrets must be in Git but cannot be stored in plaintext
- **Limitations:** Static only (no rotation), cert management (backup cluster key), no external API integration, per-cluster certs (same sealed secret won't work across clusters)

# Service Networking — Reference

## Reverse Proxy

### Traefik

- **Core model:** Dynamic routing — auto-discovers services via providers (Docker, Kubernetes, Consul, file), hot-reloads config
- **Key concepts:** Routers (HTTP/HTTPS/TCP/UDP), middlewares (rate limiting, auth, headers, retries, circuit breakers, compression, redirect), services (load balancing), entrypoints (ports), TLS automation via Let's Encrypt
- **Providers:** Docker provider (labels on containers), Kubernetes provider (IngressRoute CRD, or standard Ingress), file provider (static/dynamic YAML/TOML), Consul, etcd, Redis, ZooKeeper
- **Middleware chains:** Order matters — rate limit → auth → headers → retry; custom middleware via plugins (WebAssembly, Go), pass through ForwardAuth to external services
- **Observability:** Metrics (Prometheus, OpenTelemetry, Datadog, InfluxDB), access logs (in JSON), tracing (Jaeger, Zipkin, OpenTelemetry), dashboard UI

### nginx

- **Core model:** Static config (reload on change), high-concurrency event loop, reverse proxy, load balancer, TLS termination, caching
- **Key patterns:** `upstream` blocks for load balancing, `proxy_pass` for forwarding, `location` blocks for path routing, `map` for conditional logic, `limit_req`/`limit_conn` for rate limiting
- **Config management:** Templating (Jinja, envsubst), include directories for modular config, nginx -t for validation, reload via SIGHUP

### Caddy

- **Core model:** Automatic HTTPS (ZeroSSL/Lets Encrypt by default), simple `Caddyfile` syntax, JSON API for dynamic config
- **Key features:** HTTP/3 (QUIC) by default, on-demand TLS, HTTP->HTTPS redirects, reverse proxy with health checking, match blocks, matchers
- **Best for:** Simple deployments where Traefik's dynamic service discovery is overkill; excellent developer experience

## Mesh Networking

### Tailscale / Headscale

- **Core model:** WireGuard-based overlay network — nodes get unique Tailscale IP, communicate directly (NAT traversal), ACLs control access
- **ACL policy (huJSON):** `acls` (src/dest/proto/port), `groups` (user groupings), `tags` (device identity), `hosts` (alias mapping), `derpMap` relay configuration, `ssh` for Tailscale SSH
- **Key features:** Subnet routing (advertise routes), exit nodes (traffic to internet), ACL deny rules (refuse trailing), Funnel (allow internet traffic to local), Serve (host services on tailnet), MagicDNS
- **Headscale specifics:** Self-hosted control server, open-source, PostgreSQL/ SQLite backend, OIDC integration, CLI (`headscale users`, `headscale nodes`, `headscale routes`), DERP relay server setup
- **Lifecycle:** Node registration (pre-auth keys, web auth), expiry/node cleanup, key rotation (node keys, auth keys), multi-tailnet federation (Headscale sharing)

### WireGuard

- **Core model:** Layer 3 secure tunnel — single UDP port, kernel-level (fast), peer-to-peer, pre-shared or public-key auth
- **Config basics:** Interface (private key, address, listen port, DNS), Peer (public key, allowed IPs, endpoint, persistent keepalive)
- **Topology patterns:** Point-to-point (simple site-to-site), hub-and-spoke (central node routes), mesh (direct peer-to-peer), routed subnet (wg-quick tables, policy routing)

## Service Mesh

### Istio

- **Core model:** Sidecar proxy (Envoy) injected into pods — intercepts all traffic, applies mesh policies
- **Key concepts:** VirtualService (traffic routing, retries, timeouts, mirroring), DestinationRule (load balancing, connection pool, mTLS, circuit breaker), Gateway (ingress/egress), ServiceEntry (external services), PeerAuthentication (mTLS mode), AuthorizationPolicy (RBAC for services)
- **Observability:** Telemetry via Envoy — HTTP/gRPC metrics (Prometheus), distributed tracing (Jaeger/Zipkin/OpenTelemetry), access logs, Kiali for topology visualization

### Cilium

- **Core model:** eBPF-based — no sidecar injection, kernel-level networking and security
- **Key capabilities:** NetworkPolicy (identity-based, FQDN-based), Service Mesh (L7 policies, ingress/gateway API, L7 load balancing), Encryption (WireGuard in-kernel), Observability (Hubble: flow logs, metrics, UI, OpenTelemetry), ClusterMesh (multi-cluster networking)
- **Advantage over Istio:** No sidecar overhead, native eBPF performance, integrated with Tetragon for runtime security

---
title: "Golden Path / Self-Service Portal: [Capability Name]"
doc_id: GP-[CAPABILITY-CODE]-[VERSION]
status: draft | proposed | approved | superseded
created: [YYYY-MM-DD]
last_modified: [YYYY-MM-DD]
owner: "[Platform Team / Individual]"
approver: "[Platform Lead]"
---

# Golden Path / Self-Service Portal — [Capability Name]

## 1. Purpose and Scope

| Field | Value |
|---|---|
| **Capability** | [What developers can obtain, e.g., a new service with CI pipeline, namespace, and database] |
| **Developer need** | [The workflow this removes from a ticket queue, e.g., provision a Postgres database for a new microservice] |
| **In scope** | [List what the portal provisions automatically] |
| **Out of scope** | [List what still requires a ticket or manual review, e.g., production firewall changes] |
| **Request frequency** | [e.g., 12 requests/week — evidence that this is the highest-friction path] |
| **Current cycle time** | [e.g., 3 days from ticket to working environment] |

## 2. Developer Journey

| Step | Actor | Action | System Response | Time |
|---|---|---|---|---|
| 1 | [Developer] | [Submit request with service name, team, environment] | [Validate naming and quota] | _[fill: seconds]_ |
| 2 | [System] | [Run scaffold from template] | [Create repo, pipeline, namespace, DB via IaC] | _[fill: minutes]_ |
| 3 | [Developer] | [Approve generated PR] | [Apply to Git, reconcile via GitOps] | _[fill: minutes]_ |
| 4 | [Developer] | [First deploy] | [Verify observability baseline is live] | _[fill: minutes]_ |

- **Time-to-first-deploy target:** _[fill: e.g., under 30 minutes from request]_
- **Cognitive load target:** _[fill: e.g., no more than N decisions required from the developer]_

## 3. Template Design

### 3.1 Provisioning Template

- **IaC module used:** _[fill: e.g., terraform module for service scaffolding, version pinned]_
- **Resources created:** _[fill: repository, CI workflow, namespace, database, secrets placeholder, dashboards]_
- **Input parameters:** _[fill: name, team, environment, size limits — every input validated]_
- **Default values:** _[fill: what the template assumes when the developer leaves a field blank]_

### 3.2 Pipeline Template

- **Stages:** _[fill: build, test, artifact, deploy — mirror the platform CI/CD reference]_
- **Gates:** _[fill: where approvals sit and who can override]_
- **Artifact handling:** _[fill: registry, signing, provenance, versioning scheme]_

## 4. Guardrails and Policies

| Guardrail | Enforcement Mechanism | Escalation / Override |
|---|---|---|
| Least-privilege IAM | _[fill: generated from request scope, not admin defaults]_ | _[fill: role/person with authority]_ |
| Budget and quota limits | _[fill: tag-based budget alert, quota per namespace]_ | _[fill: cost owner approval]_ |
| Observability baseline | _[fill: mandatory dashboard + alert rules on scaffold]_ | _[fill: SRE review]_ |
| Security baseline | _[fill: secret scanning, image scanning, network policy default deny]_ | _[fill: security review]_ |
| Naming and ownership | _[fill: validated naming convention, required owner field]_ | _[fill: platform team]_ |

- **Policy-as-code location:** _[fill: where policies live in Git, e.g., OPA/kyverno rules, Terraform guardrail module]_

## 5. Escape Hatch

- **Escape hatch path:** _[fill: what a developer does when the golden path does not fit — e.g., exception request, custom module review]_
- **Exception review criteria:** _[fill: what justifies leaving the paved road and who reviews]_
- **Bounded by:** _[fill: golden paths are paved roads, not cages — the exception keeps the platform from blocking delivery]_

## 6. API-First Design

- **Portal entry points:** _[fill: CLI command, web UI, API endpoint — each invokes the same scaffold service]_
- **Request/response contract:** _[fill: schema of the request and the status response]_
- **Audit trail:** _[fill: every provisioned change is a Git commit/PR with actor and timestamp]_
- **Idempotency:** _[fill: what happens when the same request is submitted twice]_

## 7. Success Metrics

| Metric | Target | Measurement Source |
|---|---|---|
| Time-to-first-deploy | _[fill: target]_ | _[fill: portal telemetry]_ |
| Ticket volume for this capability | _[fill: target decrease]_ | _[fill: ticketing system]_ |
| Developer satisfaction / cognitive load | _[fill: survey score]_ | _[fill: survey]_ |
| Guardrail violations | _[fill: target]_ | _[fill: policy engine logs]_ |

## 8. Version History

| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | [YYYY-MM-DD] | [Author] | Initial golden path design |
| 1.1 | [YYYY-MM-DD] | [Author] | [Summary of changes] |

*Keep this record in Git next to the portal implementation so the design and the code stay in sync.*

---
title: "Infrastructure as Code Review Record: [Module / Project Name]"
doc_id: IACR-[MODULE-CODE]-[VERSION]
status: draft | in-review | approved | changes-requested
created: [YYYY-MM-DD]
last_modified: [YYYY-MM-DD]
reviewer: "[Reviewer Name]"
author: "[Module Author Name]"
---

# Infrastructure as Code Review Record — [Module / Project Name]

## 1. Review Metadata

| Field | Value |
|---|---|
| **Module / project** | [Name and path in Git] |
| **IaC tooling** | [e.g., Terraform, OpenTofu, Pulumi, Ansible, CloudFormation] |
| **Provider(s)** | [e.g., AWS, GCP, Azure, on-prem] |
| **Review scope** | [Full module / resource block / state change] |
| **Plan applied?** | [Yes/No — if yes, plan ID and date] |
| **Drift baseline** | [State of the environment before the change] |

## 2. Module Structure

| Check | Verdict | Notes |
|---|---|---|
| Single-purpose composition (no monolith root) | _[fill: pass / fail / n/a]_ | _[fill: what should be split into child modules]_ |
| Variables and defaults parameterize env specifics | _[fill: pass / fail / n/a]_ | _[fill: hardcoded IDs, account numbers, regions]_ |
| `for_each`/`count` used instead of copy-pasted blocks | _[fill: pass / fail / n/a]_ | _[fill: specific resources to convert]_ |
| Minimal output surface area | _[fill: pass / fail / n/a]_ | _[fill: outputs consumers actually need]_ |
| Version pinning of modules and providers | _[fill: pass / fail / n/a]_ | _[fill: constraints and locked versions]_ |

## 3. State and Drift

| Check | Verdict | Notes |
|---|---|---|
| Remote backend with locking configured | _[fill: pass / fail / n/a]_ | _[fill: backend type and lock mechanism]_ |
| No secrets material in state | _[fill: pass / fail / n/a]_ | _[fill: which attributes are sensitive and how they are handled]_ |
| Workspaces/environments isolated | _[fill: pass / fail / n/a]_ | _[fill: env separation approach]_ |
| Drift detection cadence defined | _[fill: pass / fail / n/a]_ | _[fill: scheduled plan or drift tooling]_ |
| State operations documented (`state mv`, `rm`, imports) | _[fill: pass / fail / n/a]_ | _[fill: any state surgery required]_ |

## 4. Security and Secrets

| Check | Verdict | Notes |
|---|---|---|
| Secrets come from a secret manager, not plaintext vars | _[fill: pass / fail / n/a]_ | _[fill: Vault / SOPS / cloud secret store reference]_ |
| Least-privilege IAM on created resources | _[fill: pass / fail / n/a]_ | _[fill: overly broad policies to tighten]_ |
| Network boundaries default to deny | _[fill: pass / fail / n/a]_ | _[fill: security groups, firewalls, network policies]_ |
| Sensitive outputs marked `sensitive = true` | _[fill: pass / fail / n/a]_ | _[fill: which outputs]_ |
| Resource naming and tagging consistent | _[fill: pass / fail / n/a]_ | _[fill: tag keys, cost center, owner, environment]_ |

## 5. Operational Readiness

| Check | Verdict | Notes |
|---|---|---|
| `plan` output reviewed for unintended changes | _[fill: pass / fail / n/a]_ | _[fill: resources that will be replaced vs updated]_ |
| `prevent_destroy` on irreplaceable resources | _[fill: pass / fail / n/a]_ | _[fill: database, state bucket, registry]_ |
| Lifecycle rules match intent (`create_before_destroy`) | _[fill: pass / fail / n/a]_ | _[fill: where ordering matters]_ |
| Rollback path defined | _[fill: pass / fail / n/a]_ | _[fill: revert commit, previous state, or forward fix]_ |

## 6. Findings

### Blocking Findings

| # | Severity | Finding | Location | Suggested Fix | Owner | Fixed? |
|---|---|---|---|---|---|---|
| 1 | [critical/high] | _[fill: what is wrong and why it blocks]_ | _[fill: file:line]_ | _[fill: concrete change]_ | _[fill: name]_ | _[fill: yes/no]_ |

### Non-Blocking Findings

| # | Severity | Finding | Location | Suggested Fix | Owner | Fixed? |
|---|---|---|---|---|---|---|
| 1 | [low/medium] | _[fill: what is suboptimal]_ | _[fill: file:line]_ | _[fill: concrete change]_ | _[fill: name]_ | _[fill: yes/no]_ |

## 7. Verdict

| Field | Value |
|---|---|
| **Verdict** | [approved / changes-requested] |
| **Blocking findings resolved** | [all / list of remaining] |
| **Re-review required** | [yes/no — and by when] |
| **Reviewer sign-off** | [Name, date] |
| **Author sign-off** | [Name, date] |

*File this record alongside the module and the applied plan output so the review is auditable.*

---
title: "Observability Contract: [Service Name]"
doc_id: OBC-[SERVICE-CODE]-[VERSION]
status: draft | reviewed | approved | superseded
created: [YYYY-MM-DD]
last_modified: [YYYY-MM-DD]
owner: "[Service Owner / Team]"
approver: "[SRE / Platform Lead]"
---

# Observability Contract — [Service Name]

## 1. Service Context

| Field | Value |
|---|---|
| **Service Name** | [Service Name] |
| **Owner** | [Team / Individual] |
| **Environments** | [dev, staging, prod] |
| **Dependencies** | [Upstream/downstream services, data stores] |
| **SLO reference** | [Link to SLO declaration or error budget policy] |

## 2. Signals Required

Every service must emit all three signals before production traffic is accepted.

### 2.1 Metrics

| Metric | Type | Name | Definition |
|---|---|---|---|
| Request rate | Counter | _[fill: e.g., svc_http_requests_total]_ | _[fill: label set, status split]_ |
| Error rate | Counter | _[fill: e.g., svc_http_errors_total]_ | _[fill: which statuses count as errors]_ |
| Latency | Histogram | _[fill: e.g., svc_http_request_duration_seconds]_ | _[fill: buckets, percentiles consumed]_ |
| Saturation | Gauge | _[fill: e.g., svc_queue_depth]_ | _[fill: what resource is near exhaustion]_ |

- **Scrape endpoint:** _[fill: e.g., /metrics on :9090]_ — must be reachable by the platform scraper.

### 2.2 Logs

- **Format:** _[fill: structured JSON with timestamp, level, service, trace_id, span_id]_
- **Shipping:** _[fill: agent/target — e.g., Promtail/Alloy/Fluent Bit]_
- **Retention requirement:** _[fill: hot/warm/cold tiers and durations]_
- **Sensitive data:** _[fill: what must never be logged — tokens, PII, full payloads]_

### 2.3 Traces

- **Instrumentation:** _[fill: OpenTelemetry SDK, auto-instrumentation, or manual spans]_
- **Context propagation:** _[fill: W3C TraceContext across all outbound calls]_
- **Sampling:** _[fill: head/tail sampling strategy and rate]_
- **Key spans:** _[fill: entry, external calls, DB queries, background jobs]_

## 3. Dashboards and Recording Rules

| Artifact | Name / Path in Git | Content |
|---|---|---|
| Service dashboard | _[fill: provisioning path]_ | _[fill: RED panels, per row: traffic, errors, latency, saturation]_ |
| Recording rules | _[fill: rules file path]_ | _[fill: rate/error-duration derivations, error budget expressions]_ |
| Dashboard links | _[fill: links to related platform dashboards]_ | _[fill: cross-service dependency view]_ |

- **Dashboard-as-code requirement:** _[fill: dashboards live in Git and change via review, not ad-hoc UI edits]_

## 4. Alerting and Error Budgets

| Alert | Condition (query) | Severity | Routing | Action |
|---|---|---|---|---|
| _[fill: High error rate]_ | _[fill: PromQL expression]_ | _[fill: critical/warning]_ | _[fill: page/Slack]_ | _[fill: incident response, freeze, rollback]_ |
| _[fill: Latency p99 breach]_ | _[fill: PromQL expression]_ | _[fill: severity]_ | _[fill: routing]_ | _[fill: action]_ |
| _[fill: Budget burn rate]_ | _[fill: multi-window burn rate expression]_ | _[fill: severity]_ | _[fill: routing]_ | _[fill: action]_ |

- **Error budget policy applied:** _[fill: link or reference to the team error budget policy]_
- **Noise control:** _[fill: `for:` durations, deduplication, silenced maintenance windows]_

## 5. Release and Verification Gate

| Gate | Requirement |
|---|---|
| Pre-release | _[fill: dashboards live, alerts firing correctly, metrics scraping, traces flowing]_ |
| Canary verification | _[fill: what SLIs are compared between canary and control and at what divergence]_ |
| Post-release | _[fill: regression check against baseline within N minutes, on-call notified]_ |

- **Verification evidence:** _[fill: where the evidence (dashboards, alert receipts, trace samples) is recorded]_

## 6. Ownership and Review

| Item | Value |
|---|---|
| **Observability owner** | [Team / Individual] |
| **Review cadence** | [Quarterly or on architecture change] |
| **Next review date** | [YYYY-MM-DD] |

### Sign-off

| Role | Name | Date |
|---|---|---|
| Service Owner | [Name] | [YYYY-MM-DD] |
| SRE / Platform Lead | [Name] | [YYYY-MM-DD] |

*This contract is part of the service's production readiness review and lives in Git next to the dashboards and rules it describes.*

# Platform exception record

- Requestor and consumer:
- Platform capability/policy:
- Requested deviation:
- Resources and environments in scope:
- Reason and duration:
- Risk/blast-radius assessment:
- Compensating controls:
- Evidence or test results:
- Accountable owner:
- Required approver(s):
- Decision: approved / rejected / needs evidence
- Narrow scope applied:
- Decision record or policy version:
- Expiry and review date:
- Closure, renewal, or product backlog follow-up:

# Platform value record

## Opportunity

- Strategic software outcome:
- Internal users and early adopters:
- Workflow/grain in scope:
- Technical product owner:
- Platform domain owners:

## Baseline evidence

| Signal | Source and measurement window | Baseline | Confidence/limitations |
|---|---|---:|---|
| Queue or lead time |  |  |  |
| Manual steps/rework |  |  |  |
| Failure or recovery effect |  |  |  |
| Cognitive load/sentiment |  |  |  |
| Cost |  |  |  |

## First slice and contract

- Minimum valuable capability:
- Consumer request and outcome:
- API/portal/CLI interface:
- Authorization, quotas, and idempotency:
- Evidence and SLO:
- Escape hatch and reviewer:

## Hypothesis and decision

- Expected value mechanism (throughput, cost, risk, or experience):
- Adoption target and measurement window:
- Success threshold:
- Stop, reshape, or escalate threshold:
- Review date:

## Follow-up

- Actual results versus baseline:
- User feedback and non-user friction:
- Decision and rationale:
- Roadmap or backlog changes: