# Evidence Path and Tooling

## Evidence path

1. Prefer live cluster evidence when a Kubernetes MCP server, `kubectl`, or the `argocd` CLI is available against the Argo CD control-plane cluster.
2. Fall back to official Argo CD documentation (argo-cd.readthedocs.io) and the upstream argo-cd GitHub repository when live inspection is unavailable.
3. Ask only for sanitized `Application` / `AppProject` / `ApplicationSet` YAML, the redacted `argocd-cm` and `argocd-rbac-cm` ConfigMaps, and `argocd app history` output when current-state proof matters.
4. Label conclusions as `live evidence`, `documentation-based`, `sanitized user evidence`, or `inference`.

## Useful live-evidence commands

```shell
# All Applications, AppProjects, and ApplicationSets in the argocd namespace
kubectl -n argocd get applications,appprojects,applicationsets -o yaml

# Detailed Application status (sync, health, lastSyncRevision)
kubectl -n argocd get application <app-name> -o yaml
argocd app get <app-name>
argocd app history <app-name>

# Argo CD configuration (the global config knobs)
kubectl -n argocd get configmap argocd-cm -o yaml
kubectl -n argocd get configmap argocd-rbac-cm -o yaml
kubectl -n argocd get configmap argocd-cmd-params-cm -o yaml

# RBAC effective policy
argocd account list
argocd account get-user-info <user>

# Cluster registrations (every destination cluster has its own Secret)
kubectl -n argocd get secrets -l argocd.argoproj.io/secret-type=cluster -o yaml

# Sync windows on an AppProject
kubectl -n argocd get appproject <project> -o jsonpath='{.spec.syncWindows}'

# Argo CD Agent (hub-and-spoke deployments)
kubectl -n argocd get agents -o yaml
```

## Argo CD install state to confirm before review

- Argo CD version (`kubectl -n argocd get deploy argocd-server -o jsonpath='{.spec.template.spec.containers[0].image}'`) — sync impersonation, RBAC granular actions, and ApplicationSet RollingSync arrived in different versions.
- `application.sync.impersonation.enabled` in `argocd-cm` — `false` (default) means every sync runs as the controller's ServiceAccount on every destination.
- `application.sync.requireOverridePrivilegeForRevisionSync` in `argocd-cm` — `true` requires explicit override permission for ad-hoc revision syncs.
- `webhook.maxPayloadSizeMB` in `argocd-cm` — large Helm value files may exceed the default.
- Whether Argo CD Agent (argocd-agent) is in use for hub-and-spoke multi-cluster — different security model.
- Whether Argo CD Autopilot manages Argo CD itself via GitOps — change review must include the Autopilot repo.

## Sanitization rules

- Never request kubeconfig contents, cluster Secret contents, repository SSH keys, or webhook signing secrets in chat.
- Replace identifiable cluster URLs and namespaces with placeholders unless the user provides them and confirms it is safe to use them.
- Do not print Git repository tokens, OCI registry tokens, or Helm OCI credentials.

# Official Sources

Load these only when needed:

- [Argo CD documentation home](https://argo-cd.readthedocs.io/en/stable/) — use as the entry point for any Argo CD authoring, install, or operator-side question.
- [Declarative setup](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/) — use for `Application`, `AppProject`, cluster Secret, repository Secret, and ConfigMap structure.
- [argocd-cm reference](https://argo-cd.readthedocs.io/en/stable/operator-manual/argocd-cm-yaml/) — use for global controller knobs including `application.sync.impersonation.enabled`, `application.sync.requireOverridePrivilegeForRevisionSync`, and `webhook.maxPayloadSizeMB`.
- [Auto-sync](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/) — use for `automated`, `prune`, `selfHeal` semantics and operational guidance.
- [Sync Options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/) — use for `Replace`, `Force`, `ServerSideApply`, `PruneLast`, `CreateNamespace`, `Validate=false`, `RespectIgnoreDifferences`.
- [Sync Windows](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) — use for deploy-freeze enforcement at the AppProject level.
- [ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) — use for `list`, `cluster`, `git`, `matrix`, `merge`, `pullRequest`, `scmProvider` generator semantics.
- [ApplicationSet Progressive Syncs (RollingSync)](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Progressive-Syncs/) — use for staged ApplicationSet rollouts.
- [Argo CD RBAC](https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/) — use for `policy.csv` syntax, default role, group bindings, and granular action permissions.
- [Sync impersonation proposal](https://argo-cd.readthedocs.io/en/stable/proposals/decouple-application-sync-user-using-impersonation/) — use for the AppProject `destinationServiceAccounts` field and the least-privilege sync identity model.
- [Argo CD upgrading guide](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/) — use when version-specific RBAC actions or API fields matter.
- [Argo CD User Management](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/) — use for SSO via OIDC / SAML / Dex and group claims.
- [Argo CD Webhook](https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/) — use for repository webhook setup and signature verification.
- [Argo CD Agent](https://github.com/argoproj-labs/argocd-agent) — use for hub-and-spoke multi-cluster topologies replacing direct cluster registrations.
- [Argo CD Autopilot](https://github.com/argoproj-labs/argocd-autopilot) — use when Argo CD itself is managed via GitOps.

## Grounded insights worth carrying into the skill

- The Argo CD controller defaults to running as cluster-admin on every destination cluster. The `application.sync.impersonation.enabled` flag in `argocd-cm` is the switch that activates per-Application ServiceAccount impersonation via `destinationServiceAccounts` on the AppProject.
- `AppProject` boundaries are the only enforced isolation between teams sharing one Argo CD instance. Wildcards in `sourceRepos`, `destinations`, `clusterResourceWhitelist`, or empty `namespaceResourceBlacklist` collapse the boundary.
- `automated.selfHeal: true` combined with `automated.prune: true` means a Git revert (or Git outage that exposes a stale ref) deletes prod resources. There is no built-in confirmation step.
- ApplicationSet's `cluster` generator with an empty selector auto-onboards every newly registered cluster. This is the most-cited blast-radius mode in Argo CD post-incident reviews.
- ApplicationSet RollingSync intentionally forces auto-sync **disabled** on generated Applications (the controller logs warnings if any have auto-sync enabled). RollingSync drives sync via OutOfSync detection, not auto-sync.
- The `Replace=true` sync option is destructive on `StatefulSet`, `Service`, `PersistentVolumeClaim`, and any resource with finalizers. Argo CD's default three-way merge (or server-side apply on newer versions) is safer.
- Argo CD RBAC granular actions (e.g., `action/apps/Deployment/restart`, `action/argoproj.io/Rollout/abort`) shipped in v2.8+. Older policies that don't list these still work but won't grant the action — operators may discover gaps after upgrade.
- The `requireOverridePrivilegeForRevisionSync: true` flag in `argocd-cm` requires explicit `override` permission to sync to a non-tracked revision (e.g., a branch instead of HEAD of the configured target). This blocks easy ad-hoc syncs that bypass Git review.
- Argo CD Autopilot's bootstrap repo manages Argo CD itself — changes to that repo can disable RBAC, weaken AppProject scopes, or rotate the admin password. Treat the Autopilot repo as a tier-0 control surface.
- The Argo CD Agent (argocd-agent) introduces a hub-and-spoke topology where the central Argo CD installation does not hold cluster credentials for spoke clusters; agents connect outbound. Different threat model from the classic direct-cluster registration.

# Workflow and Output Contract

## Workflow

### Step 1 — Identify the target and the surrounding AppProject

1. Confirm the kind: `Application`, `AppProject`, `ApplicationSet`, or a global ConfigMap (`argocd-cm`, `argocd-rbac-cm`).
2. For an `Application`, locate the `spec.project` reference and review the `AppProject` first — the AppProject defines the boundary the Application operates within.
3. For an `AppProject`, list every `Application` referencing it (`kubectl -n argocd get applications -o jsonpath='{range .items[?(@.spec.project=="<project>")]}{.metadata.name}{"\n"}{end}'`).
4. For an `ApplicationSet`, identify the generator type (`list`, `cluster`, `git`, `matrix`, `merge`, `pullRequest`, `scmProvider`) and the `spec.template`.

### Step 2 — Audit the AppProject blast radius

The AppProject defines four boundary surfaces. Each is a potential blast-radius finding:

1. **`sourceRepos`** — the Git or Helm repos this project may pull from. `['*']` means any repo. Recommended: explicit list.
2. **`destinations`** — the (cluster, namespace) tuples this project may deploy to. `[{server: '*', namespace: '*'}]` means anywhere. Recommended: explicit cluster URLs and namespace allowlist (or `namespace: 'team-*'` for multi-tenant patterns).
3. **`clusterResourceWhitelist`** — cluster-scoped resources this project may manage. Empty or `['*/*']` means any cluster-scoped resource (including ClusterRoleBindings, Namespaces). Recommended: empty for application projects; explicit list for platform projects.
4. **`namespaceResourceBlacklist`** — namespace-scoped resources this project may NOT manage. Recommended: include `[{group: 'rbac.authorization.k8s.io', kind: '*'}]` for application projects to prevent applications from binding their own RBAC.

Reference: [AppProject in declarative setup](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/).

### Step 3 — Audit sync identity (the most under-reviewed control)

Three sync identity modes exist; pick one and verify:

1. **Controller default** — Argo CD controller's ServiceAccount on the destination cluster. Default is broad (cluster-admin in many installs). **Critical finding** if production Applications use this without an audit trail of what the controller can do.
2. **Sync impersonation** (preferred for least privilege) — `application.sync.impersonation.enabled: true` in `argocd-cm` plus `destinationServiceAccounts` on the AppProject. Each Application syncs as a per-namespace ServiceAccount with scoped RBAC. See the [sync impersonation proposal](https://argo-cd.readthedocs.io/en/stable/proposals/decouple-application-sync-user-using-impersonation/).
3. **Cluster credentials** (legacy multi-cluster) — Argo CD has its own bearer token for each registered cluster. Rotate regularly.

Stress-tests:

- An Application with `spec.destination.namespace: kube-system` plus controller-default identity = sync runs as cluster-admin in kube-system.
- An AppProject with `destinationServiceAccounts` listing `defaultServiceAccount: 'default'` = effectively no impersonation; the default SA is always present.

### Step 4 — Audit the drift-handling posture

`spec.syncPolicy.automated` controls whether Argo CD reconciles drift. Three flags govern blast radius:

1. **`automated: {}` (auto-sync)** — every Git commit triggers a sync. Production-safe only with `syncWindows` and a tested CI gate.
2. **`automated.prune: true`** — resources removed from Git are deleted from the cluster. **Critical** without a rollback runbook: a misconfigured commit deletes prod resources.
3. **`automated.selfHeal: true`** — manual cluster changes are reverted on the next sync. Combined with `prune`, divergence becomes a hard reset to Git state.

Stress-tests:

- `automated.prune: true` on a `StatefulSet` Application = deletion cascades to PVCs (if `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete`). Data loss path.
- `automated.selfHeal: true` on an Application managing CRDs from a third-party operator = the operator's runtime status updates may be reverted as drift.
- `automated` with no `syncWindow` covering deploy-freeze periods = a freeze window can be bypassed by a Git commit.

Reference: [Auto-Sync](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/) and [Sync Windows](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/).

### Step 5 — Audit `syncOptions` for stateful or sensitive resources

`spec.syncPolicy.syncOptions` overrides default sync behavior. Flag these as findings:

- **`Replace=true`** — Argo CD deletes the resource and recreates it instead of patching. For `StatefulSet`, `PersistentVolume`, `PersistentVolumeClaim`, `Service` (ClusterIP rotation), `ConfigMap` consumed by hot-reload — this is data loss or downtime.
- **`Force=true`** — passes `--force` to `kubectl apply`. Disables conflict detection.
- **`ServerSideApply=false`** when Argo CD's default is server-side apply on newer versions — falls back to client-side three-way merge, which can re-introduce drift loops.
- **`PruneLast=true`** missing on Applications that delete resources — pruning happens before resource creation, briefly leaving the namespace in an unhealthy state.
- **`CreateNamespace=true`** with no namespace finalizer or RBAC scope — creates namespaces outside AppProject `destinations`.

### Step 6 — Audit `ApplicationSet` generators

ApplicationSet generators expand into multiple Applications. Risk surface depends on generator type:

- **`list` generator** — explicit list of clusters/parameters. Lowest risk.
- **`cluster` generator** — generates an Application for every registered cluster matching a label selector. **Critical** when the selector is empty (`{}`) or matches all clusters — a new cluster automatically receives the workload before review.
- **`git` generator** — generates an Application for every directory or file pattern in a Git repo. Risk: a malicious or accidental commit adds a new directory and triggers a new Application.
- **`matrix` and `merge` generators** — combine other generators. Risk multiplies.
- **`pullRequest` generator** — generates Applications for open PRs. Risk: any PR can trigger an ephemeral deployment with the PR's manifests.
- **`scmProvider` generator** — generates Applications for every repo in an org. Risk: org-wide auto-onboarding.

Reference: [ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) and [Progressive Syncs (RollingSync)](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Progressive-Syncs/).

Stress-tests:

- ApplicationSet with `cluster` generator + no selector + auto-sync = every cluster auto-onboarded in seconds.
- ApplicationSet with `pullRequest` generator + no namespace isolation = PRs deploy to shared namespaces.
- ApplicationSet with `goTemplate: true` and unsanitized template inputs = template injection if PR titles are templated into manifests.

### Step 7 — Audit `argocd-rbac-cm` policy

The Argo CD RBAC ConfigMap (`argocd-rbac-cm`) defines who can do what in the Argo CD UI/CLI/API. Check:

1. The default role (`policy.default`) — `role:readonly` is safe; `role:admin` is wrong.
2. Specific actions on resources — newer Argo CD versions ship granular actions like `action/apps/Deployment/restart` or `action/argoproj.io/Rollout/abort`. Each granted action should map to a real on-call runbook.
3. RBAC subject scopes — `g, <group>, role:admin` on broad SSO groups is a finding.

Reference: [Argo CD RBAC](https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/).

### Step 8 — Multi-cluster (Argo CD Agent) topology

If Argo CD Agent is in use:

- The control plane stores `Application` specs; each spoke runs an agent that pulls assigned Applications.
- Each agent has its own credentials and authentication path — verify rotation cadence.
- Network path from spoke to hub must be authenticated and encrypted.

Reference: [argocd-agent](https://github.com/argoproj-labs/argocd-agent).

## Output

Return:

- **target**: `Application`, `AppProject`, `ApplicationSet`, or RBAC ConfigMap, with the project boundary,
- **evidence level**: `live evidence` / `documentation-based` / `sanitized user evidence` / `inference`,
- **sync identity**: controller default vs impersonated SA vs cluster credential, with judgment on least privilege,
- **blast radius**: `sourceRepos`, `destinations`, `clusterResourceWhitelist`, `namespaceResourceBlacklist` audit,
- **drift posture**: `automated.prune`, `automated.selfHeal`, sync windows, syncOptions concerns,
- **risk findings** (with severity: high / medium / low) — covering sync identity, blast radius, drift, ApplicationSet generators, RBAC,
- **safest next actions** with sample manifest changes,
- **rollback plan**: how to revert auto-sync, disable selfHeal, narrow AppProject scope without breaking running Applications,
- **assumptions and missing facts**.

## Security notes

- Never recommend `automated.prune: true` + `automated.selfHeal: true` on production Applications without a tested rollback runbook.
- Never recommend `AppProject` with `sourceRepos: ['*']` and `destinations: ['*']` for application projects. Platform projects may need this; document the justification.
- Never recommend disabling sync impersonation as a default in production after it has been enabled.
- Never request or print Argo CD admin tokens, repo SSH keys, or destination cluster bearer tokens.