# Woodpecker CI Incident Checklist

## Scope

- Woodpecker version and image tags:
- Forge and forge URL:
- Server URL:
- Backend: Docker / Kubernetes / Local:
- Repository and pipeline event:
- First failing step and exact error:

## Evidence

- [ ] `docker compose config` or rendered deployment config captured
- [ ] Server logs captured around the first failure
- [ ] Agent logs captured around scheduling/execution
- [ ] Agent shows connected in the UI
- [ ] Repository webhook and OAuth callback checked
- [ ] Image pull and registry access tested
- [ ] Secret name/scope/event/plugin filter checked without exposing its value
- [ ] Failure reproduced with a minimal workflow where possible

## Boundary classification

- [ ] Forge/OAuth/webhook
- [ ] Server/database
- [ ] Agent registration/scheduling
- [ ] Backend/container/pod
- [ ] Checkout/network/credentials
- [ ] Workflow syntax/condition
- [ ] Application or test command

## Recovery notes

- Root cause:
- Smallest fix:
- Verification command:
- Remaining risk:

{
  "schema_version": 1,
  "skill_name": "woodpecker-ci",
  "evals": [
    {
      "id": "new-compose-deployment",
      "prompt": "Set up Woodpecker CI for our Forgejo instance using Docker Compose on a fresh host, with one agent.",
      "expected_output": "Scenario: new server/agent deployment. The agent loads woodpecker-ci and follows references/setup.md: copies templates/docker-compose.yml as compose.yaml, fills the .env from templates/forgejo.env.example (WOODPECKER_OPEN=false, shared WOODPECKER_AGENT_SECRET, OAuth client id/secret for the forge), validates with `docker compose config --quiet`, starts the stack, then verifies server health, agent connected state, and the forge connection with `docker compose ps`, logs, and scripts/woodpecker-doctor.py. SemVer image tags are pinned rather than latest.",
      "assertions": [
        "references/setup.md is consulted before writing configuration",
        "The agent secret is shared between server and agent and OPEN registration stays disabled",
        "Configuration is linted (docker compose config) before starting services",
        "Runtime health is verified after startup, including a woodpecker-doctor probe",
        "Explicit image tags are used instead of :latest"
      ]
    },
    {
      "id": "pipeline-yaml-authoring",
      "prompt": "Write a .woodpecker.yml that runs tests on pushes to main and on PRs, builds our container image only on main, and uses a Postgres service container for the test step.",
      "expected_output": "Scenario: workflow YAML authoring. The agent loads references/pipeline-syntax.md and produces a pipeline where the when conditions encode push+main and pull_request events correctly (event/branch combination understood: when lists are OR across entries, AND within an entry), defines the postgres service with proper image and environment variables, and does not treat depends_on as a readiness check — adding a wait or health handling before tests touch the database. The file is validated with `woodpecker-cli lint .woodpecker.yml` (and optionally exec locally), noting that server-managed secrets are unavailable in local exec.",
      "assertions": [
        "when conditions distinguish main-branch pushes from pull_request events",
        "A Postgres service container is defined for the test workflow",
        "depends_on is not relied on as a readiness check for the service",
        "The pipeline is linted before being committed"
      ]
    },
    {
      "id": "queued-pipeline-diagnosis",
      "prompt": "Our pipelines have been stuck in queued for 20 minutes after a push. Nothing shows up. Diagnose it.",
      "expected_output": "Scenario: incident troubleshooting. The agent runs the operating loop: probes connectivity first with scripts/woodpecker-doctor.py against the server URL, checks whether agents are connected and their labels/backend match what the repository's workflows require, inspects the forge webhook Recent Deliveries for the push event, and consults references/troubleshooting.md plus failure-signatures.md rather than randomly editing YAML. The diagnosis separates forge/webhook causes from agent-capacity or label-mismatch causes and records evidence found at each boundary.",
      "assertions": [
        "woodpecker-doctor.py or equivalent health verification happens before config changes",
        "Forge webhook delivery status is inspected",
        "Agent labels/backend matching is checked as a queue-starvation cause",
        "Troubleshooting/failure-signatures references guide the diagnosis instead of blind YAML edits"
      ]
    },
    {
      "id": "secret-safety-untrusted-prs",
      "prompt": "Our integration tests need the STAGING_API_TOKEN secret, but they also need to run on pull requests from forks. Make that work.",
      "expected_output": "Scenario: secrets vs untrusted code. The agent applies references/security.md defaults: by default secrets are not exposed to untrusted pull requests. It either restricts which events can access the secret and documents the threat model, moves fork-PR runs to a path that does not need the token, or gates on approval — never blanket-enabling the secret for fork PRs. Plugin images allowed to receive secrets are restricted too, and the trust implications of the chosen backend are stated.",
      "assertions": [
        "Blanket exposure of the secret to untrusted fork PRs is not implemented",
        "The security reference's default posture is cited",
        "An explicit alternative (restriction, gating, or token-free path) is proposed",
        "Plugin-image/event filters are part of any exposure decision"
      ]
    },
    {
      "id": "forge-admin-out-of-scope",
      "prompt": "Install Forgejo and set up its admin account, then hook Woodpecker up to it.",
      "expected_output": "Scenario: scope boundary. The skill explicitly covers Woodpecker's integration points, not forge administration. The agent states that installing/administering Forgejo itself is out of scope for this skill (use a forge-specific skill), and covers only the Woodpecker side: creating the OAuth app in the forge, setting WOODPECKER_* forge variables, callback URLs, and verifying the connection.",
      "assertions": [
        "Forge installation/administration is declared out of scope",
        "The user is pointed to a forge-specific skill for that half",
        "The Woodpecker-side integration steps are still provided"
      ]
    },
    {
      "id": "github-actions-not-this-skill",
      "prompt": "Add a GitHub Actions workflow that builds and publishes our Docker image on tags.",
      "expected_output": "Scenario: should-not-trigger. GitHub Actions is a different CI system entirely; woodpecker-ci is only for Woodpecker CI servers, agents, and their pipeline syntax. The agent does not load this skill; it writes a standard .github/workflows CI file directly.",
      "assertions": [
        "The skill is not loaded for a non-Woodpecker CI task",
        "No Woodpecker-specific YAML semantics are applied",
        "The requested workflow is handled as ordinary GitHub Actions configuration"
      ]
    },
    {
      "id": "upgrade-and-backup",
      "prompt": "We're upgrading Woodpecker from 2.x to 3.x next weekend. What's the safe procedure?",
      "expected_output": "Scenario: operations/upgrade planning. The agent loads references/operations.md (and source-index.md for version-sensitive notes): pin the target SemVer tag, take a full backup of the database before touching anything, read the breaking-changes section for the installed major version, upgrade the server before/alongside agents per documented ordering, verify health afterwards with woodpecker-doctor.py and logs, and keep a rollback path (previous image tag + restored DB) ready.",
      "assertions": [
        "A pre-upgrade database backup is mandated",
        "Breaking changes for the specific version pair are checked via the references",
        "Post-upgrade health verification is planned",
        "A rollback path exists before starting"
      ]
    }
  ]
}

# Advanced patterns

## Parallel fan-out and fan-in

Steps run serially unless their dependency graph allows parallel work. Use empty `depends_on` for independent roots and explicit dependencies for the join:

```yaml
steps:
  - name: lint
    depends_on: []
    image: golangci/golangci-lint:v2
    commands: ["golangci-lint run"]
  - name: test
    depends_on: []
    image: golang:1.23
    commands: ["go test ./..."]
  - name: package
    depends_on: [lint, test]
    image: alpine:3.20
    commands: ["./package.sh"]
```

For multiple files under `.woodpecker/`, a workflow name is derived from its filename without the leading dot and extension. `depends_on` can include an optional dependency when a path filter may skip it. Use optional dependencies deliberately; a skipped check should not silently remove a required quality gate.

## Concurrency control

Use a workflow concurrency limit for deployments that must not overlap:

```yaml
concurrency:
  limit: 1
  group: deploy-${CI_COMMIT_BRANCH}
```

The official workflows documentation notes that queued workflows in the same group start in pipeline-creation order, not in the order they become ready. This is useful for preserving deployment commit order.

## Matrix and platform routing

Matrices create a workflow for each combination. Use `include` for an explicit set of combinations, interpolate matrix variables with `${VARIABLE}`, and map a platform value to an agent label. Keep the expansion bounded and inspect the installed version's documented matrix limits before generating a large Cartesian product.

```yaml
matrix:
  include:
    - GO_VERSION: "1.22"
      platform: linux/amd64
    - GO_VERSION: "1.23"
      platform: linux/arm64
labels:
  platform: ${platform}
steps:
  - name: test
    image: golang:${GO_VERSION}
    commands: ["go test ./..."]
```

## Reusable YAML

The advanced-usage documentation supports YAML anchors and aliases. Use them for image names, plugin settings, and repeated conditions, but keep security-sensitive differences explicit enough to review. Test the rendered workflow with the CLI linter because interpolation happens before pipeline execution and may alter the value being parsed.

## Private registries and local images

Register private registry credentials in the Woodpecker UI or the documented backend mechanism. The server configuration documents `WOODPECKER_DOCKER_CONFIG` for a shared Docker credential file. Local images built during one step are backend- and host-dependent; a workflow that assumes a local image must be pinned to one suitable agent and normally requires trusted access to the Docker socket.

## Caching

Woodpecker shares the workspace between steps in one workflow, but separate workflows do not share that workspace. For cross-workflow or persistent caching, use a backend-appropriate named volume, PVC, or reviewed cache/artifact plugin. On Kubernetes, confirm the storage class supports the access mode and that the cache is not a hidden cross-branch contamination path.

## Autoscaling and ephemeral agents

The agent configuration supports one-shot execution for an agent that should run one workflow and exit. The optional autoscaler can create agents based on queue demand, but treat it as a separate operational component: verify provider credentials, min/max agent bounds, gRPC reachability, and its current feature coverage before relying on it for production capacity.

## Sources

- https://woodpecker-ci.org/docs/usage/workflows
- https://woodpecker-ci.org/docs/usage/workflow-syntax
- https://woodpecker-ci.org/docs/usage/matrix-workflows
- https://woodpecker-ci.org/docs/usage/advanced-usage
- https://woodpecker-ci.org/docs/usage/registries
- https://woodpecker-ci.org/docs/administration/configuration/autoscaler
- https://woodpecker-ci.org/docs/administration/configuration/agent

# CLI reference and local execution

The CLI surface changes between releases. Use the installed binary as the first authority:

```bash
woodpecker-cli --help
woodpecker-cli <command> --help
```

## Useful workflow

```bash
# Check the workflow before pushing
woodpecker-cli lint .woodpecker.yml

# Execute locally when the supported backend is available
woodpecker-cli exec .woodpecker.yml

# Inspect available repository operations
woodpecker-cli repo --help
woodpecker-cli pipeline --help
woodpecker-cli log --help

# Add a file-backed repository secret
woodpecker-cli repo secret add \
  --repository OWNER/REPO \
  --name deploy-key \
  --value @/secure/path/deploy-key
```

The official CLI documentation also exposes flags for setting pipeline metadata during local execution, including event, branch, changed files, commit SHA, and forge URL. Use `woodpecker-cli exec --help` for the exact installed names.

## Local execution limits

Local execution is valuable for command scripts, syntax, and metadata conditions. It does not automatically reproduce:

- the server's repository/organization/global secret lookup;
- forge webhook payloads and permissions;
- server-side plugin and secret filters;
- the exact agent labels and backend;
- production network policy or registry credentials.

Use a real non-secret pipeline after local checks pass. Never add a workaround for secrets based on an old issue or remembered flag without checking the current CLI help.

# Failure signatures and evidence-first diagnosis

| Symptom | First checks | Avoid |
|---|---|---|
| No pipeline after a push | Repository activation, forge webhook delivery, public `WOODPECKER_HOST`, matching workflow event | Rewriting steps before proving the webhook arrived |
| Pipeline queued indefinitely | Connected agent, matching labels, backend, `WOODPECKER_MAX_WORKFLOWS`, agent logs | Assuming the application image is broken before the agent receives work |
| Agent cannot connect | `WOODPECKER_SERVER` host/port, Docker DNS, gRPC port, shared secret, TLS mode | Changing workflow YAML |
| OAuth redirect loop | Public host, exact callback URL, provider URL, client/secret, clock/TLS | Regenerating pipeline files |
| Clone authentication failure | Forge reachability from agent, repository visibility, clone plugin trust, `WOODPECKER_AUTHENTICATE_PUBLIC_REPOS` | Printing credentials into a diagnostic step |
| Docker socket permission denied | Socket mount, Docker daemon, SELinux labels/policy, agent user | Permanently disabling SELinux as the first fix |
| Service hostname resolves but tests fail | Native readiness probe, service port, credentials, retry/backoff | Treating container creation as readiness |
| Kubernetes pod never starts | Namespace, RBAC, ServiceAccount, PVC/storage class, image pull secret, pod events | Enabling workflow-controlled service accounts without a trust review |
| Secret is empty | Scope precedence, event filter, plugin-image filter, `from_secret`, `$${VAR}` escaping | Broadening secret exposure to all pull requests |
| Logs are missing or truncated | Server/agent gRPC stability, log store setting, database limits, pipeline log commands | Deleting the database or volumes |
| Upgrade starts but old data is unavailable | Backup restore test, migration log, image tag alignment, file ownership | Downgrading binaries without restoring a pre-migration database |

## Evidence collection

```bash
docker compose ps
docker compose logs --tail=200 woodpecker-server
docker compose logs --tail=200 woodpecker-agent
woodpecker-cli lint --help
python3 scripts/woodpecker-doctor.py --url https://ci.example.com --server woodpecker-server:9000 --json
```

For clone failures, use `skip_clone: true` only in a temporary diagnostic workflow, verify network and forge API access from the same backend, then remove the diagnostic pause. Capture the first error and the exact installed version in the incident record.

## Sources

- https://woodpecker-ci.org/docs/usage/troubleshooting
- https://woodpecker-ci.org/docs/administration/configuration/agent
- https://woodpecker-ci.org/docs/administration/configuration/backends/docker
- https://woodpecker-ci.org/docs/administration/configuration/backends/kubernetes
- https://woodpecker-ci.org/docs/administration/configuration/server
- https://woodpecker-ci.org/docs/usage/secrets
- https://woodpecker-ci.org/docs/usage/project-settings

# Operations: secrets, CLI, persistence, and observability

## Secret administration

Use the UI or CLI. Load values from files so line breaks are preserved:

```bash
woodpecker-cli repo secret add \
  --repository OWNER/REPO \
  --name ssh_key \
  --value @/path/to/id_rsa
```

Add image and event restrictions for deployment credentials. Never print the value or place it in a workflow file. Woodpecker masks secrets from its own store, but external secrets fetched by a pipeline can still appear in logs.

## Registry credentials

Use Woodpecker's registry settings or the backend's documented credential mechanism. For Kubernetes, pull credentials can be supplied as Kubernetes Secrets named through `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`. Verify the agent can pull the exact image before blaming the workflow.

## CLI workflow

The CLI documentation is version-sensitive. Start with:

```bash
woodpecker-cli --help
woodpecker-cli repo --help
woodpecker-cli pipeline --help
woodpecker-cli exec --help
```

Authenticate using the method documented for the installed release, then use the CLI for repository, pipeline, log, and secret operations. Prefer `--help` and `woodpecker-cli lint` over memorizing flags from an older release.

`exec` runs a workflow locally. It can validate commands and metadata, but server-managed secrets and forge event context may not be present. Do not use local success as proof that a server pipeline will succeed.

## Database, metrics, and health

Woodpecker automatically creates and migrates its database in normal upgrades unless release notes say otherwise. It does **not** perform backups. Back up the database and persistent server data before upgrades, test restoration, and use a SemVer image tag rather than a floating `latest` tag. Keep server and agent majors aligned.

SQLite is the default embedded database path. The server configuration also documents MySQL and PostgreSQL through `WOODPECKER_DATABASE_DRIVER` and `WOODPECKER_DATABASE_DATASOURCE`. Verify the exact connection string for the installed release and create an external database before pointing Woodpecker at it. Never delete the server volume during routine troubleshooting.

Woodpecker exposes Prometheus metrics when `WOODPECKER_PROMETHEUS_AUTH_TOKEN` is set. `WOODPECKER_METRICS_SERVER_ADDR` enables an unprotected metrics listener; an empty value disables it. Treat metrics as a deployment-specific surface: confirm the endpoint, bind address, authentication, and network policy before adding a monitor.

Useful checks:

```bash
docker compose ps
docker compose logs --tail=200 woodpecker-server
docker compose logs --tail=200 woodpecker-agent
python3 scripts/woodpecker-doctor.py --url https://ci.example.com --json
```

## Safe upgrade loop

1. Read the release notes for the current major and target tag.
2. Back up the database and persistent server data.
3. Render the new deployment configuration.
4. Upgrade server and agent together in a disposable or staging environment.
5. Confirm login, repository listing, webhook delivery, agent connection, a non-secret test pipeline, and metrics/health behavior.
6. Upgrade production and retain the previous image tag for rollback.

## Sources

See `references/source-index.md` for server configuration, CLI, Docker Compose, agent, and Helm documentation.

# Pipeline syntax and reusable workflow patterns

## Minimal workflow

Woodpecker loads `.woodpecker.yml` or files under the configured `.woodpecker/` directory. A workflow is a serial list of steps by default; a non-zero step exits the workflow unless failure handling or a failure-status step applies.

```yaml
when:
  - event: [push, pull_request]
steps:
  - name: test
    image: golang:1.23
    commands:
      - go test ./...
```

The implicit clone step checks out the commit into the shared workspace. Changes made by one step are available to later steps.

## Step fields

- `image`: container image, plugin image, or shell source for the local backend.
- `commands`: serial shell commands for a build step. Do not combine with a plugin or service-only step.
- `environment`: step variables; use `from_secret` for Woodpecker-managed secrets.
- `settings`: plugin settings; a secret setting becomes a `PLUGIN_<NAME>` environment variable.
- `when`: conditional execution.
- `failure`: `fail` (default), `ignore`, or `cancel`.
- `depends_on`: allow independent steps to run in parallel and express ordering.
- `pull: true`: pull an updated image instead of using an already-present image.
- `detach: true`: run a long-lived step until the pipeline ends.
- `services`: service containers available by their step name as a hostname.
- `volumes`: backend-specific mounts; treat host mounts as privileged operations.
- `backend_options`: backend-specific settings such as Kubernetes resources.

## Conditions

A `when` list is OR across entries; predicates inside one entry are AND. Use `event` with `branch` when a branch filter must apply only to pushes. Available event names include `push`, `pull_request`, `pull_request_closed`, `pull_request_metadata`, `tag`, `release`, `deployment`, `cron`, and `manual`.

```yaml
when:
  - event: push
    branch: main
  - event: tag
    ref: refs/tags/v*
```

`status` accepts `success` and `failure` and is useful for notifications:

```yaml
- name: notify
  image: alpine:3.20
  commands: ["notify-command"]
  when:
    - status: [success, failure]
```

`path` conditions apply to push and pull-request events. `cron` filters cron event names. `platform` and `matrix` are useful with matrix workflows. Check the current workflow-syntax page for the complete condition list and glob semantics.

## Services and readiness

Services are reachable by their declared name and container port:

```yaml
services:
  - name: database
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: example
steps:
  - name: test
    image: postgres:16
    commands:
      - sleep 10
      - pg_isready -h database
```

A service being created does not mean it is ready. Prefer a bounded retry loop or a native readiness probe over an arbitrary long sleep. Service containers receive SIGTERM at the end of the pipeline and may be force-killed.

## Secrets

```yaml
steps:
  - name: publish
    image: registry.example/publisher:1
    environment:
      TOKEN:
        from_secret: publish-token
    settings:
      api_token:
        from_secret: publish-token
```

Woodpecker has repository, organization, and global secret scopes; the more specific scope wins according to the current documentation's precedence rules. Secrets are not exposed to pull requests by default. Restrict secrets by event and plugin image. When a shell must receive a literal variable expression, escape preprocessing with `$${TOKEN}`.

## Plugins

Plugins are pipeline steps with pre-defined behavior, configured through `settings`. Treat plugin images as executable code. Pin versions, restrict secret use by image, and prefer trusted or reviewed plugins for deployments.

## Matrices and labels

A matrix expands a workflow into one execution per combination. Use labels to route combinations to agents with the required platform or toolchain. Keep the matrix small and make the selected values visible in step names or logs.

```yaml
matrix:
  GO_VERSION: ["1.22", "1.23"]
  platform: [linux/amd64, linux/arm64]
labels:
  platform: ${platform}
steps:
  - name: test-${GO_VERSION}
    image: golang:${GO_VERSION}
    commands: ["go test ./..."]
```

## Multiple workflows

Files in `.woodpecker/` become separate workflows. Their names derive from filenames without the path, leading dots, or `.yml`/`.yaml` extension when used in `depends_on`. Use dependencies to express a DAG, not to create an accidental serial bottleneck.

## Advanced YAML

The official advanced-usage documentation covers YAML anchors and aliases for reducing duplication. Use them sparingly: explicit repeated steps are easier to review when the configuration is security-sensitive or contains different conditions.

## Local checks

```bash
woodpecker-cli lint .woodpecker.yml
woodpecker-cli exec .woodpecker.yml
```

Local execution does not reproduce forge webhooks, server-side secret scope, or every backend behavior. Treat it as a fast syntax/command check, then run a real pipeline.

# Security and trust boundaries

## Assume pipeline code is hostile unless proven otherwise

A repository author controls workflow commands. The backend determines what those commands can reach. Treat every agent as a trust domain and isolate agents by repository trust, network access, credentials, and host capability.

## Docker backend

The Docker agent commonly mounts `/var/run/docker.sock`. A pipeline can often use that socket to control the host daemon, so do not place untrusted pull-request workloads on an agent with sensitive host access. Use dedicated agents, labels, network segmentation, and least-privilege credentials.

## Local backend

The local backend runs directly in the agent's filesystem and user context. Official documentation warns that a malicious pipeline can access agent configuration and the `WOODPECKER_AGENT_SECRET`. Use it only when every pipeline author is trusted and the host is disposable or tightly isolated.

## Pull requests and secrets

Secrets are not exposed to pull requests by default. Keep that default. If a workflow truly needs a secret on pull requests:

- enable the event deliberately;
- restrict the secret to the smallest repository/organization scope;
- restrict it to exact plugin images where possible;
- avoid arbitrary shell access to the secret;
- assume a public repository can exfiltrate it;
- use a short-lived, low-privilege credential.

Woodpecker can mask secrets from its own store, but an external secret fetched by a pipeline may appear in logs.

## Plugins

Plugins are executable images, not declarative configuration. Pin image tags or digests, review the source, restrict which secrets can be passed to them, and avoid privileged plugins unless the agent is dedicated and trusted. Keep deployment credentials out of general-purpose build steps.

## Kubernetes

Keep workflow-controlled ServiceAccount selection disabled unless required. If enabled, it can allow a user with push access to run pods under another ServiceAccount in the namespace. Use namespace isolation, RBAC, network policy, resource limits, non-root settings where compatible, and private registry pull secrets.

## Registration and transport

Use a high-entropy agent secret, separate credentials per environment where supported, and encrypted gRPC for remote agents. Do not expose the agent endpoint broadly. `WOODPECKER_OPEN=false` is the safer default for registration.

## Public configuration audit

Before publishing a workflow or skill, scan for:

- real OAuth client secrets, agent tokens, registry passwords, and private keys;
- internal hostnames, domains, usernames, filesystem paths, or repository names;
- unpinned images used for privileged operations;
- examples that accidentally expose secrets to pull requests.

Use placeholders in templates and state where values come from; do not include real values.

# Setup and backend selection

## Docker Compose baseline

The official Compose deployment uses a server and one or more agents. The server persists `/var/lib/woodpecker`; a Docker agent needs access to `/var/run/docker.sock` and the agent config directory. Both server and agent must receive the same `WOODPECKER_AGENT_SECRET`. Generate it with `openssl rand -hex 32` and keep it out of Git.

Before starting:

1. Create an OAuth application in the forge.
2. Set the callback URL to the Woodpecker callback documented for the installed major version and public host.
3. Set `WOODPECKER_HOST` to the externally reachable URL, not the container hostname.
4. Set the forge server URL and OAuth client/secret variables.
5. Decide whether registration is open. Prefer `WOODPECKER_OPEN=false` and explicitly grant admin access.
6. Render the Compose model with `docker compose config --quiet`.
7. Start, inspect `docker compose ps`, and read server and agent logs.

Use the included `templates/docker-compose.yml` as a starting point, then pin the exact server and agent image tag together. Do not mix major versions.

## Forgejo and Gitea

Woodpecker has built-in Forgejo and Gitea integrations. Use the provider-specific variables documented for the installed major version:

```text
# Forgejo
WOODPECKER_FORGEJO=true
WOODPECKER_FORGEJO_URL=https://forge.example.com
WOODPECKER_FORGEJO_CLIENT=<oauth-client-id>
WOODPECKER_FORGEJO_SECRET=<oauth-client-secret>

# Gitea
WOODPECKER_GITEA=true
WOODPECKER_GITEA_URL=https://forge.example.com
WOODPECKER_GITEA_CLIENT=<oauth-client-id>
WOODPECKER_GITEA_SECRET=<oauth-client-secret>
```

Do not assume `GITEA` and `FORGEJO` variable names are interchangeable. Confirm the callback URL, OAuth scopes, skip-verification behavior, and provider-specific behavior from the current provider page.

After login, activate the repository in Woodpecker. Activation requires repository administration rights because Woodpecker installs a webhook. Confirm the webhook reaches the server before debugging the workflow file.

## Agent registration

Minimum agent configuration:

```text
WOODPECKER_SERVER=woodpecker-server:9000
WOODPECKER_AGENT_SECRET=<same shared secret as server>
WOODPECKER_MAX_WORKFLOWS=2
```

On first connection, the server registers the agent and returns an identity. Persist the file configured by `WOODPECKER_AGENT_CONFIG_FILE`; otherwise a recreated agent may register again. For manually registered agents, create the agent in the UI under `Settings -> Agents -> Add agent` and provide the generated token as `WOODPECKER_AGENT_SECRET`.

Use labels to route workloads to suitable agents. Check that a workflow's labels match an available agent before diagnosing application failures.

## Backend choice

| Backend | Use when | Main boundary |
|---|---|---|
| Docker | You need isolated container steps on a Docker host | Docker socket and host trust |
| Kubernetes | You need pod scheduling, resource requests, and cluster isolation | RBAC, PVC/storage, namespace and service accounts |
| Local | You control every pipeline author and need host-native execution | No isolation; pipelines can access the agent host/config |

The Docker backend starts each step in a separate container and shares the workspace volume. The Kubernetes backend creates standalone pods and a temporary PVC for pipeline file transfer. The local backend runs commands in the agent's own context and is unsafe for untrusted repositories.

## Remote agents and TLS

Agents connect to the server's gRPC endpoint. For agents outside the private network, use the documented secure gRPC settings for the installed release, validate certificates, and expose only the required port through a firewall. A TCP connection is not proof of a valid agent handshake: check the agent log and UI state.

## Kubernetes essentials

Set the namespace and backend options explicitly. For private images, place pull credentials in Kubernetes Secrets and list them through `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`. Add per-step resource requests and limits. Keep service-account selection from workflow files disabled unless the repository authors are trusted; enabling it can permit privilege escalation within the namespace.

## Source

See `references/source-index.md` for the official installation, forge, agent, Docker, Kubernetes, and server URLs.

# Source index and freshness notes

The skill was researched with GroktoCrawl against primary Woodpecker documentation and the official repository. URLs below are the source of truth for claims; paths can move between major versions.

## Official documentation

- General administration: https://woodpecker-ci.org/docs/administration/general
- Docker Compose installation: https://woodpecker-ci.org/docs/administration/installation/docker-compose
- Server configuration: https://woodpecker-ci.org/docs/administration/configuration/server
- Agent configuration and registration: https://woodpecker-ci.org/docs/administration/configuration/agent
- Forge overview: https://woodpecker-ci.org/docs/administration/configuration/forges/overview
- Gitea integration: https://woodpecker-ci.org/docs/administration/configuration/forges/gitea
- Forgejo integration: https://woodpecker-ci.org/docs/administration/configuration/forges/forgejo
- Docker backend: https://woodpecker-ci.org/docs/administration/configuration/backends/docker
- Kubernetes backend: https://woodpecker-ci.org/docs/administration/configuration/backends/kubernetes
- First pipeline: https://woodpecker-ci.org/docs/usage/intro
- Workflow syntax: https://woodpecker-ci.org/docs/usage/workflow-syntax
- Multiple workflows/dependencies: https://woodpecker-ci.org/docs/usage/workflows
- Matrix workflows: https://woodpecker-ci.org/docs/usage/matrix-workflows
- Services: https://woodpecker-ci.org/docs/usage/services
- Secrets: https://woodpecker-ci.org/docs/usage/secrets
- Plugins: https://woodpecker-ci.org/docs/usage/plugins/overview
- Environment variables: https://woodpecker-ci.org/docs/usage/environment
- Advanced YAML: https://woodpecker-ci.org/docs/usage/advanced-usage
- Troubleshooting: https://woodpecker-ci.org/docs/usage/troubleshooting
- CLI: https://woodpecker-ci.org/docs/cli
- Helm chart and metrics: https://woodpecker-ci.org/docs/administration/installation/helm-chart
- Project settings, trust, and approval: https://woodpecker-ci.org/docs/usage/project-settings
- Registries: https://woodpecker-ci.org/docs/usage/registries
- Autoscaler: https://woodpecker-ci.org/docs/administration/configuration/autoscaler

## Official repository

- Source and release information: https://github.com/woodpecker-ci/woodpecker
- Compose example: https://github.com/woodpecker-ci/woodpecker/blob/main/docker-compose.example.yaml

## Freshness rules

- Verify environment-variable names against the provider page for the installed major version.
- Keep server and agent image majors aligned.
- Prefer a SemVer tag or controlled major/minor tag; do not use an unreviewed floating `latest` tag.
- Treat issue discussions and old examples as historical context, not current API documentation.
- If docs and a local `--help` output disagree, record the installed version and follow the local binary for that invocation while checking release notes for migration impact.

# Troubleshooting playbooks

## First classify the boundary

Capture the first error, not the last cascade. Check in order:

1. Forge OAuth/webhook and repository activation
2. Server startup, database, and public URL
3. Agent handshake, labels, backend, and capacity
4. Clone/network/credentials
5. Image pull or backend permissions
6. Workflow syntax/conditions/secrets
7. Application command or test

## Server or login failure

- Inspect server logs from startup through the request.
- Confirm `WOODPECKER_HOST` is the public URL and the OAuth callback matches it exactly.
- Confirm the forge enable flag, forge server URL, client ID, and client secret are for the same provider.
- Check proxy headers, TLS termination, and clock skew.
- Do not edit pipeline YAML until repository activation and login work.

## Agent connected but idle

- Confirm server and agent use the same `WOODPECKER_AGENT_SECRET`.
- Confirm the agent's `WOODPECKER_SERVER` is the gRPC endpoint, not the HTTP UI URL.
- Check agent labels against workflow labels and backend selection.
- Check `WOODPECKER_MAX_WORKFLOWS` and whether all slots are occupied.
- Read agent logs for handshake, registration, image, and backend errors.
- Persist the agent config file so its identity is not regenerated after every restart.

## Clone failure

For errors such as `fatal: could not read Username`:

1. Check repository visibility and forge credentials.
2. Check whether `WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=true` is needed for internal/public repository behavior in the installed release.
3. Prove network reachability from the agent/container. Temporarily use `skip_clone: true` and a diagnostic step with `ping`, `wget`, or equivalent, then pause the container.
4. Enter the exact running container and reproduce the `git init`, `git remote add`, and `git fetch` commands with safe test credentials.
5. Remove the diagnostic pause and restore normal checkout.

Never put credentials in logs or a committed workflow.

## Docker backend failure

- Confirm the agent can access Docker and the socket path is correct.
- On SELinux hosts, use the documented `:z`/`:Z` labeling or an appropriate policy; do not permanently disable SELinux merely to test.
- Check image architecture, registry authentication, disk space, and daemon logs.
- Remember that mounting the Docker socket gives pipeline workloads control over the host Docker daemon.

## Kubernetes backend failure

- Check agent namespace, ServiceAccount, RBAC, PVC/storage class, and pull secrets.
- Verify the temporary workspace PVC can be created and mounted.
- Add per-step CPU/memory requests and limits to distinguish scheduling from application failure.
- If a workflow sets `serviceAccountName`, check whether the agent allows it. Enabling this setting can grant arbitrary namespace permissions to anyone who can push.
- Inspect pod events and the labels Woodpecker adds (`woodpecker-ci.org/repo-id`, `repo-full-name`, `branch`, `task-uuid`, and `step`).

## Workflow and condition surprises

- `when` entries are OR; fields inside one entry are AND.
- A branch filter can match a pull request's target branch. Add `event: push` for push-only behavior.
- A step normally runs only after prior success. Use `status: [failure]` for failure handlers.
- `commands` are shell commands in build steps; plugin/service containers do not use them the same way.
- If a secret appears empty, check its scope, event filter, plugin-image filter, and `$${VAR}` escaping.
- Run `woodpecker-cli lint` before pushing.

## Service readiness

A service hostname resolving only proves DNS/network setup. Add a bounded readiness loop using the service's native probe (`pg_isready`, `mysqladmin ping`, HTTP health endpoint) and fail with a useful message. Avoid unbounded sleeps.

## SELinux symptoms

`permission denied while trying to connect to the Docker daemon socket` on RHEL-like hosts may be an SELinux labeling/policy issue. Check audit logs and use a correctly labeled mount or policy. Permissive mode is a temporary diagnostic only.

## Evidence template

Use `assets/troubleshooting-checklist.md`; preserve the first error, exact image tags, relevant configuration names, and commands run. Redact secret values.

#!/usr/bin/env python3
"""Small Woodpecker CI connectivity/configuration probe; stdlib only."""
import argparse, json, os, socket, sys, urllib.error, urllib.request


def check_url(url):
    try:
        req = urllib.request.Request(url, method="GET")
        with urllib.request.urlopen(req, timeout=5) as r:
            return {"ok": True, "status": r.status, "url": url}
    except urllib.error.HTTPError as e:
        return {"ok": e.code < 500, "status": e.code, "url": url, "error": str(e)}
    except Exception as e:
        return {"ok": False, "url": url, "error": str(e)}


def main():
    ap = argparse.ArgumentParser(description="Probe Woodpecker URL and deployment environment")
    ap.add_argument("--url", default=os.getenv("WOODPECKER_HOST", ""), help="Woodpecker HTTP URL")
    ap.add_argument("--server", default=os.getenv("WOODPECKER_SERVER", ""), help="agent gRPC host:port")
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args()
    out = {"checks": [], "environment": {"WOODPECKER_HOST": bool(os.getenv("WOODPECKER_HOST")), "WOODPECKER_AGENT_SECRET": bool(os.getenv("WOODPECKER_AGENT_SECRET")), "WOODPECKER_SERVER": bool(os.getenv("WOODPECKER_SERVER"))}}
    if args.url:
        base = args.url.rstrip("/")
        out["checks"].append(check_url(base))
        out["checks"].append(check_url(base + "/healthz"))
    else:
        out["checks"].append({"ok": False, "error": "set --url or WOODPECKER_HOST"})
    if args.server:
        host, sep, port = args.server.rpartition(":")
        if sep:
            try:
                with socket.create_connection((host or "localhost", int(port)), timeout=5):
                    out["checks"].append({"ok": True, "tcp": args.server})
            except Exception as e:
                out["checks"].append({"ok": False, "tcp": args.server, "error": str(e)})
        else:
            out["checks"].append({"ok": False, "error": "--server must be host:port"})
    out["ok"] = all(c.get("ok", False) for c in out["checks"])
    print(json.dumps(out, indent=2) if args.json else "\n".join([("PASS" if c.get("ok") else "FAIL") + " " + json.dumps(c, sort_keys=True) for c in out["checks"]]))
    return 0 if out["ok"] else 1

if __name__ == "__main__":
    sys.exit(main())

# Woodpecker CI v3-style deployment skeleton.
# Pin a specific SemVer release for production instead of relying on a moving tag.
services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:v3
    ports:
      - "8000:8000"
      - "9000:9000"
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker
    environment:
      WOODPECKER_HOST: ${WOODPECKER_HOST:?set the public Woodpecker URL}
      WOODPECKER_OPEN: ${WOODPECKER_OPEN:-false}
      WOODPECKER_ADMIN: ${WOODPECKER_ADMIN:?set the admin username}
      WOODPECKER_AGENT_SECRET: ${WOODPECKER_AGENT_SECRET:?set the shared agent secret}
      WOODPECKER_FORGEJO: ${WOODPECKER_FORGEJO:-true}
      WOODPECKER_FORGEJO_URL: ${WOODPECKER_FORGEJO_URL:?set the Forgejo URL}
      WOODPECKER_FORGEJO_CLIENT: ${WOODPECKER_FORGEJO_CLIENT:?set the Forgejo OAuth client ID}
      WOODPECKER_FORGEJO_SECRET: ${WOODPECKER_FORGEJO_SECRET:?set the Forgejo OAuth client secret}

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:v3
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - woodpecker-agent-config:/etc/woodpecker
    environment:
      WOODPECKER_SERVER: woodpecker-server:9000
      WOODPECKER_AGENT_SECRET: ${WOODPECKER_AGENT_SECRET:?set the shared agent secret}
      WOODPECKER_MAX_WORKFLOWS: ${WOODPECKER_MAX_WORKFLOWS:-2}

volumes:
  woodpecker-server-data:
  woodpecker-agent-config:

# Conservative workflow skeleton. Adapt images and commands to the project.
when:
  - event: [push, pull_request]

steps:
  - name: test
    image: alpine:3.20
    commands:
      - printf '%s\n' "run project tests here"

  - name: deploy
    image: alpine:3.20
    commands:
      - printf '%s\n' "deploy only from a protected branch"
    when:
      - event: push
        branch: main

  - name: notify-failure
    image: alpine:3.20
    commands:
      - printf '%s\n' "notify the team"
    when:
      - status: [failure]