pypi-ops
Publish Python packages to PyPI via OIDC Trusted Publishing (PEP 740 attestations, gh-action-pypi-publish) instead of stored tokens. Use for: invalid-publisher errors, pending-publisher 404s, uv publish/twine, TestPyPI, environment approval gates, and rotating/auditing publish to
Install
npx skills add https://github.com/0xDarkMatter/claude-mods/tree/main/skills/pypi-ops
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install 0xdarkmatter-claude-mods@llmmart
git clone https://github.com/0xDarkMatter/claude-mods.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole 0xdarkmatter/claude-mods collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
PyPI Operations
Publish Python packages to PyPI on the 2026 best-practice path: OIDC Trusted
Publishing with signed PEP 740 attestations, no long-lived token to leak. This
skill owns the publish layer (the registry handshake, the first-publish
gotchas, the recovery playbook). General GitHub Actions syntax is ci-cd-ops;
the install-side worm defense is supply-chain-defense; gh/release-page
mechanics are github-ops.
Where this fits — the release pipeline
A release spans several skills; pypi-ops owns the registry step. Chain them:
- Vet dependencies before cutting a release —
supply-chain-defense(cooldown + behavioural scan). The build runs dependency code before it touches your publish credential, so a poisoned build dep can steal the token. - Preflight —
scripts/publish-preflight.sh --build .(this skill). - Bump → tag → push —
git-ops(its push-gate scans for secrets / forbidden files before the tag goes up). - CI publishes via OIDC — this skill's
assets/publish.yml; you approve at thepypienvironment gate. - Release page (optional, GitHub) —
github-ops, human-reviewed notes.
The one decision: OIDC vs API token
Default to OIDC Trusted Publishing. Reach for a token only when OIDC is impossible (publishing from a non-supported CI, or a one-off local push).
| Trusted Publishing (OIDC) ← default | API token | |
|---|---|---|
| Secret stored | None — short-lived OIDC token minted per run | Long-lived pypi-… token in a secret |
| Leak/phish blast radius | None to steal | Full publish rights until rotated |
| Provenance | PEP 740 attestations (signed, verifiable) | None by default |
| Setup | One-time publisher registration on PyPI | Generate token + store secret |
| Best for | All CI/CD releases | Legacy CI, emergency local upload |
If a repo currently uses a token, migrating to OIDC is strictly an upgrade — see references/trusted-publishing.md.
The #1 gotcha: first publish needs a pending publisher
A Trusted Publisher is normally configured under the project's settings on PyPI — but on the first ever publish the project doesn't exist yet, so there's nothing to configure it under. The fix is a pending publisher, registered at the account level before the first upload.
Symptom (the exact failure this skill exists to kill):
Trusted publishing exchange failure:
* invalid-publisher: valid token, but no corresponding publisher
(Publisher with matching claims was not found)
The OIDC token was valid; PyPI just has no publisher matching the claims. Fix:
PyPI → https://pypi.org/manage/account/publishing/ → Add a pending publisher
Field Value PyPI Project Name the dist name from pyproject.toml[project].nameOwner GitHub org/user Repository name repo name Workflow name the filename, e.g. publish.yml(not thename:)Environment name must equal the job's environment:(e.g.pypi)
All four claims must match the run's OIDC token exactly. After the first
successful publish, the pending publisher auto-converts to a normal project
publisher — no further action. Run diagnose-publish.sh on a failed run to read
the exact claims it presented and compare them field-by-field.
This is the most common silent-failure mode: a package's
publish.ymllooks perfect and every release builds green, yet nothing ever reaches PyPI because the publisher was never registered. Check it first.
Recommended workflow (copy assets/publish.yml)
The shipped template is hardened to the patterns below — adapt the marked points
and drop it in .github/workflows/. Non-negotiables it encodes:
on: push: tags: ['v*']— release on a version tag, never on every push.- OIDC, no token: the
publishjob haspermissions: id-token: writeandpypa/gh-action-pypi-publishwithattestations: true. Nopassword:/token. environment: pypion the publish job → a human approves every release (defense-in-depth: even a compromised repo can't auto-ship).- Build/publish split: a
buildjob (no elevated perms) produces + uploads thedistartifact;publishdownloads it. Least privilege per job. uv sync --locked+pip-audit: the release is built against the committed, hash-verified lockfile and blocked if a dep has a known CVE.twine check/ metadata validation before upload.- SHA-pinned actions with a trailing
# vXcomment (mutable tags get hijacked — seecheck-action-pins.py). - Verify-on-PyPI tail job — polls the JSON API so a silent publish failure (accepted-but-not-live, CDN lag) surfaces loudly instead of looking fine.
Supply-chain hardening — the publisher side
Stealing your publish credential lets an attacker ship malware to everyone who installs you — so the publish path is the surface the 2026 worm campaign (Mini Shai-Hulud) targets, minting PyPI/npm tokens from stale OIDC trust and orphaned workflows. The template above isn't just convention; each choice is a defense:
| Control | Defends against |
|---|---|
| OIDC, no stored token | Credential theft/phishing — there is no long-lived secret to steal |
| PEP 740 attestations | Tampered artifacts — provenance is signed and verifiable |
environment: pypi + reviewers |
A compromised repo/CI auto-shipping — a human still gates the release |
pip-audit gate |
A knowingly-vulnerable dependency reaching the release build |
SHA-pinned actions (check-action-pins.py) |
Action-tag hijacks (tj-actions, 2025) repointing @vN to a malicious commit |
permissions: {} + per-job least privilege |
A poisoned build step escalating beyond read |
uv sync --locked |
Build-time dependency injection / silent re-resolution |
Then audit the trust itself, not just the workflow:
- Revoke stale Trusted Publishers / OIDC federation you no longer use — an orphaned publisher bound to a deletable workflow is the Mini Shai-Hulud entry point. Review PyPI → project → Publishing periodically.
- If a token is in play, rotate it (project-scoped, short-lived) — better, migrate to OIDC and delete it. See trusted-publishing.md.
- Vet build dependencies before a release, not after — a poisoned
uv syncstep runs before your OIDC token is even minted.
Division of labour: pypi-ops owns publisher hardening; supply-chain-defense
owns the install side and ships integrity-audit.sh (hunts pull_request_target
- OIDC misconfig and worm persistence) — run it on any repo that publishes, and gate dependency bumps through its cooldown + behavioural scan.
Cutting a release — preflight then tag
Before tagging, run the preflight so a release never fails on something mechanical (version skew, dirty lock, missing publisher config):
scripts/publish-preflight.sh . # human summary; exit 10 = not ready
scripts/publish-preflight.sh --build . # also build + twine-check the dist
scripts/publish-preflight.sh --json . | jq '.data[] | select(.ok==false)'
It checks: pyproject version == __init__.__version__, the version is not
already on PyPI (uploads are immutable — you cannot re-push 1.2.3), the
lockfile self-version matches, a tag (if present) matches the version, and the
publish workflow uses OIDC (flags a stored token). --build additionally
verifies the package actually builds and passes twine check. Dynamic-versioned
projects (hatch-vcs / setuptools-scm) are read from the HEAD tag. Green → bump,
commit, tag, push the tag; CI builds, waits at the pypi environment gate, you
approve.
When a publish fails — classify, don't guess
scripts/diagnose-publish.sh <run-id> # reads gh run log, names the cause + fix
gh run view <run-id> --log-failed | scripts/diagnose-publish.sh - # or pipe a log
The high-frequency failure classes and their fixes:
| Symptom | Cause | Fix |
|---|---|---|
invalid-publisher / claims not found |
No (pending) publisher on PyPI | Register the pending publisher (above) |
File already exists / 400 on upload |
Version already on PyPI (immutable) | Bump the version; never reuse — see recovery |
| Job stuck "Waiting" | environment: pypi needs approval |
Approve the deployment in the run's UI |
environment … not found |
Publisher claim names an env the job lacks | Make environment: and the publisher's Environment match |
| Built green, not on PyPI | Silent accept / no verify step | Add the verify-on-PyPI job; re-run |
non-OIDC/token rejected |
Token wrong/expired, or OIDC misread as token | Prefer OIDC; if token, rotate + re-store |
Full catalogue with the underlying mechanics: references/recovery-playbook.md.
Local & manual publishing (uv / twine)
For a one-off or a non-CI environment. Prefer uv in 2026 (faster, native):
uv build # sdist + wheel into dist/
uv publish --trusted-publishing automatic # OIDC if in supported CI, else prompts
# token path (store in ~/.pypirc or env, never inline on the CLI history):
UV_PUBLISH_TOKEN="pypi-…" uv publish
twine remains the canonical fallback and the metadata validator (the GitHub
Action wraps it internally):
python -m twine check dist/* # ALWAYS run before any upload
python -m twine upload dist/* # token from ~/.pypirc; legacy path
Never hand-roll the HTTP upload. Details + ~/.pypirc shape:
references/uv-publish.md.
Dry-run on TestPyPI first
For a brand-new package or a risky metadata change, publish to test.pypi.org
first — it has its own separate accounts and its own pending-publisher
registration. Point the action at repository-url: https://test.pypi.org/legacy/
and register the pending publisher on TestPyPI. See
references/trusted-publishing.md.
Keeping the workflow from rotting
The pinned action SHAs and pypa/gh-action-pypi-publish major drift over time.
The verifier flags it before a release does:
scripts/check-action-pins.py --offline .github/workflows/publish.yml # structure: all pinned + commented
scripts/check-action-pins.py --live .github/workflows/publish.yml # resolve tags → flag SHA drift
--offline is the PR gate (every uses: is SHA-pinned with a # vX comment);
--live runs scheduled (resolves each pin against GitHub and exits 10 on drift,
7 if GitHub is unreachable — advisory, never a flaky block).
Publishing many packages (a fleet)
When several repos publish the same way, don't copy publish.yml N times — each
copy drifts its own SHA pins. Hoist the publish job into a reusable workflow
(on: workflow_call) in one repo, and have each package's tiny caller pass its
dist name. OIDC still works: the caller's workflow_ref is what PyPI matches,
so register each package's pending publisher against the caller filename
(e.g. release.yml), not the shared one. One place to refresh pins
(check-action-pins.py on the reusable workflow); one approval gate definition;
per-package publishers. See references/trusted-publishing.md
for the claim that must match.
Reference files
| File | Load when |
|---|---|
| references/trusted-publishing.md | Setting up OIDC, pending vs project publisher, OIDC claim semantics, environments, TestPyPI, token→OIDC migration |
| references/recovery-playbook.md | A publish failed and you need the full failure-class catalogue + mechanics |
| references/uv-publish.md | Local/manual publishing, uv build/uv publish, twine, ~/.pypirc, build backends |
Files (claude-mods)
-
assets
-
publish.yml 4.4 KB
# PyPI publish — OIDC Trusted Publishing, hardened (claude-mods/pypi-ops template). # # ADAPT POINTS (search "ADAPT"): # 1. <DIST-NAME> — your distribution name (pyproject [project].name), used by # the verify job. Must match the project on PyPI. # 2. environment — keep `pypi`; if you change it, change the PyPI publisher's # "Environment name" claim to match (they must be equal). # 3. Build steps — the template assumes uv + a committed uv.lock. If you don't # use uv, replace the "Sync locked environment" + "Build" # steps with `python -m build` and drop `uv sync --locked`. # # PREREQUISITE (the #1 first-publish failure): register a Trusted Publisher on # PyPI. First publish → a *pending publisher* at # https://pypi.org/manage/account/publishing/ with claims matching THIS file: # Owner=<org/user> Repository=<repo> Workflow=publish.yml Environment=pypi # Without it the publish step fails `invalid-publisher`. See pypi-ops SKILL.md. # # SHA PINS: every `uses:` is pinned to a full commit SHA with a trailing `# vX` # comment (mutable @vN tags have been hijacked in the wild — tj-actions, 2025). # Refresh with: scripts/check-action-pins.py --live .github/workflows/publish.yml name: Publish to PyPI on: push: tags: - "v*" # release on a version tag only — never on ordinary pushes permissions: {} # least privilege: grant per-job below concurrency: # never let two tag pushes publish the same project at once group: pypi-publish-${{ github.repository }} cancel-in-progress: false # let an in-flight publish finish; queue the next jobs: build: name: Build + audit (locked) runs-on: ubuntu-latest permissions: contents: read steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 with: enable-cache: false # ADAPT (3): reproducible, hash-verified build env from the committed lock. - name: Sync locked environment run: uv sync --locked --no-dev # A freshly-disclosed CVE in the dep set blocks the release. - name: Audit dependencies run: uvx pip-audit --progress-spinner off - name: Build sdist + wheel run: uv build - name: Validate artifacts (PyPI ingestion check) run: uvx twine check dist/* - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: dist path: dist/ if-no-files-found: error publish: name: Publish to PyPI (trusted publishing) needs: build runs-on: ubuntu-latest environment: pypi # ADAPT (2): human-approval gate; matches publisher claim permissions: id-token: write # OIDC for trusted publishing — the only elevated grant contents: read steps: - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: dist path: dist/ - uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0 with: attestations: true # PEP 740 signed provenance # NO password: — OIDC trusted publishing supplies the credential. # TestPyPI: add repository-url: https://test.pypi.org/legacy/ verify: name: Verify on PyPI needs: publish runs-on: ubuntu-latest permissions: contents: read steps: # Catches "PyPI accepted the upload but the version isn't live" — silent # rejection / CDN propagation lag that makes a failed release look fine. - name: Confirm version is queryable on PyPI env: DIST_NAME: "<DIST-NAME>" # ADAPT (1) run: | version="${GITHUB_REF_NAME#v}" echo "Verifying ${DIST_NAME}==${version} is live on PyPI..." for attempt in 1 2 3 4 5; do if curl -fsS "https://pypi.org/pypi/${DIST_NAME}/${version}/json" >/dev/null; then echo "OK ${DIST_NAME}==${version} is live" exit 0 fi echo " attempt ${attempt}: not visible yet, waiting 15s..." sleep 15 done echo "FAIL ${DIST_NAME}==${version} not visible after 75s" exit 1
-
-
references
-
recovery-playbook.md 5.1 KB
# PyPI publish recovery playbook The failure classes `diagnose-publish.sh` recognises, the mechanics behind each, and the fix. Run the diagnoser first; this is the depth behind its verdict. ## `invalid-publisher` — no matching Trusted Publisher **Mechanics.** The OIDC token was valid; PyPI found no publisher whose registered claims match the run. Almost always: a first publish with no *pending publisher*, or a claim mismatch (workflow filename, environment, renamed repo/owner). **Fix.** Register the (pending) publisher — see [trusted-publishing.md](trusted-publishing.md). Compare the run's presented claims (the action prints them; `diagnose-publish.sh` extracts them) field-by-field against what you registered. Then `gh run rerun <id> --failed` — no re-tag needed; the same claims will match once the publisher exists. ## `File already exists` — immutable version **Mechanics.** PyPI versions are **write-once**. A filename (`pkg-1.2.3-*.whl`) can never be re-uploaded, *even after you delete the release* — deletion does not free the name. This is deliberate, to keep installs reproducible. **Fix.** Bump to a new version (patch is fine), commit the bump across `pyproject.toml` + `__init__` + lockfile, re-tag, push. `skip-existing: true` on the action only tolerates a *partial* re-run (some files already up) — it never replaces an existing file and is not a way to "re-release" a version. ## Job stuck "Waiting" — environment approval **Mechanics.** `environment: pypi` with required reviewers pauses the job until a human approves in the run UI. Not a failure — by design. **Fix.** Approve the deployment (or remove the reviewer requirement if the gate isn't wanted). The environment claim still binds the publisher either way. ## `environment not allowed/found` — claim mismatch **Mechanics.** The Trusted Publisher was registered with an Environment name the job doesn't set, or the job sets one the publisher doesn't list. **Fix.** Make `jobs.publish.environment` equal the publisher's Environment name verbatim. Leaving the publisher's Environment blank means the job must NOT set one — they must agree. ## `403 Forbidden` / `isn't allowed to upload` **Mechanics.** Credential refused. For OIDC: the identity is publishing to a project it has no publisher for (or to create a project without a pending publisher). For tokens: wrong/expired/insufficient scope. **Fix.** OIDC path — confirm the publisher exists for this exact project. Token path — rotate the token, re-store the secret, ensure project scope. New project + OIDC always needs a pending publisher first. ## Built green but not live on PyPI — silent accept **Mechanics.** The upload returned success but the version isn't queryable (rejected post-accept, or CDN propagation lag). Without a verify step the run looks fully green while nothing is installable. **Fix.** Add the verify-on-PyPI job (polls `https://pypi.org/pypi/<name>/<ver>/json`) from `assets/publish.yml`. If it was a real rejection, the cause is usually metadata — fix and bump. ## `twine check` failed — bad metadata **Mechanics.** The wheel/sdist metadata is malformed (README `content-type` mismatch, missing fields). Caught before upload by `twine check`. **Fix.** Correct `[project]` metadata (notably `readme` + its content type), rebuild, `python -m twine check dist/*` until clean locally. Run it in CI before the upload step so this never reaches PyPI. ## pip-audit / build gate failed (not a publish failure) **Mechanics.** The `build` job failed before publish — a dependency CVE (`pip-audit`), a lock/pyproject divergence (`uv sync --locked`), or a build error. **Fix.** This is a dependency/build issue, not PyPI. Patch the dep (see `supply-chain-defense`), re-resolve the lock, or fix the build; the publish never ran. ## Shipped a broken release — yank, don't delete **Mechanics.** Deleting a release frees nothing (the version name stays burned forever) and *breaks* anyone who pinned it. **Yanking** is the right tool: a yanked version stays installable by an exact pin (`pkg==1.2.3`) so existing lockfiles keep working, but resolvers skip it for new/range installs. **Fix.** PyPI → project → Manage → the release → **Yank** (with a reason). Then publish a fixed **higher** version. Reserve deletion for secrets/PII leaks where availability is worse than the breakage. ## Account preconditions (fail before you start) - **2FA is mandatory** on PyPI for all maintainers. Without it you cannot create tokens or configure publishers — set it up first. - **Trusted Publishing needs no token at all**; if you're creating an API token "just in case", you probably don't need it (and it's a liability). Prefer a **project-scoped** token over account-wide if you must. - A **pending publisher** is per-project and consumed on first publish; register one per new package. ## General recovery loop ```bash scripts/diagnose-publish.sh <run-id> --repo OWNER/REPO # name the class # … apply the fix above … gh run rerun <run-id> --failed # re-run only failed jobs # (bump+re-tag instead only when the fix changed the artifact, e.g. VERSION_EXISTS) ``` -
trusted-publishing.md 4.7 KB
# Trusted Publishing (OIDC) — setup, claims, environments The 2026 default for publishing to PyPI from CI. No stored token; GitHub mints a short-lived OIDC token per run, PyPI exchanges it for an upload credential, and PEP 740 attestations sign the build provenance. ## How the exchange works 1. The publish job declares `permissions: id-token: write`. 2. GitHub mints an OIDC JWT whose **claims** describe the run: `repository`, `repository_owner`, `workflow_ref` (→ the workflow filename), `environment`, `ref`, `sub`. 3. `pypa/gh-action-pypi-publish` sends that JWT to PyPI's mint endpoint. 4. PyPI looks for a **Trusted Publisher** whose registered fields match the claims. Match → a short-lived API token scoped to that project. No match → `invalid-publisher`. The whole security model is "the claims must match a publisher you registered." Four fields must line up **exactly**: | Claim | Registered as | Common mismatch | |---|---|---| | `repository_owner` | Owner | org vs personal account | | `repository` | Repository name | renamed repo | | `workflow_ref` | Workflow name | the **filename** `publish.yml`, not the `name:` field | | `environment` | Environment name | job has no `environment:`, or a different one | ## Two registration paths ### Project publisher (project already exists on PyPI) PyPI → your project → **Settings → Publishing → Add a new publisher**. Use this for every release *after* the first. ### Pending publisher (FIRST publish — project doesn't exist yet) You cannot add a project publisher to a project that doesn't exist. Register a **pending publisher** at the account level **before** the first upload: > https://pypi.org/manage/account/publishing/ → "Add a pending publisher" > > - **PyPI Project Name** — the dist name (`pyproject.toml` `[project].name`) > - **Owner** / **Repository name** — GitHub `owner` / `repo` > - **Workflow name** — the filename, e.g. `publish.yml` > - **Environment name** — must equal the job's `environment:` (e.g. `pypi`) On the first successful publish it auto-converts to a normal project publisher. **This is the single most common first-release failure** — every release builds green but nothing reaches PyPI because this step was skipped. ## The environment gate (defense-in-depth) Put `environment: pypi` on the publish job. In **Settings → Environments → pypi** add **Required reviewers**. Now every release pauses for a human click, even if CI or the repo is compromised — OIDC proves *what* is publishing, the environment gate decides *whether*. The environment name is also one of the four matched claims, so it doubles as a publisher binding. ## TestPyPI {#testpypi} `test.pypi.org` is a **separate instance** — separate account, separate project namespace, **separate pending-publisher registration**. To dry-run: ```yaml - uses: pypa/gh-action-pypi-publish@<sha> # vX with: attestations: true repository-url: https://test.pypi.org/legacy/ ``` Register the pending publisher on TestPyPI (same four fields) and install from it with `pip install -i https://test.pypi.org/simple/ <pkg>`. TestPyPI prunes old releases and is not a reliability guarantee — use it for the metadata/flow rehearsal, not as a staging registry. ## Migrating an existing token-based workflow to OIDC 1. Add a Trusted Publisher (or pending publisher) for the project on PyPI. 2. In the publish job: add `permissions: id-token: write` (+ `contents: read`), and **remove** `password: ${{ secrets.PYPI_API_TOKEN }}` from the `gh-action-pypi-publish` step. Do not pass both — a token present alongside OIDC is what `publish-preflight.sh` flags. 3. Add `environment: pypi` and (recommended) required reviewers. 4. Delete the now-unused `PYPI_API_TOKEN` secret and revoke the token on PyPI. ## Verifying provenance (consumer side) Attestations are only worth emitting if someone can check them. As a consumer: - **PyPI project page** shows a "provenance"/attestation badge linking the release to the exact repo + workflow run that built it — a quick human check that a release came from the expected source. - **`gh attestation verify <artifact> --repo OWNER/REPO`** verifies a downloaded wheel/sdist against its signed provenance from the command line. - `pip` does **not** verify attestations at install time yet (2026) — provenance is currently an audit/forensic control, not an install-time gate. Don't assume `pip install` checks it. ## References - PyPI Trusted Publishers: https://docs.pypi.org/trusted-publishers/ - Troubleshooting (the `invalid-publisher` page): https://docs.pypi.org/trusted-publishers/troubleshooting/ - PEP 740 (attestations): https://peps.python.org/pep-0740/ - Action: https://github.com/pypa/gh-action-pypi-publish -
uv-publish.md 2.9 KB
# Local & manual publishing — uv, twine, build backends For a one-off release, a non-GitHub CI, or an emergency upload. **CI should still use OIDC** ([trusted-publishing.md](trusted-publishing.md)); this is the manual path. Never hand-roll the HTTP upload. ## Build first (backend-agnostic) `pyproject.toml` declares a build backend in `[build-system]`. Common ones: `hatchling`, `setuptools`, `flit-core`, `pdm-backend`, `maturin` (Rust ext), `scikit-build-core` (C/C++). The build command is the same regardless: ```bash uv build # → dist/<pkg>-<ver>.tar.gz (sdist) + …-py3-none-any.whl # or the PyPA-canonical: python -m build # needs: pip install build ``` Always validate before upload: ```bash python -m twine check dist/* # metadata/README sanity — catches the common reject ``` ## Publish with uv (preferred in 2026) ```bash # OIDC if running in a supported CI, otherwise prompts / uses configured creds uv publish --trusted-publishing automatic # token path — token via env, never inline (shell history leak) UV_PUBLISH_TOKEN="pypi-…" uv publish # TestPyPI uv publish --publish-url https://test.pypi.org/legacy/ ``` `uv publish` uploads whatever is in `dist/`. Build then publish; `uv` does not re-resolve or rebuild at publish time. ## Publish with twine (canonical fallback) ```bash python -m twine upload dist/* # PyPI python -m twine upload --repository testpypi dist/* # TestPyPI (see .pypirc) python -m twine upload --skip-existing dist/* # tolerate partial re-run ``` The GitHub Action wraps twine internally — so CI and local share the same upload engine and validation. ## `~/.pypirc` (token storage for the manual path) ```ini [distutils] index-servers = pypi testpypi [pypi] username = __token__ password = pypi-AgEI… # a PyPI API token; username is literally __token__ [testpypi] repository = https://test.pypi.org/legacy/ username = __token__ password = pypi-AgEN… # a SEPARATE TestPyPI token ``` `chmod 600 ~/.pypirc`. Prefer a **project-scoped** token (PyPI → project → Settings → API tokens) over an account-wide one. Rotate periodically; a token is a long-lived bearer credential — exactly what OIDC exists to eliminate. ## When to use which | Situation | Tool | |---|---| | CI/CD release | OIDC + `gh-action-pypi-publish` (not this file) | | Local one-off, uv project | `uv build` + `uv publish` | | Local one-off, non-uv | `python -m build` + `python -m twine upload` | | Metadata validation (any path) | `twine check` | | Dry run | TestPyPI via `--publish-url` / `--repository testpypi` | ## References - uv publish: https://docs.astral.sh/uv/guides/publish/ - twine: https://twine.readthedocs.io/ - build: https://build.pypa.io/ - Packaging guide: https://packaging.python.org/en/latest/tutorials/packaging-projects/
-
-
scripts
-
check-action-pins.py 6.2 KB
#!/usr/bin/env python3 """Verify a publish workflow's GitHub Action pins — SHA-pinned, commented, undrifted. Mutable @vN action tags get hijacked (tj-actions, 2025), so every `uses:` must be pinned to a full commit SHA with a trailing `# vX` comment. This is the §7 staleness verifier for the pypi-ops `assets/publish.yml` and any release workflow. Usage: check-action-pins.py [--offline | --live] [--json] <workflow.yml> Input: a GitHub Actions workflow file Output: stdout = per-action records (text, or --json envelope) Stderr: progress, the human summary Exit: 0 all good, 2 usage, 3 file not found, 7 github-unreachable (live only), 10 a problem found (unpinned/uncommented offline; SHA drift live) --offline (default) structural: every external `uses:` is SHA-pinned + `# vX` --live resolve each `# vX` tag via the GitHub API; flag when the pinned SHA no longer matches that tag (retag / stale pin). Honors GITHUB_TOKEN/GH_TOKEN for a higher rate limit. Examples: check-action-pins.py --offline .github/workflows/publish.yml check-action-pins.py --live .github/workflows/publish.yml check-action-pins.py --live --json publish.yml | jq '.data[] | select(.ok==false)' """ from __future__ import annotations import argparse import json import os import re import sys import urllib.error import urllib.request EXIT_OK, EXIT_USAGE, EXIT_NOTFOUND, EXIT_UNAVAIL, EXIT_FOUND = 0, 2, 3, 7, 10 # uses: owner/repo@<ref> with optional trailing # comment USES_RE = re.compile( r"""^\s*-?\s*uses:\s* (?P<action>[A-Za-z0-9._-]+/[A-Za-z0-9._/-]+) # owner/repo[/path] @(?P<ref>[^\s#]+) # ref (sha or tag) (?:\s*\#\s*(?P<comment>.+?))?\s*$ # optional # comment """, re.VERBOSE, ) SHA_RE = re.compile(r"^[0-9a-f]{40}$") TAGISH_RE = re.compile(r"\bv?\d+(?:\.\d+){0,2}\b") def parse_uses(path: str) -> list[dict]: out = [] with open(path, encoding="utf-8") as fh: for i, line in enumerate(fh, 1): m = USES_RE.match(line.rstrip("\n")) if not m: continue action = m.group("action") # local (./.github/...) and docker:// actions are not pinnable tags if action.startswith(".") or "://" in action: continue out.append( {"line": i, "action": action, "ref": m.group("ref"), "comment": (m.group("comment") or "").strip()} ) return out def gh_tag_sha(action: str, tag: str) -> tuple[str | None, str | None]: """Resolve owner/repo@tag -> commit sha via the GitHub API. Returns (sha, err).""" owner_repo = "/".join(action.split("/")[:2]) url = f"https://api.github.com/repos/{owner_repo}/commits/{tag}" req = urllib.request.Request(url, headers={ "Accept": "application/vnd.github+json", "User-Agent": "claude-mods-pypi-ops", }) tok = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") if tok: req.add_header("Authorization", f"Bearer {tok}") try: with urllib.request.urlopen(req, timeout=12) as resp: return json.load(resp).get("sha"), None except urllib.error.HTTPError as e: if e.code in (403, 429): return None, "rate-limited" if e.code == 404: return None, "tag-not-found" return None, f"http-{e.code}" except (urllib.error.URLError, TimeoutError, OSError): return None, "unreachable" def main() -> int: ap = argparse.ArgumentParser(add_help=True, description="Verify GitHub Action pins.") mode = ap.add_mutually_exclusive_group() mode.add_argument("--offline", action="store_true", help="structural checks only (default)") mode.add_argument("--live", action="store_true", help="resolve tags via the GitHub API") ap.add_argument("--json", action="store_true", help="emit the JSON envelope") ap.add_argument("workflow", help="path to a workflow .yml") args = ap.parse_args() if not os.path.isfile(args.workflow): print(f"ERROR: no such file: {args.workflow}", file=sys.stderr) return EXIT_NOTFOUND uses = parse_uses(args.workflow) live = args.live records, problem, unavailable = [], False, False if not uses: print("no external `uses:` actions found", file=sys.stderr) for u in uses: rec = {"action": u["action"], "ref": u["ref"], "line": u["line"], "ok": True, "note": ""} pinned = bool(SHA_RE.match(u["ref"])) comment_tag = TAGISH_RE.search(u["comment"]) if not pinned: rec["ok"], rec["note"] = False, f"not SHA-pinned (ref={u['ref']}); pin to a 40-char commit SHA" problem = True elif comment_tag is None: rec["ok"], rec["note"] = False, "SHA-pinned but missing a `# vX` version comment" problem = True elif live: tag = comment_tag.group(0) sha, err = gh_tag_sha(u["action"], tag) if err: unavailable = True rec["note"] = f"live check skipped ({err})" elif sha and sha != u["ref"]: rec["ok"], rec["note"] = False, f"DRIFT: pin != {tag} (tag now {sha[:12]}…); retag or refresh the pin" problem = True else: rec["note"] = f"matches {tag}" else: rec["note"] = "pinned + commented" records.append(rec) if not args.json: mark = "ok" if rec["ok"] else "XX" print(f" [{mark}] {rec['action']}@{rec['ref'][:12]} {rec['note']}", file=sys.stderr) if args.json: print(json.dumps({ "data": records, "meta": {"count": len(records), "mode": "live" if live else "offline", "ok": not problem, "schema": "claude-mods.pypi-ops.check-action-pins/v1"}, }, indent=2)) if problem: print("=== pin check FAILED ===", file=sys.stderr) return EXIT_FOUND if unavailable: print("=== structurally ok; some live checks were unavailable (advisory) ===", file=sys.stderr) return EXIT_UNAVAIL print("=== all action pins ok ===", file=sys.stderr) return EXIT_OK if __name__ == "__main__": sys.exit(main()) -
diagnose-publish.sh 6.7 KB
#!/usr/bin/env bash # Classify a failed PyPI publish — name the cause and the exact fix. # # Reads a GitHub Actions run log (by run-id via `gh`, or piped on stdin) and # matches it against the known PyPI-publish failure classes, so the agent acts on # a named cause instead of re-reading a 2,000-line log. Prints the OIDC claims the # run presented when the failure is a publisher mismatch. # # Usage: diagnose-publish.sh <run-id> | diagnose-publish.sh - [--json] [--repo OWNER/REPO] # Input: a numeric run-id (resolved with `gh run view --log-failed`), OR "-" to # read a log from stdin # Output: stdout = the classified finding (text, or --json envelope) # Stderr: progress, the human explanation # Exit: 0 no failure recognised (clean/unknown), 2 usage, 5 missing-dep (gh), # 7 gh/run unavailable, 10 a known failure class was identified # # Examples: # diagnose-publish.sh 27662335544 --repo 0xDarkMatter/flarecrawl # gh run view 27662335544 --log-failed | diagnose-publish.sh - # diagnose-publish.sh 27662335544 --json | jq '.data.fix' set -uo pipefail EXIT_OK=0; EXIT_USAGE=2; EXIT_MISSING_DEP=5; EXIT_UNAVAIL=7; EXIT_FOUND=10 JSON=0; REPO=""; SRC="" while [[ $# -gt 0 ]]; do case "$1" in --json) JSON=1 ;; --repo) REPO="${2:-}"; shift ;; -h|--help) sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; exit "$EXIT_OK" ;; -) SRC="stdin" ;; -*) echo "ERROR: unknown flag: $1 (try --help)" >&2; exit "$EXIT_USAGE" ;; *) SRC="$1" ;; esac shift done [[ -z "$SRC" ]] && { echo "ERROR: give a run-id or '-' for stdin (try --help)" >&2; exit "$EXIT_USAGE"; } # --- obtain the log ------------------------------------------------------------ LOG="" if [[ "$SRC" == "stdin" ]]; then LOG="$(cat)" else [[ "$SRC" =~ ^[0-9]+$ ]] || { echo "ERROR: run-id must be numeric (or '-' for stdin)" >&2; exit "$EXIT_USAGE"; } command -v gh >/dev/null 2>&1 || { echo "ERROR: gh required to fetch a run by id (or pipe a log with '-')" >&2; exit "$EXIT_MISSING_DEP"; } GHARGS=(run view "$SRC" --log-failed) [[ -n "$REPO" ]] && GHARGS+=(--repo "$REPO") LOG="$(gh "${GHARGS[@]}" 2>/dev/null)" || { echo "ERROR: could not fetch run $SRC (auth? wrong --repo? run still in progress?)" >&2; exit "$EXIT_UNAVAIL"; } fi [[ -n "$LOG" ]] || { echo "ERROR: empty log" >&2; exit "$EXIT_UNAVAIL"; } # --- classify ------------------------------------------------------------------ CLASS=""; SUMMARY=""; FIX=""; CLAIMS="" has() { grep -qiE "$1" <<<"$LOG"; } if has 'invalid-publisher|Trusted publishing exchange failure|no corresponding publisher|Publisher with matching claims was not found'; then CLASS="PENDING_PUBLISHER" SUMMARY="OIDC token was valid but PyPI has no Trusted Publisher matching the run's claims." FIX="Register a publisher at https://pypi.org/manage/account/publishing/ — for a FIRST publish use a *pending publisher* (the project doesn't exist yet). Match all claims: Owner, Repository, Workflow filename, Environment. Then re-run the failed job (gh run rerun <id> --failed)." # surface the claims block the action prints, for field-by-field comparison # strip the gh-log column prefix (".*\t"), the "* " bullet, and backticks CLAIMS="$(grep -iE '`?(sub|repository|repository_owner|workflow_ref|environment)`?[[:space:]]*:' <<<"$LOG" \ | sed -E 's/.*\t//; s/^[^A-Za-z`]*//; s/`//g; s/[[:space:]]+$//' | sort -u | head -8)" elif has 'File already exists|filename has already been used|already exists on|400 Bad Request.*[Rr]eupload'; then CLASS="VERSION_EXISTS" SUMMARY="The version is already on PyPI. Uploads are immutable — a version can never be re-published, even after deletion." FIX="Bump the version in pyproject.toml (and __init__/lock), commit, re-tag, push. Do NOT reuse the number. For a transient artifact mix-up, 'skip-existing: true' tolerates partial re-uploads but never replaces a file." elif has 'environment.*not allowed|environment.*is not defined|environment.*not found'; then CLASS="ENV_MISMATCH" SUMMARY="The publisher claim names an environment the publish job does not declare (or vice-versa)." FIX="Make the job's 'environment:' value equal the Trusted Publisher's 'Environment name' claim on PyPI exactly (e.g. both 'pypi'). They are matched verbatim." elif has '403 Forbidden|Invalid or non-existent authentication|isn.t allowed to upload|Non-user identities cannot create new projects'; then CLASS="AUTH_FORBIDDEN" SUMMARY="PyPI refused the credential (403). Either a token is wrong/expired, or an OIDC identity is uploading to a project it isn't trusted for." FIX="Prefer OIDC: confirm the Trusted Publisher exists for THIS project/workflow/environment. If using a token, rotate it and re-store the secret; ensure its scope includes this project. New project via OIDC needs a pending publisher first." elif has 'id-token.*write|OIDC.*not|aud claim|token request failed' && ! has 'invalid-publisher'; then CLASS="OIDC_CONFIG" SUMMARY="The OIDC token exchange itself failed (permissions or audience), before publisher matching." FIX="Ensure the publish job has 'permissions: id-token: write' and 'contents: read', and that it runs gh-action-pypi-publish with no 'password:'. Trusted publishing must not be mixed with a token." elif has 'twine.*check.*fail|InvalidDistribution|Metadata is missing|long_description'; then CLASS="METADATA_INVALID" SUMMARY="Artifact metadata failed validation (twine check) before upload." FIX="Fix the packaging metadata (README content-type, required fields, classifiers), rebuild, and re-run 'twine check dist/*' locally until clean." else echo "No recognised PyPI-publish failure pattern in the log." >&2 echo "Inspect manually — common non-publish causes: pip-audit CVE gate, build error, lock drift (uv sync --locked)." >&2 if [[ "$JSON" -eq 1 ]]; then echo '{"data":{"class":"UNKNOWN","summary":"no known publish failure pattern matched","fix":null},"meta":{"schema":"claude-mods.pypi-ops.diagnose-publish/v1","found":false}}' fi exit "$EXIT_OK" fi # --- output -------------------------------------------------------------------- if [[ "$JSON" -eq 1 ]]; then if command -v jq >/dev/null 2>&1; then jq -n --arg c "$CLASS" --arg s "$SUMMARY" --arg f "$FIX" --arg cl "$CLAIMS" \ '{data:{class:$c, summary:$s, fix:$f, presented_claims:($cl|if length>0 then split("\n") else [] end)}, meta:{schema:"claude-mods.pypi-ops.diagnose-publish/v1", found:true}}' else printf '{"data":{"class":"%s","found":true}}\n' "$CLASS" echo "WARN: jq missing; emitted minimal JSON" >&2 fi else printf '%s\n' "$CLASS" echo " cause: $SUMMARY" >&2 echo " fix: $FIX" >&2 [[ -n "$CLAIMS" ]] && { echo " claims the run presented (match these on PyPI):" >&2; sed 's/^/ /' <<<"$CLAIMS" >&2; } fi exit "$EXIT_FOUND" -
publish-preflight.sh 10.4 KB
#!/usr/bin/env bash # Pre-release readiness check for a PyPI package — catch mechanical failures before tagging. # # Verifies the things that silently break a release: version skew across # pyproject/__init__/lockfile, a version that is ALREADY on PyPI (uploads are # immutable — you cannot re-push 1.2.3), a git tag that disagrees with the # version, and a publish workflow that uses a stored token instead of OIDC. # Read-only; queries the public PyPI JSON API (no auth). # # Usage: publish-preflight.sh [--json] [--build] [-q] [<repo-root>] # Input: repo root as an optional positional (default "."); reads pyproject.toml, # the package __init__.py, uv.lock, and .github/workflows/*.yml. # --build additionally builds the dist and runs `twine check` (slower). # Output: stdout = per-check records (TSV: check<TAB>ok<TAB>detail, or --json envelope) # Stderr: headers, progress, the human summary # Exit: 0 ready (all checks pass/skip), 2 usage, 3 no pyproject, 5 missing-dep, # 7 pypi-unreachable, 10 not-ready (>=1 check failed) # # Examples: # publish-preflight.sh . # publish-preflight.sh --build . # also verify it builds + twine check # publish-preflight.sh --json ~/code/mypkg | jq '.data[] | select(.ok==false)' # publish-preflight.sh -q . && echo "ready to tag" set -uo pipefail EXIT_OK=0; EXIT_USAGE=2; EXIT_NOPROJ=3; EXIT_MISSING_DEP=5; EXIT_UNAVAIL=7; EXIT_NOTREADY=10 JSON=0; QUIET=0; BUILD=0; ROOT="." while [[ $# -gt 0 ]]; do case "$1" in --json) JSON=1 ;; --build) BUILD=1 ;; -q|--quiet) QUIET=1 ;; -h|--help) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit "$EXIT_OK" ;; -*) echo "ERROR: unknown flag: $1 (try --help)" >&2; exit "$EXIT_USAGE" ;; *) ROOT="$1" ;; esac shift done command -v curl >/dev/null 2>&1 || { echo "ERROR: curl required" >&2; exit "$EXIT_MISSING_DEP"; } HAS_JQ=0; command -v jq >/dev/null 2>&1 && HAS_JQ=1 if [[ "$JSON" -eq 1 && "$HAS_JQ" -eq 0 ]]; then echo '{"error":{"code":"MISSING_DEPENDENCY","message":"jq required for --json"}}' echo "ERROR: jq required for --json" >&2; exit "$EXIT_MISSING_DEP" fi ROOT="$(cd "$ROOT" 2>/dev/null && pwd)" || { echo "ERROR: no such directory" >&2; exit "$EXIT_NOPROJ"; } PYPROJECT="$ROOT/pyproject.toml" [[ -f "$PYPROJECT" ]] || { echo "ERROR: no pyproject.toml in $ROOT" >&2; exit "$EXIT_NOPROJ"; } emit() { [[ "$QUIET" -eq 1 ]] && return; printf '%s\n' "$*" >&2; } # --- minimal TOML field reads (regex; avoids a tomllib/py3.11 dependency) ------ toml_field() { # key -> first key = "value" at column 0 sed -n -E "s/^$1[[:space:]]*=[[:space:]]*[\"']([^\"']+)[\"'].*/\1/p" "$PYPROJECT" | head -1 } NAME="$(toml_field name)" VERSION="$(toml_field version)" DYNAMIC_VERSION=0 grep -Eq '^dynamic[[:space:]]*=.*version' "$PYPROJECT" && DYNAMIC_VERSION=1 # hatch-vcs / setuptools-scm: the version is the VCS tag, not a literal. Derive it # from a version tag on HEAD so dynamic-versioned projects get real checks too. VCS_VERSION=0 if [[ -z "$VERSION" && "$DYNAMIC_VERSION" -eq 1 ]] && command -v git >/dev/null 2>&1; then _ht="$(git -C "$ROOT" tag --points-at HEAD 2>/dev/null | grep -E '^v?[0-9]' | head -1)" [[ -n "$_ht" ]] && { VERSION="${_ht#v}"; VCS_VERSION=1; } fi # --- check accumulator --------------------------------------------------------- NOTREADY=0; UNAVAIL=0; RECORDS=() add() { # check ok(true/false/skip) detail local ok="$2" [[ "$ok" == "false" ]] && NOTREADY=1 RECORDS+=("$(printf '%s\t%s\t%s' "$1" "$ok" "$3")") if [[ "$QUIET" -ne 1 ]]; then local mark="·"; [[ "$ok" == "true" ]] && mark="ok"; [[ "$ok" == "false" ]] && mark="XX"; [[ "$ok" == "skip" ]] && mark="--" printf ' [%s] %-22s %s\n' "$mark" "$1" "$3" >&2 fi } emit "=== publish preflight: ${NAME:-?} ${VERSION:-?} ($ROOT) ===" # 1. name present [[ -n "$NAME" ]] && add "name" true "$NAME" || add "name" false "pyproject [project].name not found" # 2. version resolvable if [[ -n "$VERSION" && "$VCS_VERSION" -eq 1 ]]; then add "pyproject-version" true "$VERSION (dynamic, from VCS tag)" elif [[ -n "$VERSION" ]]; then add "pyproject-version" true "$VERSION" elif [[ "$DYNAMIC_VERSION" -eq 1 ]]; then add "pyproject-version" skip "dynamic version, no tag on HEAD — tag first, then re-run" else add "pyproject-version" false "no [project].version and not declared dynamic" fi # 3. __init__ __version__ agreement (find the literal assignment, skip vendored trees) INIT_FILE="$(grep -rIlE --include='__init__.py' '^__version__[[:space:]]*=' "$ROOT/src" "$ROOT" 2>/dev/null \ | grep -vE '/(\.venv|venv|site-packages|\.tox|build|dist|node_modules)/' | head -1)" if [[ -n "$INIT_FILE" ]]; then IVER="$(sed -n -E "s/^__version__[[:space:]]*=[[:space:]]*[\"']([^\"']+)[\"'].*/\1/p" "$INIT_FILE" | head -1)" REF="${VERSION:-}" if [[ "$DYNAMIC_VERSION" -eq 1 && -z "$REF" ]]; then REF="$IVER"; fi if [[ -z "$IVER" ]]; then add "init-version" skip "no __version__ literal in $(basename "$INIT_FILE")" elif [[ "$IVER" == "$REF" ]]; then add "init-version" true "$IVER matches" else add "init-version" false "__version__=$IVER != pyproject=$REF" fi else add "init-version" skip "no __init__.py with __version__ found" fi # 4. lockfile self-version (uv.lock) agreement EFFVER="${VERSION:-${IVER:-}}" if [[ -f "$ROOT/uv.lock" && -n "$NAME" && -n "$EFFVER" ]]; then LVER="$(awk -v n="\"$NAME\"" ' $1=="name" && $3==n {f=1; next} f==1 && $1=="version" {gsub(/"/,"",$3); print $3; exit}' \ FS=' ' "$ROOT/uv.lock" 2>/dev/null)" if [[ -z "$LVER" ]]; then add "lock-version" skip "package not pinned in uv.lock" elif [[ "$LVER" == "$EFFVER" ]]; then add "lock-version" true "$LVER matches" else add "lock-version" false "uv.lock has $LVER != $EFFVER (run: uv lock)" fi else add "lock-version" skip "no uv.lock" fi # 5. version not already on PyPI (immutable); flag a brand-new project (first publish) if [[ -n "$NAME" && -n "$EFFVER" ]]; then code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 12 \ "https://pypi.org/pypi/${NAME}/${EFFVER}/json" 2>/dev/null)" case "$code" in 404) add "not-on-pypi" true "${EFFVER} is free to publish" # distinguish "new version of an existing project" from "first ever publish" pcode="$(curl -s -o /dev/null -w '%{http_code}' --max-time 12 \ "https://pypi.org/pypi/${NAME}/json" 2>/dev/null)" if [[ "$pcode" == "404" ]]; then add "first-publish" skip "NEW project — register a PENDING publisher on PyPI BEFORE tagging, or the publish fails 'invalid-publisher' (SKILL.md)" fi ;; 200) add "not-on-pypi" false "${NAME}==${EFFVER} ALREADY on PyPI — bump the version" ;; 000) UNAVAIL=1; add "not-on-pypi" skip "PyPI unreachable (advisory)" ;; *) UNAVAIL=1; add "not-on-pypi" skip "PyPI returned HTTP $code (advisory)" ;; esac else add "not-on-pypi" skip "need name+version to query" fi # 6. git tag agreement (if HEAD is tagged, or a v<version> tag exists) if command -v git >/dev/null 2>&1 && git -C "$ROOT" rev-parse --git-dir >/dev/null 2>&1; then HEADTAG="$(git -C "$ROOT" tag --points-at HEAD 2>/dev/null | grep -E '^v?[0-9]' | head -1)" if [[ -n "$HEADTAG" && -n "$EFFVER" ]]; then if [[ "$HEADTAG" == "v$EFFVER" || "$HEADTAG" == "$EFFVER" ]]; then add "git-tag" true "HEAD tagged $HEADTAG" else add "git-tag" false "HEAD tag $HEADTAG != v$EFFVER" fi else add "git-tag" skip "HEAD not tagged (tag after preflight passes)" fi else add "git-tag" skip "not a git repo" fi # 7. publish workflow uses OIDC, not a stored token WF="" for f in "$ROOT"/.github/workflows/*.yml "$ROOT"/.github/workflows/*.yaml; do [[ -f "$f" ]] || continue if grep -Eq 'pypi|gh-action-pypi-publish|twine upload|uv publish' "$f"; then WF="$f"; break; fi done if [[ -n "$WF" ]]; then if grep -Eq 'id-token:[[:space:]]*write' "$WF"; then if grep -Eq '^[[:space:]]*password:|PYPI_API_TOKEN|PYPI_TOKEN' "$WF"; then add "workflow-oidc" false "$(basename "$WF") has OIDC but ALSO a stored token — drop the token" else add "workflow-oidc" true "$(basename "$WF") uses OIDC trusted publishing" fi elif grep -Eq '^[[:space:]]*password:|PYPI_API_TOKEN|PYPI_TOKEN' "$WF"; then add "workflow-oidc" false "$(basename "$WF") uses a stored token — migrate to OIDC (id-token: write)" else add "workflow-oidc" skip "$(basename "$WF") found but no OIDC/token marker recognised" fi else add "workflow-oidc" skip "no PyPI publish workflow found under .github/workflows/" fi # 8. (opt-in) the package actually builds + passes twine check if [[ "$BUILD" -eq 1 ]]; then BUILDER="" command -v uv >/dev/null 2>&1 && BUILDER="uv" if [[ -z "$BUILDER" ]] && command -v python >/dev/null 2>&1 && python -c "import build" >/dev/null 2>&1; then BUILDER="build"; fi if [[ -z "$BUILDER" ]]; then add "build" skip "no uv or python-build (pip install build) — skipping build check" else BOUT="$(mktemp -d)"; bok=0 if [[ "$BUILDER" == "uv" ]]; then ( cd "$ROOT" && uv build --out-dir "$BOUT" ) >/dev/null 2>&1 && bok=1 else ( cd "$ROOT" && python -m build --outdir "$BOUT" ) >/dev/null 2>&1 && bok=1 fi if [[ "$bok" -ne 1 ]]; then add "build" false "build failed — run '$BUILDER build' directly to see why" else TW="" command -v twine >/dev/null 2>&1 && TW="twine" [[ -z "$TW" ]] && command -v uvx >/dev/null 2>&1 && TW="uvx twine" if [[ -n "$TW" ]]; then if $TW check "$BOUT"/* >/dev/null 2>&1; then add "build" true "builds + twine check ok" else add "build" false "built, but twine check failed (fix packaging metadata)"; fi else add "build" true "builds ok (no twine to validate metadata)" fi fi rm -rf "$BOUT" fi fi # --- output -------------------------------------------------------------------- if [[ "$JSON" -eq 1 ]]; then printf '%s\n' "${RECORDS[@]}" | jq -R 'split("\t") | {check:.[0], ok:(.[1]=="true"), status:.[1], detail:.[2]}' \ | jq -s --arg name "$NAME" --arg version "$EFFVER" \ '{data: ., meta:{name:$name, version:$version, count:length, ready:(any(.[]; .status=="false")|not), schema:"claude-mods.pypi-ops.publish-preflight/v1"}}' else printf '%s\n' "${RECORDS[@]}" fi if [[ "$NOTREADY" -eq 1 ]]; then emit "=== NOT READY — resolve the [XX] checks before tagging ===" exit "$EXIT_NOTREADY" fi [[ "$UNAVAIL" -eq 1 ]] && { emit "=== checks passed; PyPI lookup was advisory (unreachable) ==="; exit "$EXIT_UNAVAIL"; } emit "=== READY to tag $EFFVER ===" exit "$EXIT_OK"
-
-
tests
-
run.sh 7.1 KB
#!/usr/bin/env bash # Self-test for pypi-ops scripts. # # Offline-deterministic: builds throwaway fixtures, asserts documented exit codes # and key output, then cleans up. The one network-touching check # (publish-preflight's PyPI lookup) is asserted only where a *deterministic* # failure dominates the exit code; the all-pass fixture tolerates {0 ok, 7 # pypi-unreachable}. Resolves paths relative to itself so it works in-repo and # once installed to ~/.claude/skills/pypi-ops/. # # Usage: bash tests/run.sh # Exit: 0 all pass, 1 one or more failures set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SKILL="$(dirname "$HERE")" SCRIPTS="$SKILL/scripts" PYTHON="" for c in python python3 py; do if command -v "$c" >/dev/null 2>&1 && "$c" -c "" >/dev/null 2>&1; then PYTHON="$c"; break; fi done [[ -z "$PYTHON" ]] && { echo "no working python found" >&2; exit 1; } SB="$(mktemp -d)"; trap 'rm -rf "$SB"' EXIT SHA="de0fac2e4500dabe0009e67214ff5f5447ce83dd" # a real 40-hex sha (actions/checkout v6.0.2) PASS=0; FAIL=0 ok() { PASS=$((PASS+1)); printf ' PASS %s\n' "$1"; } no() { FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; } expect_exit() { [[ "$2" == "$3" ]] && ok "$1 (exit $3)" || no "$1 (want $2 got $3)"; } expect_in() { case " $2 " in *" $3 "*) ok "$1 (exit $3)";; *) no "$1 (want one of [$2] got $3)";; esac; } expect_has() { case "$3" in *"$2"*) ok "$1";; *) no "$1 (missing '$2')";; esac; } echo "=== pypi-ops self-test ===" # ── publish-preflight.sh ─────────────────────────────────────────────────── echo "-- publish-preflight.sh --" PF="$SCRIPTS/publish-preflight.sh" bash "$PF" --help >/dev/null 2>&1; expect_exit "--help" 0 $? bash "$PF" --bogus >/dev/null 2>&1; expect_exit "bad flag -> 2" 2 $? # --build is wired (the actual build is network/backend-dependent, not exercised offline) # Captured, not `--help | grep -q`: under `set -o pipefail` grep -q's early exit # SIGPIPEs the producer (141) and flakes the assert even on a match. See # skills/fleet-ops/tests/run.sh for the pattern. pf_help="$(bash "$PF" --help 2>&1)" expect_has "--help documents --build" "--build" "$pf_help" mkdir -p "$SB/empty" bash "$PF" "$SB/empty" >/dev/null 2>&1; expect_exit "no pyproject -> 3" 3 $? # clean fixture: versions agree, workflow uses OIDC, bogus pkg name (offline-safe) mk_repo() { # dir pyproject_version init_version workflow_snippet local d="$1" mkdir -p "$d/src/zzzpkg" "$d/.github/workflows" cat > "$d/pyproject.toml" <<EOF [project] name = "zzz-claudemods-pypiops-fixture-xyz" version = "$2" EOF printf '__version__ = "%s"\n' "$3" > "$d/src/zzzpkg/__init__.py" printf '%s\n' "$4" > "$d/.github/workflows/publish.yml" } OIDC_WF=$'on:\n push:\n tags: ["v*"]\njobs:\n publish:\n environment: pypi\n permissions:\n id-token: write\n steps:\n - uses: pypa/gh-action-pypi-publish@'"$SHA"$' # v1.14.0' TOKEN_WF=$'jobs:\n publish:\n steps:\n - uses: pypa/gh-action-pypi-publish@'"$SHA"$'\n with:\n password: ${{ secrets.PYPI_API_TOKEN }}' mk_repo "$SB/clean" "1.2.3" "1.2.3" "$OIDC_WF" out="$(bash "$PF" "$SB/clean" 2>&1)"; rc=$? expect_in "clean fixture -> {0,7}" "0 7" "$rc" expect_has "clean: init-version ok" "init-version" "$out" expect_has "clean: workflow OIDC recognised" "OIDC" "$out" # version-skew fixture: deterministic failure regardless of network mk_repo "$SB/skew" "1.2.3" "1.2.4" "$OIDC_WF" out="$(bash "$PF" "$SB/skew" 2>&1)"; rc=$? expect_exit "version skew -> 10" 10 "$rc" expect_has "skew names the mismatch" "1.2.4" "$out" # token-in-workflow fixture: workflow-oidc must fail -> 10 mk_repo "$SB/token" "1.2.3" "1.2.3" "$TOKEN_WF" out="$(bash "$PF" "$SB/token" 2>&1)"; rc=$? expect_exit "stored token -> 10" 10 "$rc" expect_has "token: flags stored token" "token" "$out" # --json envelope shape (skew fixture, deterministic) if command -v jq >/dev/null 2>&1; then out="$(bash "$PF" --json "$SB/skew" 2>/dev/null)" expect_has "json envelope schema" "publish-preflight/v1" "$out" else echo " SKIP --json shape (jq not installed)" fi # ── diagnose-publish.sh ──────────────────────────────────────────────────── echo "-- diagnose-publish.sh --" DG="$SCRIPTS/diagnose-publish.sh" bash "$DG" --help >/dev/null 2>&1; expect_exit "--help" 0 $? bash "$DG" >/dev/null 2>&1; expect_exit "no arg -> 2" 2 $? bash "$DG" --bogus >/dev/null 2>&1; expect_exit "bad flag -> 2" 2 $? INVALID_LOG=$'Trusted publishing exchange failure:\n* `invalid-publisher`: valid token, but no corresponding publisher\n* `repository`: `0xDarkMatter/flarecrawl`\n* `environment`: `pypi`' out="$(printf '%s' "$INVALID_LOG" | bash "$DG" - 2>&1)"; rc=$? expect_exit "invalid-publisher -> 10" 10 "$rc" expect_has "classes PENDING_PUBLISHER" "PENDING_PUBLISHER" "$out" expect_has "surfaces presented claims" "0xDarkMatter/flarecrawl" "$out" out="$(printf 'ERROR: File already exists (pkg-1.2.3.tar.gz)' | bash "$DG" - 2>&1)"; rc=$? expect_exit "file-exists -> 10" 10 "$rc" expect_has "classes VERSION_EXISTS" "VERSION_EXISTS" "$out" out="$(printf 'all green, nothing wrong here' | bash "$DG" - 2>&1)"; rc=$? expect_exit "no pattern -> 0" 0 "$rc" if command -v jq >/dev/null 2>&1; then out="$(printf '%s' "$INVALID_LOG" | bash "$DG" - --json 2>/dev/null)" expect_has "json schema present" "diagnose-publish/v1" "$out" expect_has "json carries class" "PENDING_PUBLISHER" "$out" else echo " SKIP --json shape (jq not installed)" fi # ── check-action-pins.py ─────────────────────────────────────────────────── echo "-- check-action-pins.py --" CP="$SCRIPTS/check-action-pins.py" "$PYTHON" "$CP" --help >/dev/null 2>&1; expect_exit "--help" 0 $? "$PYTHON" "$CP" "$SB/no-such.yml" >/dev/null 2>&1; expect_exit "missing file -> 3" 3 $? printf 'jobs:\n x:\n steps:\n - uses: actions/checkout@%s # v6.0.2\n' "$SHA" > "$SB/good.yml" "$PYTHON" "$CP" --offline "$SB/good.yml" >/dev/null 2>&1; expect_exit "pinned+commented -> 0" 0 $? printf 'jobs:\n x:\n steps:\n - uses: actions/checkout@v4\n' > "$SB/tag.yml" "$PYTHON" "$CP" --offline "$SB/tag.yml" >/dev/null 2>&1; expect_exit "tag not sha -> 10" 10 $? printf 'jobs:\n x:\n steps:\n - uses: actions/checkout@%s\n' "$SHA" > "$SB/nocomment.yml" "$PYTHON" "$CP" --offline "$SB/nocomment.yml" >/dev/null 2>&1; expect_exit "sha w/o comment -> 10" 10 $? out="$("$PYTHON" "$CP" --offline --json "$SB/good.yml" 2>/dev/null)" expect_has "json schema present" "check-action-pins/v1" "$out" # dogfood: the shipped asset must pass our own offline pin check "$PYTHON" "$CP" --offline "$SKILL/assets/publish.yml" >/dev/null 2>&1 expect_exit "shipped publish.yml pins ok -> 0" 0 $? # ── summary ──────────────────────────────────────────────────────────────── echo "=== $PASS passed, $FAIL failed ===" [[ "$FAIL" -eq 0 ]] || exit 1
-
-
SKILL.md 12.4 KB
--- name: pypi-ops description: "Publish Python packages to PyPI via OIDC Trusted Publishing (PEP 740 attestations, gh-action-pypi-publish) instead of stored tokens. Use for: invalid-publisher errors, pending-publisher 404s, uv publish/twine, TestPyPI, environment approval gates, and rotating/auditing publish tokens." license: MIT compatibility: "Python 3.8+ packaging; GitHub Actions for the OIDC flow; uv or twine for local publish" allowed-tools: "Read Write Edit Bash Glob Grep WebFetch" metadata: author: claude-mods related-skills: "supply-chain-defense, github-ops, git-ops, ci-cd-ops, python-env" --- # PyPI Operations Publish Python packages to PyPI on the **2026 best-practice path: OIDC Trusted Publishing with signed PEP 740 attestations**, no long-lived token to leak. This skill owns the *publish* layer (the registry handshake, the first-publish gotchas, the recovery playbook). General GitHub Actions syntax is `ci-cd-ops`; the install-side worm defense is `supply-chain-defense`; `gh`/release-page mechanics are `github-ops`. ## Where this fits — the release pipeline A release spans several skills; pypi-ops owns the **registry** step. Chain them: 1. **Vet dependencies** before cutting a release — `supply-chain-defense` (cooldown + behavioural scan). The build runs dependency code *before* it touches your publish credential, so a poisoned build dep can steal the token. 2. **Preflight** — `scripts/publish-preflight.sh --build .` (this skill). 3. **Bump → tag → push** — `git-ops` (its push-gate scans for secrets / forbidden files before the tag goes up). 4. **CI publishes** via OIDC — this skill's `assets/publish.yml`; you approve at the `pypi` environment gate. 5. **Release page** (optional, GitHub) — `github-ops`, human-reviewed notes. ## The one decision: OIDC vs API token **Default to OIDC Trusted Publishing.** Reach for a token only when OIDC is impossible (publishing from a non-supported CI, or a one-off local push). | | **Trusted Publishing (OIDC)** ← default | **API token** | |---|---|---| | Secret stored | None — short-lived OIDC token minted per run | Long-lived `pypi-…` token in a secret | | Leak/phish blast radius | None to steal | Full publish rights until rotated | | Provenance | PEP 740 attestations (signed, verifiable) | None by default | | Setup | One-time publisher registration on PyPI | Generate token + store secret | | Best for | **All CI/CD releases** | Legacy CI, emergency local upload | If a repo currently uses a token, migrating to OIDC is strictly an upgrade — see [references/trusted-publishing.md](references/trusted-publishing.md). ## The #1 gotcha: first publish needs a *pending* publisher A Trusted Publisher is normally configured **under the project's settings** on PyPI — but on the **first ever publish the project doesn't exist yet**, so there's nothing to configure it under. The fix is a **pending publisher**, registered at the account level *before* the first upload. Symptom (the exact failure this skill exists to kill): ``` Trusted publishing exchange failure: * invalid-publisher: valid token, but no corresponding publisher (Publisher with matching claims was not found) ``` The OIDC token was valid; PyPI just has no publisher matching the claims. Fix: > **PyPI → https://pypi.org/manage/account/publishing/ → Add a pending publisher** > > | Field | Value | > |---|---| > | PyPI Project Name | the dist name from `pyproject.toml` `[project].name` | > | Owner | GitHub org/user | > | Repository name | repo name | > | Workflow name | the **filename**, e.g. `publish.yml` (not the `name:`) | > | Environment name | must equal the job's `environment:` (e.g. `pypi`) | All four claims must match the run's OIDC token exactly. After the first successful publish, the pending publisher auto-converts to a normal project publisher — no further action. Run `diagnose-publish.sh` on a failed run to read the exact claims it presented and compare them field-by-field. > This is the most common silent-failure mode: a package's `publish.yml` looks > perfect and every release builds green, yet nothing ever reaches PyPI because > the publisher was never registered. Check it **first**. ## Recommended workflow (copy [assets/publish.yml](assets/publish.yml)) The shipped template is hardened to the patterns below — adapt the marked points and drop it in `.github/workflows/`. Non-negotiables it encodes: - **`on: push: tags: ['v*']`** — release on a version tag, never on every push. - **OIDC, no token:** the `publish` job has `permissions: id-token: write` and `pypa/gh-action-pypi-publish` with `attestations: true`. No `password:`/token. - **`environment: pypi`** on the publish job → a human approves every release (defense-in-depth: even a compromised repo can't auto-ship). - **Build/publish split:** a `build` job (no elevated perms) produces + uploads the `dist` artifact; `publish` downloads it. Least privilege per job. - **`uv sync --locked` + `pip-audit`:** the release is built against the committed, hash-verified lockfile and blocked if a dep has a known CVE. - **`twine check` / metadata validation** before upload. - **SHA-pinned actions** with a trailing `# vX` comment (mutable tags get hijacked — see `check-action-pins.py`). - **Verify-on-PyPI** tail job — polls the JSON API so a *silent* publish failure (accepted-but-not-live, CDN lag) surfaces loudly instead of looking fine. ## Supply-chain hardening — the publisher side Stealing your publish credential lets an attacker ship malware to everyone who installs you — so the publish path is the surface the 2026 worm campaign (Mini Shai-Hulud) targets, minting PyPI/npm tokens from **stale OIDC trust and orphaned workflows**. The template above isn't just convention; each choice is a defense: | Control | Defends against | |---|---| | OIDC, no stored token | Credential theft/phishing — there is no long-lived secret to steal | | PEP 740 attestations | Tampered artifacts — provenance is signed and verifiable | | `environment: pypi` + reviewers | A compromised repo/CI auto-shipping — a human still gates the release | | `pip-audit` gate | A knowingly-vulnerable dependency reaching the release build | | SHA-pinned actions (`check-action-pins.py`) | Action-tag hijacks (tj-actions, 2025) repointing `@vN` to a malicious commit | | `permissions: {}` + per-job least privilege | A poisoned build step escalating beyond read | | `uv sync --locked` | Build-time dependency injection / silent re-resolution | Then audit the **trust** itself, not just the workflow: - **Revoke stale Trusted Publishers / OIDC federation** you no longer use — an orphaned publisher bound to a deletable workflow is the Mini Shai-Hulud entry point. Review PyPI → project → *Publishing* periodically. - **If a token is in play, rotate it** (project-scoped, short-lived) — better, migrate to OIDC and delete it. See [trusted-publishing.md](references/trusted-publishing.md). - **Vet build dependencies before a release**, not after — a poisoned `uv sync` step runs before your OIDC token is even minted. Division of labour: **pypi-ops owns publisher hardening**; `supply-chain-defense` owns the install side and ships `integrity-audit.sh` (hunts `pull_request_target` + OIDC misconfig and worm persistence) — run it on any repo that publishes, and gate dependency bumps through its cooldown + behavioural scan. ## Cutting a release — preflight then tag Before tagging, run the preflight so a release never fails on something mechanical (version skew, dirty lock, missing publisher config): ```bash scripts/publish-preflight.sh . # human summary; exit 10 = not ready scripts/publish-preflight.sh --build . # also build + twine-check the dist scripts/publish-preflight.sh --json . | jq '.data[] | select(.ok==false)' ``` It checks: `pyproject` version == `__init__.__version__`, the version is **not already on PyPI** (uploads are immutable — you cannot re-push `1.2.3`), the lockfile self-version matches, a tag (if present) matches the version, and the publish workflow uses OIDC (flags a stored token). `--build` additionally verifies the package actually builds and passes `twine check`. Dynamic-versioned projects (hatch-vcs / setuptools-scm) are read from the HEAD tag. Green → bump, commit, tag, push the tag; CI builds, waits at the `pypi` environment gate, you approve. ## When a publish fails — classify, don't guess ```bash scripts/diagnose-publish.sh <run-id> # reads gh run log, names the cause + fix gh run view <run-id> --log-failed | scripts/diagnose-publish.sh - # or pipe a log ``` The high-frequency failure classes and their fixes: | Symptom | Cause | Fix | |---|---|---| | `invalid-publisher` / claims not found | No (pending) publisher on PyPI | Register the pending publisher (above) | | `File already exists` / 400 on upload | Version already on PyPI (immutable) | Bump the version; never reuse — see [recovery](references/recovery-playbook.md) | | Job stuck "Waiting" | `environment: pypi` needs approval | Approve the deployment in the run's UI | | `environment … not found` | Publisher claim names an env the job lacks | Make `environment:` and the publisher's Environment match | | Built green, not on PyPI | Silent accept / no verify step | Add the verify-on-PyPI job; re-run | | `non-OIDC`/token rejected | Token wrong/expired, or OIDC misread as token | Prefer OIDC; if token, rotate + re-store | Full catalogue with the underlying mechanics: [references/recovery-playbook.md](references/recovery-playbook.md). ## Local & manual publishing (uv / twine) For a one-off or a non-CI environment. **Prefer `uv` in 2026** (faster, native): ```bash uv build # sdist + wheel into dist/ uv publish --trusted-publishing automatic # OIDC if in supported CI, else prompts # token path (store in ~/.pypirc or env, never inline on the CLI history): UV_PUBLISH_TOKEN="pypi-…" uv publish ``` `twine` remains the canonical fallback and the metadata validator (the GitHub Action wraps it internally): ```bash python -m twine check dist/* # ALWAYS run before any upload python -m twine upload dist/* # token from ~/.pypirc; legacy path ``` Never hand-roll the HTTP upload. Details + `~/.pypirc` shape: [references/uv-publish.md](references/uv-publish.md). ## Dry-run on TestPyPI first For a brand-new package or a risky metadata change, publish to **test.pypi.org** first — it has its own separate accounts *and its own pending-publisher registration*. Point the action at `repository-url: https://test.pypi.org/legacy/` and register the pending publisher on TestPyPI. See [references/trusted-publishing.md](references/trusted-publishing.md#testpypi). ## Keeping the workflow from rotting The pinned action SHAs and `pypa/gh-action-pypi-publish` major drift over time. The verifier flags it before a release does: ```bash scripts/check-action-pins.py --offline .github/workflows/publish.yml # structure: all pinned + commented scripts/check-action-pins.py --live .github/workflows/publish.yml # resolve tags → flag SHA drift ``` `--offline` is the PR gate (every `uses:` is SHA-pinned with a `# vX` comment); `--live` runs scheduled (resolves each pin against GitHub and exits 10 on drift, 7 if GitHub is unreachable — advisory, never a flaky block). ## Publishing many packages (a fleet) When several repos publish the same way, don't copy `publish.yml` N times — each copy drifts its own SHA pins. Hoist the publish job into a **reusable workflow** (`on: workflow_call`) in one repo, and have each package's tiny caller pass its dist name. OIDC still works: the *caller's* `workflow_ref` is what PyPI matches, so **register each package's pending publisher against the caller** filename (e.g. `release.yml`), not the shared one. One place to refresh pins (`check-action-pins.py` on the reusable workflow); one approval gate definition; per-package publishers. See [references/trusted-publishing.md](references/trusted-publishing.md) for the claim that must match. ## Reference files | File | Load when | |---|---| | [references/trusted-publishing.md](references/trusted-publishing.md) | Setting up OIDC, pending vs project publisher, OIDC claim semantics, environments, TestPyPI, token→OIDC migration | | [references/recovery-playbook.md](references/recovery-playbook.md) | A publish failed and you need the full failure-class catalogue + mechanics | | [references/uv-publish.md](references/uv-publish.md) | Local/manual publishing, `uv build`/`uv publish`, `twine`, `~/.pypirc`, build backends |
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.