Last verified: 2026-07-14 — refresh when the blessed container tags change.

# Container Workflow

Concrete `docker run` invocations for the two blessed images, plus the bare-pip fallback.

## NGC PyTorch container

General-purpose training/inference base:

```bash
docker run --runtime=nvidia --gpus all -it --rm \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$(pwd)/finetuning:/workspace/finetuning" \
  nvcr.io/nvidia/pytorch:25.09-py3
```

**Tag guidance:** `25.09-py3` is the tag actually verified working on this hardware (torch `2.9.0a0+50eac811a6.nv25.09`, CUDA 13.0 baked in, matches the ABI Rule's expectations — no functional delta observed for the packages exercised in fine-tuning workflows). Treat `SKILL.md`'s mention of a newer blessed tag as guidance to pull when locally available, not a hard requirement — if the cited newer tag isn't locally cached and pulling isn't practical, fall back to the newest available `25.x` tag and record the gap in the run's notes rather than blocking on it.

- `--runtime=nvidia --gpus all` gives the container access to the GB10 GPU; without it, PyTorch inside the container will report no CUDA device even though the host sees one fine.
- `--ipc=host` and the `ulimit` flags avoid shared-memory starvation for PyTorch's DataLoader workers.
- Mount the repo's `finetuning/` run directory so checkpoints and logs land on the host filesystem, not inside the ephemeral container layer — the `--rm` flag deletes the container (and anything not mounted out) on exit.

## Unsloth container

For Unsloth-centric fine-tuning runs, prefer the purpose-built image over the generic NGC one — it ships the pinned Triton/xformers/transformers combination already validated for this hardware.

**`dgxspark-latest` is a moving tag, unlike the NGC image's dated `25.09-py3` tag above.** Don't run it directly as the invocation you'll rely on for a real run — resolve and pin its digest first, then run by digest:

```bash
# 1. Discovery step: pull the moving tag and confirm it starts.
docker pull unsloth/unsloth:dgxspark-latest

# 2. Resolve the tag to its current digest.
docker inspect --format='{{index .RepoDigests 0}}' unsloth/unsloth:dgxspark-latest
# -> unsloth/unsloth@sha256:<resolved digest>

# 3. Run by digest — this is the reproducible invocation.
docker run --runtime=nvidia --gpus all -it --rm \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$(pwd)/finetuning:/workspace/finetuning" \
  unsloth/unsloth@sha256:<resolved digest>
```

Substitute the pinned `@sha256:...` digest for the tag in CI or any pipeline where reproducibility matters — a run recorded against `dgxspark-latest` by tag alone cannot be reproduced later if the tag has moved on. Re-resolve and re-pin the digest whenever picking up a new blessed release (see "Pull vs rebuild" below).

Same flag rationale as the NGC invocation above. Prefer this over rebuilding a custom Unsloth image from the NGC base.

## Pull vs rebuild

Pull a fresh tag when: a new blessed release is announced, or you're chasing a bug that a recent tag's changelog says it fixes.

Rebuild locally (starting `FROM` one of the two images above) when: you need an extra system package or Python dependency layered in for a specific project, and that dependency doesn't conflict with the pinned training stack. Don't rebuild to "upgrade" a component that the base image already pins — that reintroduces the version-matrix problem the container exists to avoid.

## Bare-pip escape hatch

If a container genuinely doesn't fit (see `SKILL.md`'s Container-First Rule), isolate the environment with `uv` rather than the system Python, and follow the NVIDIA playbook install sequence from `SKILL.md` inside it.

Caveat: if `uv` insists on a dependency version that conflicts with what the playbook pins (a common outcome given how young the aarch64/CUDA-13 wheel ecosystem is), use `uv pip install --override` to force the pinned versions through rather than letting the resolver silently substitute an incompatible build. Verify the result with the Verification Commands section of `SKILL.md` before trusting the environment.

Last verified: 2026-07-14 — refresh when CUDA, PyTorch, or Unsloth major versions change.

# Spark Stack Matrix

Full component-by-component status for the ML training/inference stack on DGX Spark (GB10, SM121, aarch64, CUDA 13). This is the detail table behind the "Component Quick Table" in `SKILL.md`.

| Component | Status | Notes |
|---|---|---|
| PyTorch (cu130, aarch64) | ✅ | Official wheels at `download.pytorch.org/whl/cu130`. Matches the system CUDA 13 ABI — see the ABI Rule in `SKILL.md`. |
| bitsandbytes | ✅ | 0.48+ works out of the box. |
| Triton | ✅ (with env var) | Needs `TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas` set, or kernel compilation fails to find `ptxas`. |
| flash-attn | ❌ skip | No sm_121 kernels shipped or buildable yet. PyTorch's SDPA backend is faster on this hardware anyway — don't spend time chasing a flash-attn build. |
| xformers | source build only | No prebuilt aarch64/SM121 wheel. Build with `TORCH_CUDA_ARCH_LIST=12.1` set, or the build targets the wrong architecture and either fails or silently produces non-functional kernels. |
| vLLM | nightly wheels only | Use `wheels.vllm.ai/nightly/cu130`. The SM121 fix landed in the nightly channel around 2026-06; stable/release wheels predate it. |
| TransformerEngine / NVFP4 training | container-only | Not practical via bare pip; use the NGC PyTorch container. `NVFP4BlockScaling` targets SM100 — treat SM121 support as caveated, not guaranteed. |
| Unsloth | ✅ (container preferred) | Official Docker image `unsloth/unsloth:dgxspark-latest` (a moving tag — resolve and pin its digest for reproducible/CI use, see `references/container-workflow.md`), or the NVIDIA playbook pip sequence (see `SKILL.md`). Bare pip installs have hit torchcodec and GPU-detection gotchas. |
| Axolotl / TRL / PEFT | ✅ | Standard install, no special handling needed. |
| LLaMA-Factory / NeMo | fragile / in progress | Known to be unreliable on this platform as of this writing; expect breakage and check upstream issues before depending on either for a run. |

## Known-Good Version Matrix (Dated)

`SKILL.md`'s bare-pip sequence pins `datasets`/`trl` explicitly
for a reason: an unpinned `pip install transformers peft
hf_transfer datasets trl accelerate` resolves current PyPI
versions of `transformers`/`trl`/`datasets` that sit well
outside what a given Unsloth release declares support for — pip
installs them anyway and only warns after the fact. The
combination below was confirmed working end-to-end (bf16 LoRA
load + attach + a full SFT run) on `nvcr.io/nvidia/pytorch:25.09-py3`
as of the date above; treat it as a dated snapshot to re-verify,
not a permanent pin:

| Package | Verified-working version |
|---|---|
| `transformers` | 5.13.1 |
| `trl` | 1.8.0 |
| `peft` | 0.19.1 |
| `datasets` | 4.3.0 (pin as-is; not re-verified independently of the combination above) |
| `unsloth` / `unsloth_zoo` | 2026.7.2 |
| `torchao` | 0.17.0 (pure-Python wheel; NGC base image ships 0.13.0+git, too old — `pip install -U torchao` after the Unsloth line) |
| `bitsandbytes` | 0.49.2 |
| `hf_transfer` | 0.1.9 (current stable; see the deprecation note below before relying on it) |

If a bare-pip install lands on a different combination than
this table (pip resolver drift is expected as new releases
ship), re-run the load+LoRA-attach smoke test in `SKILL.md`'s
Verification Commands before trusting the environment, and
check `gh issue list --repo NVIDIA/dgx-spark-playbooks` for a
version-skew report matching the symptom before assuming it's
novel.

**`HF_HUB_ENABLE_HF_TRANSFER` is deprecated on `huggingface_hub`
1.23+.** Setting it now only produces `FutureWarning: The
HF_HUB_ENABLE_HF_TRANSFER environment variable is deprecated ...
Please use HF_XET_HIGH_PERFORMANCE instead`, and downloads route
through Xet rather than hf_transfer regardless. This is cosmetic
(downloads still succeed, and fast) on current `huggingface_hub`
— stale task instructions or older recipes that still reference
`hf_transfer`-based env setup should be read as intent ("make
downloads fast"), not a literal current-API requirement; set
`HF_XET_HIGH_PERFORMANCE=1` instead on `huggingface_hub` 1.23+.

## GPU-Detection False Negative: Per-Hypothesis Detail

The full discriminating check behind `SKILL.md`'s Verification
Commands hypothesis table, in the order to work through them:

1. **Runtime/flags.** If `docker run` was missing
   `--runtime=nvidia --gpus all`, `nvidia-smi` run *inside* the
   container fails or shows no devices even though the host sees
   the GPU fine. Fix: re-run with both flags.
2. **Device visibility.** `echo $CUDA_VISIBLE_DEVICES` — an
   empty string set explicitly (not merely unset) hides all
   devices from CUDA; a stale index (e.g. `1` on a single-GPU
   box) hides the only device present. Fix: `unset
   CUDA_VISIBLE_DEVICES` or set it to `0`.
3. **Permissions.** `ls -l /dev/nvidia*` — missing entries or a
   `Permission denied` on read means the container/user can't
   open the device nodes (common when running rootless or with a
   restrictive seccomp/AppArmor profile). Fix: match the host's
   device-cgroup rules, or don't run rootless for GPU workloads.
4. **CUDA init state.** A prior process that crashed mid-kernel
   can leave the driver's CUDA context wedged for that process
   tree. Retrying in a fresh shell or a freshly started container
   (not just a new Python process in the same shell) rules this
   out cheaply before assuming anything deeper is wrong.
5. **ABI mismatch.** The last hypothesis to check, not the
   first: `python3 -c "import torch; print(torch.version.cuda)"`
   not starting with `13` confirms a `libcudart.so.12`-linked
   wheel on a CUDA-13-only system — see the ABI Rule in
   `SKILL.md`. This is the only one of the five that a wheel
   reinstall actually fixes; reinstalling before ruling out 1-4
   wastes a cycle without changing the outcome if the real cause
   is a flag, an env var, or a permission.

A torchcodec/driver interaction is the most frequently reported
instance of (5) on this hardware specifically — see
`gh issue list --repo NVIDIA/dgx-spark-playbooks` for current
reports before assuming a novel cause.

## sm_121 vs sm_121a

GB10's GPU identifies as `sm_121`. Some newer kernel features — notably NVFP4's native `cvt.e2m1x2` conversion instruction — require code compiled for `sm_121a`, a superset target, not plain `sm_121`. If NVFP4 inference is ~32% slower than FP8 on this hardware, this is why: the kernel likely wasn't compiled with the `a` variant. Check the build flags of whatever wheel or container you're using before assuming the hardware itself is the bottleneck.

## Canonical resources

- `github.com/NVIDIA/dgx-spark-playbooks`
- `build.nvidia.com/spark/unsloth`
- `github.com/natolambert/dgx-spark-setup`
- `github.com/albond/DGX_Spark_Unsloth_Lossless_Speedup`
- `github.com/NvMayMay/nvfp4-lora-spark`

Official playbooks have shipped broken before. Check each repo's recent issues before starting a long run, not after it fails.