DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121\nGPU, 128GB unified memory, CUDA 13. This is a narrower and\nyounger platform than a standard x86 CUDA 12 box, so package\nselection and ABI matching matter more than usual — the wheel\necosystem for aarch64 + CUDA 13 is still filling in.
\nlibcudart, a missing\nsymbol, or a wheel that \"installed fine but won't load.\"Each of these accepts the same general fix: match the\ncontainer/wheel combination to CUDA 13 and SM121, don't fight\nthe ABI.
\nQuick decision, before the detail below:
\nDefault to a container. Use nvcr.io/nvidia/pytorch:25.09-py3\nas the base for general work — the newest tag confirmed working\non this hardware; pull a newer blessed tag if locally available\nrather than hard-blocking on 25.11-py3. NGC's tag is dated, so\nrunning it directly is fine:
docker run --runtime=nvidia --gpus all -it --rm \\\n nvcr.io/nvidia/pytorch:25.09-py3\nunsloth/unsloth:dgxspark-latest is a moving tag by\ncontrast — resolve and pin its digest before running it for\nanything reproducible; the bare tag is a discovery step only,\nnot the default invocation. Full pull-inspect-pin sequence and\nflag rationale/volume mounts for finetuning/ run dirs:\nreferences/container-workflow.md. Treat bare pip as the exception.
The reason for the container-first stance is pinning, not\nconvenience. Triton, xformers, and transformers versions\ninteract narrowly with GB10's SM121 target and CUDA 13; a\ncontainer locks all of them together against a combination\nalready validated on this hardware. Bare pip leaves that\nresolution to you, one broken import at a time.
\nWhen bare pip is warranted, follow the NVIDIA playbook's\ninstall sequence verbatim and in order:
\npip install \"transformers==5.13.1\" \"peft==0.19.1\" \"hf_transfer==0.1.9\" \"datasets==4.3.0\" \"trl==1.8.0\"\npip install --no-deps \"unsloth==2026.7.2\" \"unsloth_zoo==2026.7.2\" \"bitsandbytes==0.49.2\"\npip install -U \"torchao==0.17.0\"\nThe second command's --no-deps flag is not optional —\nletting pip re-resolve Unsloth's dependency tree on aarch64 is\na common way to pull in an incompatible torch or triton build.\nThe third line is not optional either: the NGC base image's\nbundled torchao is too old for current peft's LoRA-attach\npath (ImportError: ... torchao ... only versions above 0.16.0 are supported) — a hard blocker, not a warning. Every == pin\nabove is load-bearing, taken from the dated known-good version\nmatrix in references/stack-matrix.md (its Last verified date\ngoverns staleness) — an unpinned install resolves current PyPI\nversions well outside what this Unsloth release supports.
Pull a fresh tag when a new blessed release is announced.\nRebuild locally from one of the two bases only when a project\nneeds an extra system package layered in — not to \"upgrade\" a\ncomponent the image already pins. Details on both paths:\nreferences/container-workflow.md.
One more preflight: official DGX Spark playbooks have shipped\nbroken before. Check recent issues on\ngithub.com/NVIDIA/dgx-spark-playbooks (and the other\nresources in references/stack-matrix.md) before trusting a\nrecipe verbatim for a long run.
The single most common failure on Spark is a CUDA 12/13 ABI\nmismatch: a wheel built against libcudart.so.12 loaded on a\nsystem that only has libcudart.so.13. The install usually\nsucceeds; the failure surfaces later as a missing-symbol error\nor a segfault that doesn't obviously point at CUDA.
Fix: pull wheels from download.pytorch.org/whl/cu130 (the\ncu130-tagged aarch64 builds), or use one of the containers\nabove, which already carry a matched build. Before chasing a\nstack trace that mentions a CUDA symbol, check which CUDA tag\nthe installed wheel was built against:
python3 -c \"import torch; print(torch.version.cuda)\"\nIf that output doesn't start with 13, the ABI mismatch is the\nfirst thing to fix. NGC container builds (e.g.\nnvcr.io/nvidia/pytorch:25.09-py3) build torch internally\nagainst CUDA 13 with no +cu130 wheel tag — pip show torch\nwon't say cu130 there, and that absence alone is not a failure.
Typical symptoms:
\nImportError: undefined symbol referencing a CUDA runtime\nfunction..cuda() call, no useful traceback.The fix is the same regardless of symptom: match the wheel's\nCUDA tag to the system, or use a container that already does.
\nCondensed status for the components most likely to come up.\nFull table with wheel URLs, build flags, the sm_121 vs sm_121a\ndistinction, and the dated known-good version matrix:\nreferences/stack-matrix.md.
| Component | \nStatus | \n
|---|---|
| PyTorch | \n✅ official cu130 aarch64 wheels | \n
| bitsandbytes | \n✅ works out of the box | \n
| Triton | \n✅ needs the TRITON_PTXAS_PATH parameter set | \n
| flash-attn | \n❌ skip pip build; NGC bundles a working one — see spark-training-gotchas G2 | \n
| xformers | \nsource build only (TORCH_CUDA_ARCH_LIST=12.1) | \n
| vLLM | \nnightly wheels only | \n
| TransformerEngine / NVFP4 train | \ncontainer-only | \n
Everything else — Unsloth, Axolotl, TRL, PEFT — installs\ncleanly through the container-first path above. LLaMA-Factory\nand NeMo are fragile on Spark; check upstream issues first.
\nConfirm the environment can actually see the GPU before\nrunning anything expensive:
\nimport torch\nprint(torch.cuda.is_available(), torch.version.cuda)\nThis call returns two values; the exact output format is one\nline, <bool> <cuda-version>:
True 13.0\nIf it prints False instead, don't jump straight to a wheel\nreinstall — ABI mismatch is one cause among several:
| Hypothesis | \nQuick check | \n
|---|---|
| Runtime/flags | \nnvidia-smi fails in-container too | \n
| Device visibility | \necho $CUDA_VISIBLE_DEVICES | \n
| Permissions | \nls -l /dev/nvidia* | \n
| CUDA init state | \nwedged process; retry fresh shell/container | \n
| ABI mismatch (usual culprit) | \ntorch.version.cuda not 13.x | \n
Check nvidia-smi first — if it doesn't show the GPU, it's one\nof the first three, not ABI. Reinstall a wheel only once ABI is\nconfirmed. Per-hypothesis detail: references/stack-matrix.md.\nRun right after the container starts, before installing\nproject-specific packages.
One more check: if Triton kernel compilation fails once\ntraining starts, set\nTRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas and retry — see\nreferences/stack-matrix.md for the full workaround list.
A verified environment is only the starting point. See also:\nspark-training-gotchas for failure preflights before a\ntraining run, and spark-memory-thermal-ops for unified-memory\nOOMs and thermal throttling during long ones.