TensorFold

TensorFold serves a local LLM on Apple Silicon or NVIDIA GPUs at an OpenAI-compatible endpoint, fast and exact. Name a model on Hugging Face, choose the context window and sampling, and TensorFold downloads it, loads it with Metal or CUDA kernels written for that model family, and serves /v1/chat/completions. On DGX Spark it decodes 1.6 to 3x faster than vLLM with MTP drafts, one Spark or two (DGX Spark).

Setting this up with an AI agent? Give it the AI agent runbook for the install, model download, server startup and a request that checks the result.

pip install git+https://github.com/ashhart/TensorFold.git
tensorfold serve Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --context 65536

Any OpenAI client can then use http://127.0.0.1:8080/v1, including coding agents, SDKs and curl:

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit", "messages": [{"role": "user", "content": "Hi"}]}'

TensorFold needs a Mac with Apple Silicon and Python 3.11 or newer. It is tested with MLX 0.31.2 on an M5 Max and MLX 0.32.0 on an M3 Ultra. Flash Next and Nemotron check at load that drafted rows reproduce one-row decoding on your MLX and GPU, and draft only when they do. On an M5, MLX 0.32.2 fails that check for Nemotron, which then runs without drafts (same output, slower); pip install mlx==0.31.2 brings them back. The Qwen3.8-27B lane kernels do their own arithmetic, so their exactness does not depend on the MLX version.

Models

Each model family has its own package of kernels, picked from the checkpoint's config.json. These are the checkpoints TensorFold is built and tested with, all on Hugging Face:

The three main checkpoints come from the Vontra Hugging Face namespace; Qwen3.8-27B's optional DFlash2 drafter comes from z-lab.

tensorfold pull Vontra/Qwen3.8-27B-MLX-4bit z-lab/Qwen3.8-27B-DFlash2
tensorfold serve Vontra/Qwen3.8-27B-MLX-4bit

serve downloads a model it doesn't have yet; pull downloads ahead of time. Models go into the Hugging Face cache (~/.cache/huggingface), and a local model directory works too. tensorfold models lists the families and their checkpoints.

What each checkpoint needs:

• Qwen3.8 Flash Next drafts with the MTP head stored in its checkpoint, and its kernels read 4-bit weights in groups of 32. Use the -MLX-4bit-MTP conversion. TensorFold refuses other bit widths before downloading anything, and a conversion without the MTP head runs without drafts.
• Qwen3.8-27B drafts with the DFlash2 draft model once it has been pulled; serve picks it up automatically. Its lane kernels need 4-bit weights in groups of 64 and Metal 4 tensor units (M5-generation GPUs). Elsewhere it runs on MLX's own kernels: every token is still the model's own sample, but drafted rows are checked at width rather than bit-identical to one-row decoding.
• Nemotron 3.5 Lightning drafts with its MTP head, which the checkpoint above ships as mtp-4bit.safetensors (converted from NVIDIA's BF16 release; the standard MLX conversion drops it), and from the context. pull checks for the head, and serve completes an older cache that lacks it before loading.

Want another model? The recipe book describes what we did for each family and how to add yours.

Speed