Greenboost Windows

Windows port of GreenBoost - GPU VRAM extension with system RAM & NVMe for larger LLMs. Original by Ferran Duarri (GPL v2).

LLM Mart 4 views 344k listing impressions

Keep selected model experts in VRAM and load the rest from RAM or a GGUF file when CUDA needs them.

GreenBoost's experimental MoE backend gives expert weights stable GPU addresses while reusing a bounded pool of physical VRAM. The model's own router chooses which experts run. GreenBoost loads their original bytes before the existing llama.cpp CUDA kernels use them. Model calculations stay on the GPU.

Native Windows throughout: the launcher and library are Windows C/C++ binaries built with MSVC. File reads use Win32 APIs, and NVIDIA's Windows CUDA driver runs the GPU kernels. Building and running the Windows product requires no WSL, Linux installation or Linux service. Host integration runs in user mode; the separately retained KMDF driver is legacy and optional.

The latest offloading changes are source only and have not been run or benchmarked. They add CUDA hot-expert tracking, upcoming file reads and optional greedy speculative decoding. No throughput improvement is claimed. The older beta 3 ZIP does not contain this backend.

Pick the path you need

Your situation Use What it does
A supported quantised MoE model needs more expert storage than you want to keep in VRAM Experimental MoE backend, below Loads selected expert weights through a pinned RAM cache into a fixed VRAM pool.
Your model fits normally, or you want to launch another CUDA application Portable allocation shim Uses ordinary CUDA allocations first, with an optional managed-memory fallback and reported capacity allowance.

You choose the path before launch. The portable shim does not automatically turn an existing llama.cpp, Ollama or LM Studio installation into the MoE backend. Dense weights, KV cache and temporary CUDA buffers still need memory outside the expert pool. A configured allowance is not reserved RAM or extra physical VRAM.

Build and run MoE offloading

You need Windows x64, an NVIDIA GPU and driver, Visual Studio C++ Build Tools, CMake, Ninja, a CUDA Toolkit, and your own supported quantised MoE GGUF. Use a single GGUF file. The build uses pinned official llama.cpp b12019.

From this repository's root, build for an RTX 3090 or 3090 Ti:

.\windows-port\tools\build_moe.ps1 `
    -CudaRoot 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3' `
    -Architectures 86

Use architecture 75 for RTX 2000, 86 for RTX 3000, 89 for RTX 4000, or 120 for RTX 5000, with a Toolkit that supports that architecture. RTX 2000 hardware execution remains unverified. Building requires the Toolkit; running the resulting directory uses the copied CUDA runtime libraries.

For a machine with 24 GiB VRAM and 128 GiB RAM, this is an illustrative starting configuration. It gives experts 12 GiB of VRAM and a 64 GiB pinned RAM cache. The remaining memory must cover everything else the model and Windows need.

.\windows-port\tools\run_moe.ps1 `
    -LlamaDirectory '.\windows-port\tools\backends\greenboost-moe-b12019\build\bin' `
    -Model 'D:\models\your-moe.gguf' -Gpu 0 `
    -ExpertVramMiB 12288 -ExpertRamMiB 65536 `
    -HotExperts 8 -PrefetchMiB 16 `
    -Context 4096 -MicroBatch 8 -Tokens 128 `
    -Prompt 'Explain how a heat pump works.'

This does not promise that a particular model fits or runs quickly. Use the CUDA GPU number from llama-completion.exe --list-devices; NVIDIA tools can number the same cards differently. The launcher exposes that card as CUDA0, runs one completion and restores its environment settings afterward.

From the project's README.

Comments (0)

Sign in to join the conversation.

No comments yet.