Crit
Agentic toolkit for building and optimizing high-performance inference engines tailored to your GPU and desired LLM.
Each engine is crit Core plus plugins, one per slot, with 33 plugins in crit's library. It uses low-bit weight projections where weights are stored as low-bit codes with one scale per group of values. Crit implements Paged KV cache with prefix sharing, where the KV cache is cut into fixed pages.
Token sampling is performed on the GPU, with the GPU applying the whole sampling law to the full vocabulary. Crit records execution choices for each phase and shape, picking eager launches, a full graph, or a partial graph from measured costs.
Summary drafted from the project's own website. Every sentence is backed by text on that page and was reviewed before publishing.