Mafdet

You type. It already knows. Local LLMs that read and think while you type.

LLM Mart 5 views 295k listing impressions

MAFDET reads your message while you type it, and starts thinking about it before you send it.
Hit Enter, and most of the work is already done.

License: Apache-2.0 PyPI Runtime: local Status: prototype

Italiano


Why

Chat interfaces make the model wait for you to press Enter before it does anything. While you type, the GPU sits idle. When you send, the model first has to read your whole message (prefill). Then a reasoning model, like Qwen3 in thinking mode, thinks for hundreds of tokens before the first word of its answer. On a mid-range GPU that thinking is the longest wait of all: ten seconds and more on a word problem, minutes on a model larger than the VRAM.

MAFDET uses the time you spend typing for both. It reads your message as you write it, so the prefill is done before Enter. And it thinks ahead: the model starts reasoning about your message while you are still typing it, and keeps that reasoning when you change the text. On Qwen3-4B, with 24 word problems typed in real time, the mean wait after Enter goes from 12.2 s to 3.2 s, with the same 24/24 correct answers; on Qwen3-14B, from 108 s to 47 s. On a MacBook Pro M2 Pro, from 18.1 s to 5.6 s (4B, 24/24 both ways) and from 32.7 s to 12.1 s (14B).

MAFDET is built for the hardware most people actually own. The reference machine is an RTX 2060 with 6 GB of VRAM; it also runs on Apple Silicon Macs (Metal, measured on an M2 Pro with 16 GB), and everything also runs on CPU only. AMD Radeon cards run through Vulkan, measured so far on one card (first numbers, more wanted: help wanted).

Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

How it works

Reading ahead

An LLM processes text in two phases:

Phase What happens Speed
Prefill reads the prompt; all tokens are processed in parallel fast
Decode generates the answer, one token at a time slow

While reading, the model stores intermediate results for every token in the KV cache. Those results never change for a given prefix, so they can be computed ahead of time.

MAFDET keeps the KV cache in sync with the text while you type:

  1. On every pause, it tokenizes the current text.
  2. It holds back the last few tokens, because a word you are still typing can still change how the text is split into tokens.
  3. It finds the longest common prefix between the new tokens and the ones already in cache, trims the cache there, and prefills only what is new.
  4. When you hit enter, only a handful of tokens are left to process.

Correctness does not depend on how you type. Backspaces, edits in the middle of the text, pasted blocks, missing punctuation or spaces: the common-prefix check always brings the cache back to exactly what a from-scratch computation would produce. Heuristics such as hold-back and debounce only affect how much work is wasted, never the result.

Thinking ahead

From the project's README.

Comments (0)

Sign in to join the conversation.

No comments yet.