MAFDET reads your message while you type it, and starts thinking about it before you send it.
Hit Enter, and most of the work is already done.
Why
Chat interfaces make the model wait for you to press Enter before it does anything. While you type, the GPU sits idle. When you send, the model first has to read your whole message (prefill). Then a reasoning model, like Qwen3 in thinking mode, thinks for hundreds of tokens before the first word of its answer. On a mid-range GPU that thinking is the longest wait of all: ten seconds and more on a word problem, minutes on a model larger than the VRAM.
MAFDET uses the time you spend typing for both. It reads your message as you write it, so the prefill is done before Enter. And it thinks ahead: the model starts reasoning about your message while you are still typing it, and keeps that reasoning when you change the text. On Qwen3-4B, with 24 word problems typed in real time, the mean wait after Enter goes from 12.2 s to 3.2 s, with the same 24/24 correct answers; on Qwen3-14B, from 108 s to 47 s. On a MacBook Pro M2 Pro, from 18.1 s to 5.6 s (4B, 24/24 both ways) and from 32.7 s to 12.1 s (14B).
MAFDET is built for the hardware most people actually own. The reference machine is an RTX 2060 with 6 GB of VRAM; it also runs on Apple Silicon Macs (Metal, measured on an M2 Pro with 16 GB), and everything also runs on CPU only.
Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.
How it works
Reading ahead
An LLM processes text in two phases:
| Phase | What happens | Speed |
|---|---|---|
| Prefill | reads the prompt; all tokens are processed in parallel | fast |
| Decode | generates the answer, one token at a time | slow |
While reading, the model stores intermediate results for every token in the KV cache. Those results never change for a given prefix, so they can be computed ahead of time.
MAFDET keeps the KV cache in sync with the text while you type:
- On every pause, it tokenizes the current text.
- It holds back the last few tokens, because a word you are still typing can still change how the text is split into tokens.
- It finds the longest common prefix between the new tokens and the ones already in cache, trims the cache there, and prefills only what is new.
- When you hit enter, only a handful of tokens are left to process.
Correctness does not depend on how you type. Backspaces, edits in the middle of the text, pasted blocks, missing punctuation or spaces: the common-prefix check always brings the cache back to exactly what a from-scratch computation would produce. Heuristics such as hold-back and debounce only affect how much work is wasted, never the result.
Thinking ahead
Reading ahead takes the prefill out of the wait; the reasoning is still there. So the model also reasons while you type: in short chunks, on the message as it is at that moment. When you change the text, the reasoning is not thrown away. It stays in the prompt as notes after the new version of the message, which costs only a prefill, and the model goes on from there. On Enter it finishes reasoning from where it got to and answers. On Qwen3-4B about two thirds of the reasoning is done before Enter. How the notes are framed, and what that changed, is in Thinking ahead, in detail below.
No comments yet.