LanguageIn development
NanoLLM
A cross-platform LLM runtime for edge computing. It serves Google's Gemma 4 E4B, quantized to 4 bits, from a single binary with an OpenAI-compatible API, on the CPU you already have.
Speed on a plain server CPU
No GPU involved. Tokens per second on an AMD EPYC 7351P, a 2017 server processor with AVX2.
| Setup | Generation | Prompt processing |
|---|---|---|
| 16 cores | about 25 tok/s | about 140 tok/s |
| 4 cores | about 12 tok/s | about 44 tok/s |
Gemma 4 E4B, 4-bit weights, measured with llama-bench on our test server. With speculative decoding, chat on 16 cores reaches about 33 tok/s. Your numbers will differ: run the benchmark.
What it does
Built for the CPU
Hand-tuned 4-bit kernels for AVX2 processors make prompt processing about 1.8 to 1.9 times faster than stock llama.cpp, with bit-identical output. Model loading is parallel and NUMA-aware.
OpenAI-compatible API
Chat completions with streaming, tool calls and JSON output, plus a native API with sessions and a prompt cache. One Go binary, no Python.
Speculative decoding
Multi-token prediction and prompt lookup draft several tokens at once. Chat gets up to 1.4 times faster on top of the kernels.
Fits in 8 GB
The default preset peaks near 6.5 GiB. Smaller memory budgets trade speed for space, down to modes that stream the weights from disk.
Optimizations never change the answer
Every build is checked against golden outputs and a KL-divergence gate, so a faster kernel cannot quietly change what the model says.
Measured in the open
The benchmark runs the same suite on any machine and shows medians over many installations.
Status
NanoLLM is in development. Linux on x86-64 is verified end to end on the CPU. Libraries for Linux, macOS and Windows on x86-64 and arm64 are built and tested in CI. GPU backends are next.
Run the benchmark
See how NanoLLM's presets run on real hardware, and add your own machine to the picture.