TurboQuant in gguf-runner: roughly half the memory at nearly the same speed

A more technical follow-up

In the previous post I introduced gguf-runner , a small Rust CLI for running GGUF models locally on CPU.

In the follow-up I wrote about vision support, release binaries, and a number of smaller improvements.

This post is about one of the more practical changes: adding TurboQuant as a new KV-cache mode that cuts KV-cache memory sharply without giving up much throughput.

Repo: https://github.com/apimeister/gguf-runner

Why the KV-cache matters so much

For long-context inference, the KV-cache quietly becomes one of the dominant costs.