{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidn2yo52lgbk7nvpkmfekc5qxzwsij5yqkma3kzeiii4fhibmaxcq",
    "uri": "at://did:plc:7fzsvgbjbyp2vxleodbcwyif/app.bsky.feed.post/3mhyoy6xhqgu2"
  },
  "path": "/2026/03/26/turboquant-kv-cache-optimization.html",
  "publishedAt": "2026-03-26T00:00:00.000Z",
  "site": "https://jens.dev",
  "tags": [
    "previous post",
    "follow-up",
    "https://github.com/apimeister/gguf-runner"
  ],
  "textContent": "## A more technical follow-up\n\nIn the previous post I introduced **gguf-runner** , a small Rust CLI for running GGUF models locally on CPU.\n\nIn the follow-up I wrote about vision support, release binaries, and a number of smaller improvements.\n\nThis post is about one of the more practical changes: adding **TurboQuant** as a new KV-cache mode that cuts KV-cache memory sharply without giving up much throughput.\n\nRepo: https://github.com/apimeister/gguf-runner\n\n* * *\n\n## Why the KV-cache matters so much\n\nFor long-context inference, the KV-cache quietly becomes one of the dominant costs.",
  "title": "TurboQuant in gguf-runner: roughly half the memory at nearly the same speed"
}