{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihwqplqzqablw3z74lexdofak5gipvpsujsp23w4b5lu7bg47tezu",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mkleqevdnkw2"
  },
  "path": "/t/svsk-q-quantization-method/175634#post_1",
  "publishedAt": "2026-04-28T18:05:56.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "https://github.com/Dookoo2/SVSK](https://github.com/Dookoo2/SVSK)"
  ],
  "textContent": "Hello! I’m a newbie here, but I want to show my last project – **SVSK** (Structured Vector Sidecar).\n\nNB! It seems I initially created a topic in the wrong section, I’ll delete the previous one in “newbies”.\n\nIt’s a post‑training quantization method that keeps a strong 4‑bit base and adds a tiny low‑rank sidecar (rank 8/16) to recover the most harmful quantization error.\n\n**Key numbers (Qwen3‑4B, Wikitext validation):**\n\n| Variant | ΔNLL (↓ better) | PPL ratio |\n\n|---------|----------------|------------|\n\n| SVSK r16 (dense restore) | **0.028** | 1.028 |\n\n| Q4_K_M (llama.cpp) | 0.032 | 1.033 |\n\n| Q4_K_XL (llama.cpp) | 0.036 | 1.037 |\n\n-> SVSK has **~15% lower degradation** than Q4_K_M in this test.\n\n**What makes it different?**\n\n- Activation‑aware 4‑bit base (AA‑NativeQ4) – clips per channel.\n\n- Tile‑local low‑rank sidecar (U·V) stored in int8.\n\n- Total budget: 4.44 bpw (r8) / 4.6 bpw (r16) – not cheating with hidden 6‑bit.\n\n- No fine‑tuning, just calibration on 128 chunks of Wikitext.\n\n**Current status:**\n\n-  Offline quality better than Q4_K_M (on Qwen3‑4B).\n\n-  Alpha runtime with Triton kernels – ~34 tok/s on RTX 4000.\n\n-  No CUDA yet, not integrated into llama.cpp.\n\n-  Not production‑ready.\n\n**What I need help with:**\n\nAll that I need - your feedback! I need all of the meanings about it, usefull or useless - the answer is up to you:)\n\n**Full code, instructions and autotune script:**\n\n[https://github.com/Dookoo2/SVSK](https://github.com/Dookoo2/SVSK)\n\nYou can reproduce the PPL comparison in about 1 or 2 hours - I tried to write good README with “step by step” guide.\n\nThanks for reading! Any feedback is welcome.",
  "title": "SVSK -Q quantization method"
}