Building Local: My 2026 Headless AI Server Journey

"You’re totally right to keep an eye on them—local models have come a long way! Even with a 3060 Ti, you can actually run some impressive stuff.

Because of how GGUF works now, you can ‘offload’ specific layers to your 8GB VRAM and let the rest spill over into your 32GB of system RAM. If you want to try it, I’d recommend starting with Gemma-4-E4B-it-Q8_0.gguf.

At that quantization, the model is about 7.5GB. If you set your context to PARAMETER num_ctx 32768, it should fit comfortably across your VRAM and RAM. You’ll probably see speeds around 8–15 tokens/sec —not blazing fast, but the reasoning quality is excellent for a local setup.

If you want to go even bigger, you could technically run the Gemma-4 26B (A4B). By putting as many layers as possible on the GPU and the rest on your memory, you’d likely hit 6–8 tokens/sec. Even if you went full system memory, you’d still get about 3–4 tokens/sec. It’s definitely worth a shot if you want cloud-level smarts without the privacy concerns!"