Mellum2-12B-A2.5B-Instruct Q4_K_M on Jetson Orin Nano 8GB

Oh. That is an important data point. I was estimating the headroom quite conservatively because Jetson uses shared CPU/GPU RAM, but it looks like there may be room for larger models than I expected:

---

Thanks for reporting the actual measurements. I would update my earlier take based on your Qwen2.5-Coder-7B-Instruct Q4_K_M result.

The important correction is:

7B Q4_K_M is clearly not out of range for Jetson Orin Nano 8GB. It can be practical when the model architecture, GGUF size, context length, JetPack/L4T version, and llama.cpp backend are friendly.

Your Qwen test is especially useful because it is not just a “loaded once” result. It looks like a stable practical baseline:

Observation	What I think it means
Qwen2.5-Coder-7B-Instruct Q4_K_M loaded and ran normally	7B Q4 is a real target on this board, not just a theoretical fit.
~11.2–11.3 tok/s generation	Very usable for local coding assistance on a small edge board.
~5.1GB RAM under load	Consistent with a ~4.7GB GGUF plus runtime/KV/buffer overhead.
Modest swap usage, stable system	This does not sound like “barely alive by swapping”; it sounds usable.
-ngl 60, 80, 99 made little difference	Once the important path is offloaded, decode may be memory-bandwidth/runtime limited rather than layer-count limited.
Context 1024 vs 2048 made little difference	Qwen2.5 7B has GQA and a relatively small KV cache at ordinary chat lengths.
Flash Attention made little difference	At short/medium context, attention may not be the dominant cost.
Batch/microbatch tuning made little difference	Mostly affects prompt processing, not necessarily steady-state decode.
KV cache q8_0 was significantly slower	KV quantization is not automatically faster; it can hit slower kernels or dequant overhead.
KV cache q4_0 degraded quality	Useful as an emergency memory lever, not necessarily a good default when the model already fits.

So I would now treat Qwen2.5-Coder-7B-Instruct Q4_K_M as the known-good coding baseline for this device.

Revised practical tiering

My previous framing was too conservative if read as “Jetson Nano 8GB can only really do 2B–4B”. Based on your Qwen result, I would revise the tiers like this:

Tier	Orin Nano 8GB interpretation
2B–4B Q4/Q5	Safe baseline. Good first test for JetPack/container/llama.cpp sanity.
7B Q4_K_M	Practical. Your Qwen2.5-Coder result proves this can be a real target.
8B Q4 / 8B low-bit	Worth testing carefully. Architecture and GGUF size matter a lot.
9B Q4	Possible but more aggressive; likely sensitive to context/runtime/settings.
12B MoE Q4	Mostly experimental on this board. Active parameters can be misleading.
30B+ total MoE / 32B total models	Not a practical 8GB shared-memory Jetson target unless the goal is boundary testing.

For a normal 8GB discrete GPU, 7B–8B Q4_K_M models are already a common practical target. The Jetson Orin Nano 8GB is a special case because its 8GB LPDDR5 is shared by CPU and GPU , so it is not the same as “8GB VRAM plus separate system RAM”.

NVIDIA lists the Orin Nano Super Developer Kit as 8GB 128-bit LPDDR5, 102GB/s , with 1024 CUDA cores, 32 Tensor Cores, and 7W–25W power modes: Jetson Orin Nano Super Developer Kit.

That still means shared-memory constraints are real. But your result shows that the practical ceiling is not as low as I first implied.

---

Why Qwen2.5-Coder-7B had more headroom than expected

I think the result makes sense for several reasons.

1. The actual Q4_K_M GGUF is only about 4.68GB

The Qwen2.5-Coder-7B-Instruct Q4_K_M file is about 4.68GB :

• Qwen/Qwen2.5-Coder-7B-Instruct-GGUF — Q4_K_M file

That leaves some room for:

• llama.cpp runtime buffers
• CUDA allocations
• KV cache
• OS/services
• mmap/page-cache behavior
• server overhead

A rough mental model is:

Qwen2.5-Coder-7B Q4_K_M weights:  ~4.68GB
KV cache at normal chat lengths:   relatively small
runtime/CUDA/server overhead:      additional memory
observed under load:               ~5.1GB RAM

So the ~5.1GB RAM observation is quite plausible. This model is not close to the same memory class as an 8GB-class GGUF or a 19GB-class GGUF.

---

2. Qwen2.5 7B uses GQA, so ordinary-context KV cache is small

Qwen2.5 7B uses Grouped-Query Attention. Its model card lists:

• 28 layers
• 28 query heads
• 4 key/value heads
• GQA
• RoPE
• SwiGLU
• RMSNorm
• QKV bias

Source:

• Qwen/Qwen2.5-7B-Instruct

The 4 KV heads matter. With GQA, the model stores fewer KV heads than ordinary full multi-head attention. That keeps KV cache smaller.

A rough f16 KV cache estimate for Qwen2.5 7B is:

K and V tensors
× 4 KV heads
× 128 head_dim
× 2 bytes per f16 value
× 28 layers
= 57,344 bytes/token
≈ 56 KiB/token

Approximate KV cache size:

Context	Approx. f16 KV cache
512	~28MB
1024	~56MB
2048	~112MB
4096	~224MB
8192	~448MB

This is small compared with the 4.68GB weight file. That explains why ctx 1024 vs 2048 did not change much.

It also explains why KV quantization was not automatically helpful. If KV cache is already small, quantizing it saves little absolute memory, while it may introduce slower kernels, dequant overhead, or output-quality loss.

Relevant references:

• Grouped-Query Attention paper
• llama.cpp server docs: --cache-type-k, --cache-type-v
• llama-server man page showing KV cache type options

---

3. Qwen2.5-Coder-7B is a mature dense-transformer path

Qwen2.5-Coder-7B-Instruct is a code-specialized dense model in the Qwen2.5-Coder family. The technical report describes the Qwen2.5-Coder series as including 0.5B, 1.5B, 3B, 7B, 14B, and 32B models, with continued pretraining on more than 5.5T tokens and evaluations across code generation, completion, reasoning, and repair.

References:

• Qwen2.5-Coder Technical Report
• Qwen/Qwen2.5-Coder-7B-Instruct

This matters because Qwen2.5-Coder-7B is not just a random 7B chat model. It is a strong code-specialized model that happens to fit into a realistic GGUF size for this board.

That makes it a very good baseline.

---

Why Granite H-Small behaved so differently

Your Granite result also makes sense, but I would not compare granite-4.0-h-small to the Qwen 7B run as if they were adjacent model sizes.

IBM’s naming is a little easy to misread here. Granite 4.0 H-Small is not a small 7B-class model. IBM describes Granite 4.0 H-Small as a 32B total / 9B active hybrid MoE model, while H-Tiny is 7B total / 1B active and H-Micro/Micro are 3B-class models.

References:

• IBM announcement: Granite 4.0 models
• Unsloth Granite 4.0 guide
• IBM Granite model docs

So the granite-4.0-h-small result is consistent with expectations:

Model	Practical memory class
Qwen2.5-Coder-7B Q4_K_M	~4.68GB dense GGUF; fits with useful headroom.
Granite 4.0 H-Small Q4_K_M	32B total / 9B active hybrid model; not a 7B-class target.
Mellum2-12B-A2.5B Q4_K_M	12B total MoE; active 2.5B does not remove weight-residency cost.

Your Granite observations are exactly what I would expect from a model in the wrong memory class for this board:

Granite H-Small result	Interpretation
-ngl 99: CUDA OOM, attempted ~18.6GiB allocation	Full/near-full offload is impossible on 8GB shared RAM.
-ngl 20: CUDA OOM, attempted ~9.0GiB allocation	Still above the practical device-memory budget.
-ngl 10: loaded but extremely slow	Technically possible to start, but the system is near limits and paging/offload/latency dominate.

For Granite on Orin Nano 8GB, I would test smaller variants instead:

Granite model	Why it is more relevant
granite-4.0-h-micro-GGUF	3B hybrid model; safer Jetson memory class.
granite-4.0-micro-GGUF	3B conventional transformer option.
granite-4.0-h-tiny-GGUF	7B total / 1B active; much more comparable to your successful Qwen 7B run.
granite-4.0-h-small-GGUF	32B total / 9B active; useful boundary test, not a practical Nano 8GB target.

Relevant model links:

• ibm-granite/granite-4.0-h-tiny-GGUF
• unsloth/granite-4.0-h-tiny-GGUF
• ibm-granite/granite-4.0-h-small-GGUF
• unsloth/granite-4.0-h-small-GGUF
• IBM GGUF conversion repo

---

Why Mellum2 is still a different case from Qwen2.5-Coder-7B

Mellum2 remains interesting, but I would still not put it in the same class as Qwen2.5-Coder-7B on this board.

Mellum2 is a 12B total / 2.5B active MoE model. The technical report describes 64 experts, 8 active experts, GQA, sliding-window attention, and a multi-token prediction head.

References:

• JetBrains/Mellum2-12B-A2.5B-Instruct
• Mellum2 Technical Report
• HF paper page: Mellum2

The crucial point is:

Active parameters reduce per-token compute, but total resident weights still matter.

So even if Mellum2 runs with 2.5B active parameters per token, it is still a 12B-total MoE model for weight storage and runtime layout. That makes it very different from a 4.68GB dense Qwen2.5-Coder-7B GGUF.

I would now phrase Mellum2 like this:

Question	Updated answer
Is Mellum2 impossible?	Not necessarily. It is worth experimenting with if the goal is boundary testing.
Is Mellum2 Q4_K_M a practical recommendation for Orin Nano 8GB?	I still doubt it.
Does Qwen2.5-Coder-7B success imply Mellum2 should also work?	No. The memory class and architecture are different.
What would make Mellum2 more interesting?	A high-quality lower-bit GGUF, careful MoE offload behavior, and very short context tests.

---

JetPack/L4T probably mattered too

Your upgrade from JetPack/L4T 36.4.7 to 36.5.0 is also important.

There is a related NVIDIA forum thread where Gemma4 E4B failed with CUDA OOM on Orin Nano, and NVIDIA later confirmed it working on r36.5 / JetPack 6.2.2. The thread also mentions a known memory issue in r36.4.7 that was fixed in r36.5.

Reference:

• Gemma4 E4B on Jetson Orin Nano fails due to CUDA out of memory issue

So part of the Qwen success may be that you were no longer testing on a release with a known memory problem.

Useful checks for future posts:

cat /etc/nv_tegra_release
dpkg-query --show nvidia-l4t-core
sudo nvpmodel -q
tegrastats

---

What I would test next

Given your result, I would not spend most of the time trying to force Mellum2 first. I would use your Qwen result as the reference point and compare other models against it.

Practical next candidates

Priority	Candidate	Reason
1	Keep Qwen2.5-Coder-7B-Instruct Q4_K_M	Known-good coding baseline on this exact board.
2	Qwen2.5-Coder-7B other quants	Compare Q4_K_M vs Q5_K_M or IQ4/IQ3 if available.
3	Qwen 7B/8B-ish newer coder/instruct GGUFs	Same broad architecture family may preserve good runtime behavior.
4	Granite 4.0 H-Tiny	7B total / 1B active; much more relevant than H-Small.
5	Granite 4.0 Micro / H-Micro	3B-class safe Granite tests.
6	LFM2 / LFM2.5 8B-A1B low-bit	Interesting edge-MoE class, but should be tested against the Qwen baseline.
7	StarCoder2-3B / OpenCoder / Granite Code 3B	Useful for code completion or smaller coding tasks.

Models I would keep as boundary tests

Model	Why
Mellum2 Q4_K_M	Interesting 12B MoE, but still likely too large for comfortable Nano 8GB use.
Granite 4.0 H-Small	32B total / 9B active. Your results already show the boundary.
Qwen 30B-A3B / 35B-A3B MoE	Interesting on 32GB RAM PCs, not this board.
Qwen3-Coder-Next 80B-A3B	Very interesting model class, wrong memory class for Orin Nano 8GB.

---

Updated conclusion

My revised interpretation is:

Orin Nano 8GB has enough practical headroom for well-behaved 7B Q4_K_M models. The successful Qwen2.5-Coder-7B run shows that clearly. But that extra headroom does not automatically extend to every active-small MoE model, because total GGUF size, resident weights, KV layout, architecture, and JetPack/L4T behavior still dominate.

So I would no longer say “stay mostly under 4B” for coding models on this board.

I would say:

1. 2B–4B is the safe zone.
2. 7B Q4_K_M is now proven practical by your Qwen result.
3. 8B low-bit / 8B Q4 is worth exploring.
4. 9B Q4 is possible but aggressive.
5. 12B MoE Q4 is still mostly experimental.
6. 32B/9B-active or 30B+ MoE is not a practical Nano 8GB target.

The most useful takeaway for me is:

Use actual GGUF size and architecture, not just parameter count or active parameter count.

That explains all three results:

Model	Result	Likely reason
Qwen2.5-Coder-7B Q4_K_M	Practical	~4.68GB dense GGUF, GQA, small KV, mature runtime path.
Granite 4.0 H-Small Q4_K_M	OOM or extremely slow	32B total / 9B active hybrid model; wrong memory class.
Mellum2-12B-A2.5B Q4_K_M	Still doubtful	12B total MoE; active 2.5B does not make it behave like a 2.5B dense model in memory.