External Publication

Mellum2-12B-A2.5B-Instruct Q4_K_M on Jetson Orin Nano 8GB

Hugging Face Forums [Unofficial] June 3, 2026

Hmm… Even outside Jetson, if the practical memory budget is only around 8 GB, I think using LLMs much beyond the 7B–8B class is already pretty rough…

Below is my current read of the situation for Jetson Orin Nano 8GB + GGUF + llama.cpp , especially for trying to run Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf.

Short version

I would not treat this as a “bad GGUF file” first.

I would treat it as a memory-budget problem plus a Jetson runtime-contract problem :

Orin Nano 8GB has 8GB shared LPDDR5 memory , not “8GB VRAM + lots of normal system RAM”.
MoE “active parameters” help compute cost, but the runtime still has to deal with the full quantized weight file , KV cache, CUDA buffers, OS memory, file cache, and server overhead.
Mellum2-12B-A2.5B-Instruct is a 12B total / 2.5B active MoE model, so “A2.5B” does not make it equivalent to a 2.5B dense model for memory residency.
For this board, I would generally start from 2B–4B models , then maybe test 8B low-bit / edge-MoE models only after the Jetson stack and llama.cpp settings are known-good.
I would keep Mellum2 on this board in the “interesting experiment” bucket, not the “practical recommendation” bucket.

Useful official/semi-official starting points:

Jetson AI Lab — Models
Jetson AI Lab — Gemma 4 E2B
NVIDIA Developer Forums — AI Models That Run on Jetson Orin Nano Super 8GB
NVIDIA Developer Forums — Gemma4 E4B on Jetson Orin Nano CUDA OOM thread
Jetson AI Lab — RAM Optimization
jetson-containers setup guide
llama.cpp server docs
llama.cpp build docs
Hugging Face Hub GGUF docs
Hugging Face Hub GGUF with llama.cpp

Why this is hard on Orin Nano 8GB

The key issue is not just parameter count. It is the whole runtime memory picture.

Component	Why it matters on Orin Nano 8GB
GGUF model weights	The quantized file still has to be mapped/loaded and used by the runtime. An 8GB-class GGUF is already too close to the total memory pool.
KV cache	Grows with context length, number of layers, KV heads, cache dtype, and parallel slots. Long context is expensive even if the weights fit.
CUDA buffers	GPU offload needs extra temporary and persistent CUDA allocations.
OS + desktop + services	They consume memory from the same 8GB pool.
File cache / mmap behavior	Can help or hurt depending on pressure; it does not create more physical RAM.
Swap	Can prevent crashes, but if actual inference is paging heavily, performance can become unusable.
CPU offload	On a desktop with 32GB+ RAM, this can be a real escape hatch. On Orin Nano 8GB, CPU and GPU share the same small memory pool.

This is why a model that is “interesting on a 32GB RAM PC” can still be a bad fit for Orin Nano 8GB.

Mellum2-specific issue

The model itself is interesting. It is not a toy model.

JetBrains/Mellum2-12B-A2.5B-Instruct is described as a coding/software-engineering model with:

12B total parameters
2.5B active parameters per token
MoE with 64 experts and 8 active experts
131,072-token context
GQA with 32 Q heads and 4 KV heads
software engineering, code generation/editing, tool use, function calling, and agentic workflows as core target use cases

Sources:

JetBrains/Mellum2-12B-A2.5B-Instruct model card
Mellum2 Technical Report

But for Orin Nano 8GB, the important part is:

12B total weights still matter. “2.5B active” helps per-token compute, but it does not magically make the full model fit like a 2.5B dense model.

That is also why larger Qwen MoE models can be interesting on a 32GB RAM CPU box , but not necessarily on Orin Nano 8GB. On a PC, llama.cpp can sometimes make good use of system RAM. On Orin Nano 8GB, the “system RAM” is still the same tiny shared pool.

Practical interpretation of NVIDIA/Jetson guidance

NVIDIA’s own forum practical guide says that, using llama.cpp with Q4_K GGUF, Orin Nano 8GB can fit approximately:

Class	Approximate upper range from the guide
LLMs	up to around 10B parameters
VLMs	up to around 4B parameters

Source: AI Models That Run on Jetson Orin Nano Super 8GB — A Practical Guide

I would read that as an upper-bound / best-case sizing guide , not as “10B will be comfortable”.

A more conservative operational reading is:

GGUF size / model class	My expectation on Orin Nano 8GB
<= 2GB	Usually comfortable if the runtime is configured correctly.
2GB–4GB	Best practical target zone.
4GB–5.5GB	Possible, but context/batch/KV/cache/offload settings matter a lot.
5.5GB–7GB	Experimental; expect tuning and instability.
7GB+	Usually not a good practical recommendation on this board.
8GB-class GGUF	The model file itself is too close to the total memory pool. Expect CUDA allocation failures, very short context, or unusable CPU fallback.

So, for this board, I would not start with a 12B MoE Q4 model. I would start with 2B–4B and then test 8B low-bit only if the baseline is stable.

First thing to check: JetPack / L4T version

There is a very relevant recent NVIDIA forum thread where Gemma 4 E4B Q4_K_M failed with CUDA OOM on Orin Nano, then NVIDIA identified a known memory issue in r36.4.7 that was fixed in r36.5.

Source:

Gemma4 E4B on Jetson Orin Nano fails due to CUDA out of memory issue

So before changing models, I would check the Jetson software stack:

cat /etc/nv_tegra_release
dpkg-query --show nvidia-l4t-core

If this shows an affected JetPack/L4T combination, update the runtime first. Otherwise model-level debugging can be misleading.

Recommended baseline: use the Jetson-oriented llama.cpp container first

Before doing custom builds, I would first try the NVIDIA AI IoT / Jetson AI Lab llama.cpp container.

The NVIDIA forum guide points to this container tag:

ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin

Source:

NVIDIA forum practical guide

Jetson AI Lab’s Gemma 4 E2B page also says the model is configured to run on Jetson with vLLM and llama.cpp, and describes E2B as the edge-first low-memory member of the Gemma 4 family:

Jetson AI Lab — Gemma 4 E2B

A conservative smoke test would be something like:

sudo docker run -it --rm --pull always \
  --runtime=nvidia \
  --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
  llama-server \
    -hf unsloth/gemma-4-E2B-it-GGUF:Q4_K_S \
    -c 1024 \
    -ngl auto \
    -fa on \
    -ctk q8_0 \
    -ctv q8_0 \
    -b 64 \
    -ub 32 \
    --host 0.0.0.0 \
    --port 8080

This is not because Gemma 4 E2B is necessarily the best coding model. It is because it is a good known-good Jetson baseline.

If this fails, I would suspect the Jetson stack, container/runtime, CUDA setup, power mode, memory pressure, or JetPack/L4T version before blaming Mellum2.

Runtime tuning checklist

These are the knobs I would try, in order.

1. Reduce context length aggressively

Do not start with 8K, 32K, or 128K context on Orin Nano 8GB.

Start with:

-c 512

Then only increase if stable:

-c 1024
-c 2048

For a very marginal model:

-c 256

In llama.cpp, --ctx-size directly affects KV cache memory. Long context is not free.

Reference:

llama.cpp server docs

2. Reduce batch and micro-batch

For memory-constrained Jetson runs, I would not use large defaults.

Try:

-b 64 -ub 32

If it still fails:

-b 32 -ub 16

This may slow prompt processing, but it can avoid allocation failures.

Reference:

llama.cpp server docs

3. Quantize the KV cache

Try safer KV cache quantization first:

-ctk q8_0 -ctv q8_0

If the model is still too tight:

-ctk q4_0 -ctv q4_0

I would not start with q4_0 unless memory is really tight. It can be less robust, but it is worth trying for a last-resort fit test.

Reference:

llama.cpp server docs

4. Use `-fit` / fitting mode when available

Recent llama.cpp builds have model-fitting behavior in the server path. The Gemma 4 E4B NVIDIA forum log explicitly shows:

common_init_result: fitting params to device memory
llama_params_fit_impl: projected to use 5533 MiB of device memory vs. 6387 MiB of free device memory

Source:

Gemma4 E4B CUDA OOM thread

So I would let llama.cpp fit parameters when using the Jetson container, unless debugging a suspected fitting bug.

5. Do not blindly maximize GPU layers

-ngl 999 or --n-gpu-layers 99 can be fine when memory is enough. On Orin Nano 8GB it can also push the system over the edge.

Try:

-ngl auto

or step manually:

-ngl 8
-ngl 16
-ngl 24

If CPU-only works but GPU mode fails, the model may be too tight for the CUDA path even if the weights can be mapped.

6. Try MoE CPU expert offload only as an experiment

llama.cpp has MoE-related CPU offload options such as --cpu-moe / --n-cpu-moe in recent builds.

This can help on systems with real CPU RAM headroom, for example:

desktop PC with 32GB/64GB RAM
small GPU with large system RAM

On Orin Nano 8GB, the CPU and GPU share the same limited memory pool, so this is not a true escape hatch. Still, for MoE models, it may be worth testing:

-ncmoe 20

But I would not expect it to rescue an 8GB-class Mellum2 GGUF on an 8GB Jetson.

Reference:

llama.cpp server docs

7. Avoid `--mlock` on low-memory Jetson

--mlock can be useful on some systems, but on an 8GB Jetson it can make memory pressure worse by preventing paging.

In this case I would avoid:

--mlock

Reference:

llama.cpp server docs

8. Try `--no-mmap` only as a late diagnostic

mmap behavior can interact with page cache and memory pressure. I would keep the default first, then try:

--no-mmap

only if debugging load behavior.

Reference:

llama.cpp server docs

System-level memory tuning

Jetson AI Lab has a specific RAM optimization guide:

Jetson AI Lab — RAM Optimization

The jetson-containers setup guide also has practical advice on swap and disabling the desktop GUI:

jetson-containers setup guide

Disable the desktop GUI

If using the desktop environment, stopping it can free a meaningful amount of memory.

Temporary:

sudo init 3

Restore:

sudo init 5

For Orin Nano 8GB, even a few hundred MB can matter.

Add NVMe swap

Swap is not a performance solution, but it can prevent immediate OOM kills and help identify whether the failure is a hard fit problem.

Example:

sudo systemctl disable nvzramconfig
sudo fallocate -l 16G /mnt/16GB.swap
sudo mkswap /mnt/16GB.swap
sudo swapon /mnt/16GB.swap

Persist in /etc/fstab:

/mnt/16GB.swap none swap sw 0 0

Prefer NVMe over SD card for swap if available.

Clear caches before a load test

For repeatable testing:

sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

This does not fix the model, but it makes memory-pressure tests cleaner.

Use max power / clocks after the model actually fits

Power mode will not make an oversized model fit, but it matters once the model runs.

Check and set power mode:

sudo nvpmodel -q
sudo nvpmodel -m <mode>
sudo jetson_clocks

The exact mode depends on the installed JetPack / board configuration.

If building llama.cpp yourself

If not using the Jetson container, build with CUDA explicitly.

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=87

cmake --build build --config Release -j

Orin is compute capability 8.7 / SM87, and NVIDIA forum logs for Orin show ARCHS = 870.

References:

llama.cpp build docs
Gemma4 E4B CUDA OOM thread showing Orin compute capability 8.7

If experimenting with memory-constrained CUDA behavior, these build options may be worth knowing about:

-DGGML_CUDA_FORCE_MMQ=ON
-DGGML_CUDA_FA_ALL_QUANTS=ON

But I would first test the NVIDIA container before going deep into custom builds.

A realistic Mellum2 last-try profile

If you still want to try Mellum2 just to see whether it can load, I would use an extreme low-memory profile.

This is a load experiment , not a practical recommendation:

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
  -m ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf \
  -c 256 \
  -ngl auto \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  -b 32 \
  -ub 16 \
  --host 0.0.0.0 \
  --port 8080

If your llama.cpp build supports MoE CPU expert offload:

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
  -m ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf \
  -c 256 \
  -ngl auto \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  -b 32 \
  -ub 16 \
  -ncmoe 20 \
  --host 0.0.0.0 \
  --port 8080

Again, I would not expect this to become a good daily-use setup. If it loads but runs very slowly, that is still useful information: it confirms that the bottleneck is not simply “unsupported model”, but practical memory and bandwidth limits.

Better model targets for this board

For Orin Nano 8GB, I would pick models by actual GGUF size and runtime stability , not leaderboard score alone.

Practical shortlist

Model family	Why it is more realistic than Mellum2 Q4	Suggested quant range
Gemma 4 E2B	Explicit Jetson AI Lab support; edge-first low-memory target.	Q4_K_S / Q4_K_M
Gemma 4 E4B	Supported on Jetson, but already near the upper edge on Nano 8GB.	Q3 / Q4, short context
Qwen 4B-class models	Good instruction/coding/tool-use balance if GGUF support is current.	Q4 / IQ4 / Q3
Qwen2.5-Coder-3B	Older but still a useful coding-specific baseline.	Q4 / Q5
StarCoder2-3B	Better for FIM/completion-style coding than chat-agent use.	Q4 / Q5
LFM2 / LFM2.5 8B-A1B	Edge-oriented MoE; interesting if Q3/IQ4 GGUF fits.	Q3 / IQ4
Granite Code 3B	Practical code model, enterprise-friendly posture.	Q4 / Q5
OpenCoder 1.5B/8B	Code-specialized candidate; 8B needs low-bit.	1.5B Q5/Q8, 8B Q3

Models I would not recommend first on Orin Nano 8GB

Model type	Why I would avoid it first
Mellum2 Q4_K_M	12B total MoE; Q4 file is too close to / above practical memory budget.
12B+ dense models	Even if they load, context and speed will likely be poor.
30B-A3B / 35B-A3B MoE	Interesting on 32GB RAM PCs, not on 8GB shared-memory Jetson.
Qwen3-Coder-Next 80B-A3B	Very interesting model, wrong memory class for this board.
Devstral-style 20B+ coding agents	Good benchmark story, wrong memory budget for Orin Nano 8GB.
Long-context runs	Context length will often fail before “model intelligence” matters.

Suggested debugging sequence

I would debug in this order.

Step	Goal	Command / action
1	Confirm JetPack/L4T	`cat /etc/nv_tegra_release`
2	Confirm available memory	`free -h`, `tegrastats`
3	Stop desktop GUI	`sudo init 3`
4	Add NVMe swap	16GB swap on NVMe if available
5	Test known-small GGUF	Gemma 4 E2B or a 2B–3B GGUF
6	Test 4B-class model	Qwen 4B / Gemma E4B / coder 3B
7	Tune ctx/KV/batch	`-c 512`, `-ctk q8_0`, `-b 64`, `-ub 32`
8	Try 8B low-bit	only after baseline is stable
9	Try Mellum2	only as a load experiment
10	Decide	if small models work but Mellum2 fails, the conclusion is memory budget, not setup failure

Example “known-good first” profile

sudo docker run -it --rm --pull always \
  --runtime=nvidia \
  --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
  llama-server \
    -hf unsloth/gemma-4-E2B-it-GGUF:Q4_K_S \
    -c 1024 \
    -ngl auto \
    -fa on \
    -ctk q8_0 \
    -ctv q8_0 \
    -b 64 \
    -ub 32 \
    --host 0.0.0.0 \
    --port 8080

If this works, move to a 3B/4B coding model.

If this fails, Mellum2 is not the right next test. Fix the Jetson runtime first.

Example “4B-class” profile

sudo docker run -it --rm --pull always \
  --runtime=nvidia \
  --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
  llama-server \
    -hf <repo>:<quant> \
    -c 512 \
    -ngl auto \
    -fa on \
    -ctk q8_0 \
    -ctv q8_0 \
    -b 64 \
    -ub 32 \
    --host 0.0.0.0 \
    --port 8080

If it fails:

-c 256
-ctk q4_0
-ctv q4_0
-b 32
-ub 16

If it only works with -c 256, it may technically run but not be useful for coding assistance.

How I would summarize the recommendation

For Orin Nano 8GB, I would frame it this way:

Mellum2 is an interesting coding MoE, but the Orin Nano 8GB memory budget is probably the wrong target for the Q4_K_M GGUF. The board has 8GB shared memory, so CPU offload is not the same escape hatch that it is on a 32GB RAM PC. I would first validate the Jetson stack with NVIDIA’s Jetson-oriented llama.cpp container and a known-small model such as Gemma 4 E2B. Then I would test 3B–4B coding/instruct models, or possibly 8B-A1B / 8B low-bit models. I would only try Mellum2 with very short context, quantized KV cache, tiny batch sizes, and possibly unified-memory fallback as a load experiment.

Resource list

Jetson / NVIDIA

Jetson AI Lab — Models
Jetson AI Lab — Gemma 4 E2B
Jetson AI Lab — Gemma 4 E4B
Jetson AI Lab — RAM Optimization
NVIDIA Developer Forums — AI Models That Run on Jetson Orin Nano Super 8GB
NVIDIA Developer Forums — Gemma4 E4B CUDA OOM on Orin Nano
NVIDIA Technical Blog — Bringing AI Closer to the Edge and On-Device with Gemma 4

Containers / setup

jetson-containers GitHub repo
jetson-containers setup guide
NVIDIA AI IoT packages

llama.cpp / GGUF

llama.cpp GitHub
llama.cpp server docs
llama.cpp build docs
Hugging Face GGUF docs
Hugging Face GGUF with llama.cpp

Mellum2

JetBrains/Mellum2-12B-A2.5B-Instruct
Mellum2 Technical Report
JetBrains Mellum2 collection

Model alternatives to check

Gemma 4 E2B on Jetson AI Lab
Gemma 4 E4B on Jetson AI Lab
Qwen models on Hugging Face
Liquid AI models on Hugging Face
StarCoder2 docs in Transformers

Final practical take

I would not spend too much time trying to force Mellum2-12B-A2.5B-Instruct-Q4_K_M onto an Orin Nano 8GB.

I would do this instead:

Make sure JetPack/L4T is not on a known-problem release.
Use the Jetson-oriented llama.cpp container.
Disable GUI and add NVMe swap if possible.
Validate with Gemma 4 E2B or another 2B–3B GGUF.
Try 3B–4B coding models.
Try 8B low-bit / 8B-A1B edge-MoE models only after that.
Treat Mellum2 as an experiment, not the practical target.

For this board, the useful question is probably not:

“Can I run a 12B MoE somehow?”

It is more:

“Which 2B–4B or low-bit 8B model gives the best coding usefulness per GB of actual Jetson memory?”

Short version

Why this is hard on Orin Nano 8GB

Mellum2-specific issue

Practical interpretation of NVIDIA/Jetson guidance

First thing to check: JetPack / L4T version

Recommended baseline: use the Jetson-oriented llama.cpp container first

Runtime tuning checklist

1. Reduce context length aggressively

2. Reduce batch and micro-batch

3. Quantize the KV cache

4. Use -fit / fitting mode when available

5. Do not blindly maximize GPU layers

6. Try MoE CPU expert offload only as an experiment

7. Avoid --mlock on low-memory Jetson

8. Try --no-mmap only as a late diagnostic

System-level memory tuning

Disable the desktop GUI

Add NVMe swap

Clear caches before a load test

Use max power / clocks after the model actually fits

If building llama.cpp yourself

A realistic Mellum2 last-try profile

Better model targets for this board

Practical shortlist

Models I would not recommend first on Orin Nano 8GB

Suggested debugging sequence

Example “known-good first” profile

Example “4B-class” profile

How I would summarize the recommendation

Resource list

Jetson / NVIDIA

Containers / setup

llama.cpp / GGUF

Mellum2

Model alternatives to check

Final practical take

Discussion in the ATmosphere

4. Use `-fit` / fitting mode when available

7. Avoid `--mlock` on low-memory Jetson

8. Try `--no-mmap` only as a late diagnostic