Practical match for 128Gb Strix Halo with 2x3090s? (inference for coding)

Hmm…

---

There are two different questions mixed together here:

1. Can 2x3090s beat Strix Halo for dense coding models?
2. Can 2x3090s replace Strix Halo for gpt-oss-120b, especially at large context?

For the first one, your own result already says “probably yes”.

For the second one, I think the current test is not enough to say that.

Your Qwen result looks normal to me:

Qwen3.6-27B-Q8_0
Halo:   7.8 tok/s
2x3090: 24 tok/s

That is the easy case. A dense-ish model fits in fast NVIDIA VRAM, so the 3090 box wins. I would expect that for many 20B-34B coding models, and probably many 70B Q4 cases too, depending on context/KV.

But I would not read the gpt-oss-120b result the same way:

gpt-oss-120b-Q4_K_M
Halo:   56 tok/s
2x3090: 8.8 tok/s

gpt-oss-120b is not a normal dense 120B model. The model card says it is 117B total parameters, but only 5.1B active parameters per token. It is a MoE model, and the MoE weights are MXFP4. It is described as fitting on a single 80GB GPU.

huggingface.co

openai/gpt-oss-120b · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

That changes the problem.

For a dense model, a rough mental model is:

Can I keep most/all weights in fast VRAM?
If yes, NVIDIA dGPU probably wins.

For a big MoE model, the mental model is more like:

Which tensors are always active?
Which routed experts are active only some of the time?
Which parts are on GPU?
Which parts are on CPU/system memory?
How much KV cache is allocated?
How much PCIe traffic happens per token?
Does the backend understand this placement well?

That is why I would be cautious here. Strix Halo is not beating 3090 VRAM on raw bandwidth. It is much slower than GDDR6X in that sense. But it has a large unified memory pool. For a huge MoE model with offload-like behavior, that can matter more than the simple VRAM bandwidth comparison suggests.

So I would summarize your current numbers this way:

Dense model that fits in fast VRAM:
  2x3090 wins hard. Expected.

Huge MoE model with large memory footprint:
  not obvious. Halo may be a very good fit.

Your current 2x3090 gpt-oss result:
  probably not a fair upper bound yet.

The main reason I would not trust the 8.8 tok/s number as the final answer is the command:

./llama.cpp/build2/bin/llama-cli \
  -m /root/gpt-oss-120b-Q4_K_M-00001-of-00002.gguf \
  -c 128000 \
  -fa on \
  -ngl 23 \
  -sm row \
  -ts 1,1

Several things there make the 2x3090 setup look worse than it might be.

First:

-sm row

The current llama.cpp multi-GPU docs describe row as deprecated. They describe layer as the default and most compatible mode, and tensor as experimental but intended for lower token-generation latency where the model/backend/interconnect cooperate.

github.com/ggml-org/llama.cpp

docs/multi-gpu.md

master

# Using multiple GPUs with llama.cpp

This guide explains how to run [llama.cpp](https://github.com/ggml-org/llama.cpp) across more than one GPU. It covers the split modes, the command-line flags that control them, the limitations you need to know about, and ready-to-use recipes for `llama-cli` and `llama-server`.

The CLI arguments listed here are the same for both tools - or most llama.cpp binaries for that matter.

---

## When you need multi-GPU

Reach for multi-GPU when one of these is true:

- **The model doesn't fit in a single GPU's VRAM.** By spreading the weights across two or more GPUs the whole model can stay on accelerators. Otherwise part of the model will need to be run off of the comparatively slower system RAM.
- **You want more throughput.** By distributing the computation across multiple GPUs, each individual GPU has to do less work. This can result in better prefill and/or token generation performance, depending on the split mode and interconnect speed vs. the speed of an individual GPU.

---

## The split modes

Set with `--split-mode` / `-sm`.

This file has been truncated. show original

So I would not use row as the baseline for deciding whether the 2x3090 machine can replace the Halo.

Second:

-ngl 23

That is not a “try to use as much VRAM as possible” setting. It limits how many layers are offloaded to GPU. For a first baseline I would try:

--n-gpu-layers 999

or:

--n-gpu-layers all

and only then back down if it does not fit.

Third:

-c 128000

That is a brutal starting point for 48GB total VRAM. It means you are not only testing model throughput. You are also testing huge KV cache pressure, offload behavior, and memory placement all at once.

I would not start at 128k context. I would sweep context size:

-c 8192
-c 16384
-c 32768
-c 65536
-c 128000

If the 2x3090 setup is fine at 8k/16k/32k and then collapses at 64k/128k, that tells you something very different from “2x3090 is slow”.

Fourth, the rental VM is a big unknown.

For multi-GPU inference, topology can matter a lot:

PCIe layout
P2P availability
NCCL availability
NUMA placement
CPU memory bandwidth
virtualization overhead

On a rented VM, you may not know whether the two 3090s are attached in a sane way. I would at least check:

nvidia-smi topo -m
./llama-cli --list-devices

and watch the llama.cpp logs for things like:

NCCL is unavailable, multi GPU performance will be suboptimal

or any sign that much more of the model is on CPU than expected.

The other important point is that gpt-oss-120b should probably be treated as a MoE placement problem, not just a normal -ngl problem.

This guide explains the general idea well:

huggingface.co

Performant local mixture-of-experts CPU inference with GPU acceleration in...

A Blog post by Doctor Shotgun on Hugging Face

For MoE offload, the interesting idea is:

Always-active tensors:
  use them every token
  highest priority to keep on GPU

Routed experts:
  huge part of the model
  only a subset used per token
  may be more reasonable to offload partly to CPU/system memory

So for gpt-oss-120b, I would test MoE-aware options if your llama.cpp build supports them:

--cpu-moe

or sweep:

--n-cpu-moe 32
--n-cpu-moe 30
--n-cpu-moe 28
--n-cpu-moe 26
--n-cpu-moe 24

I would not assume those are the correct values. I would sweep and measure.

A more useful first baseline might look like this:

./llama-cli \
  -m /root/gpt-oss-120b-Q4_K_M-00001-of-00002.gguf \
  -c 8192 \
  -fa on \
  --n-gpu-layers 999 \
  --split-mode layer \
  --tensor-split 1,1

Then test MoE placement:

./llama-cli \
  -m /root/gpt-oss-120b-Q4_K_M-00001-of-00002.gguf \
  -c 8192 \
  -fa on \
  --n-gpu-layers 999 \
  --split-mode layer \
  --tensor-split 1,1 \
  --n-cpu-moe 28

Then sweep:

--n-cpu-moe 32
--n-cpu-moe 30
--n-cpu-moe 28
--n-cpu-moe 26
--n-cpu-moe 24

Then raise context:

-c 8192
-c 16384
-c 32768
-c 65536
-c 128000

If tensor split is supported for this model/build, I would test it too, but separately:

--split-mode tensor

I would not mix all variables at once.

My preferred order would be:

1. layer split, small context, max GPU layers
2. layer split, small context, MoE offload sweep
3. layer split, context sweep
4. tensor split, small context
5. tensor split, MoE offload sweep
6. tensor split, context sweep

And I would measure prompt processing and token generation separately.

For coding use, both matter:

Prompt processing:
  big prompts, repo snippets, logs, RAG, long context

Token generation:
  how fast the answer streams back

If one setup has great token generation but terrible prompt processing, it may feel bad for coding with long files. If another setup has slower generation but handles large context smoothly, it may be better for actual work.

So I would benchmark something like:

same prompt
same context size
same output token count
same sampling
same llama.cpp commit
same quant
same backend
same command except the variable under test

For example:

# 8k baseline
-c 8192

# 32k realistic coding/RAG-ish baseline
-c 32768

# 128k stress test
-c 128000

I would also separate these model classes:

Class A:
  dense model that fits in one 3090

Class B:
  dense model that needs both 3090s but mostly stays in VRAM

Class C:
  huge MoE / offload-heavy / high-context model

Your results already show why this matters.

For Class A / B, 2x3090 is likely excellent.

For Class C, Strix Halo may be surprisingly good.

That is the real point here. The systems are not exact substitutes.

The 3090 box is basically:

fast VRAM
CUDA ecosystem
great for dense models
great for gaming
great for smaller/faster coding models
annoying when the model exceeds VRAM
multi-GPU complexity when using both cards

The Halo box is basically:

much larger unified memory
less raw bandwidth than 3090 VRAM
much easier for very large models
possibly very good for MoE/offload-heavy workloads
not as strong for dense models that fit in NVIDIA VRAM

That matches your numbers.

So if your real daily workload is:

Qwen / Llama / DeepSeek Coder style dense models
20B-34B
70B Q4
gaming
CUDA tools

then I would lean toward the 3090 box.

If your real daily workload is:

gpt-oss-120b
large context
MoE experiments
"make the huge model fit without fighting placement all day"

then I would keep the Halo unless a tuned 2x3090 test proves otherwise.

If you can keep both, I think the cleanest split is:

Halo:
  gpt-oss-120b
  huge context
  MoE/offload experiments
  large-memory local inference

3090 box:
  dense coding models
  fast small/medium models
  CUDA backends
  gaming

If you want to replace the Halo, I would want the 2x3090 box to pass a more controlled gpt-oss-120b test first.

Something like:

No row split
Smaller context first
--n-gpu-layers 999/all
MoE offload sweep
Topology check
Prompt processing and generation measured separately
Same prompt / output length / sampling
Bare metal if possible, not unknown rental VM topology

If after that the 2x3090 setup gets close to or beats the Halo in your real gpt-oss-120b use case, then replacing the Halo becomes much more reasonable.

If it still loses badly, then I would not treat that as a surprise. It would just mean that gpt-oss-120b is landing in the exact niche where Strix Halo’s large unified memory is useful.

One more way to phrase it:

2x3090 is probably the better dense-model machine.

Strix Halo may be the better "large weird model" machine.

gpt-oss-120b is a large weird model.

So my answer to your original question would be:

Yes for many coding models, not proven for gpt-oss-120b.

And based on the numbers you posted, I would not sell the Halo yet.