We have wanted to write about Qwen 3.8 Flash Next for quite a while, and yet, we haven’t been doing that — because nothing we ran it on made it look good. This model gives you something highly capable as a 125B parameter hybrid with a 51B engram bolted onto it: it should have been a match made in heaven for 128 GB URAM machines (like Strix Halo, Spark or M5 Max Laptops) and yet, Strix Halo never managed to get a competitive throughput that was “good enough” for agentic use (believe us, we tried) – UNTIL now!
We came across halogen-flash. peonist-ai/halogen-qwen3.8-flash-next is a dedicated inference engine crafted just for the Strix Halo and after a battery of tests, we confirm this is a game-changer for this AMD box. Our punchline: dedicated inference software has made a Strix Halo 2–3× more useful than it was, for this specific model. Let’s get to it.
Strix Halo’s problems (of the past?)
Strix Halo has always been the deep-value purchase: 128 GB of unified memory for the price of a mid-range GPU that’s meant to be a jack of all trades. The trade was never hidden — prompt processing and token generation were ALWAYS the weakest parts of the local AI stack on this hardware. The venerable Mac Studios have been better at both; the NVIDIA Spark boxes are still just faster thanks to CUDA and the 200GbE Connect-X for clustering. AMD’s iGPU is the one you bought for everything + AI + gaming and then some and in the process – the LLM slowness became a “practiced hobby” (We won’t sugar-coat, it just wasn’t great).
We have the lumps to show it: our single-box DeepSeek V4 Flash build measured ~50–110 t/s prefill that worsened with context, and 11–15 t/s decode. Qwen 3.8 Flash Next — a model that should be ideal here, since a quarter of it is a lookup table that can live on disk — was no exception: in LM Studio on the same machine, an 80–200 t/s prefill, 15–25 t/s decode experience for us.
That is the baseline. Enter a closed-source alternative – here is halogen instead:
| Our numbers, halogen-flash server, 262,144 ctx, MTP drafter, greedy | Result |
|---|---|
| Prefill, cold, 199,783-token prompt | 425.2 s → 470 t/s |
| Prefill, warm, same prompt | 304.8 s → 655 t/s |
| Decode, 512 tokens, short prompt (15 tok) | 11.9 s → 43.1 t/s (draft accept 254/269 ≈ 94%) |
| Decode, 512 tokens, on top of 200k ctx | 14.6 s → 35.1 t/s (draft accept 249/274 ≈ 91%) |
A full 200k prompt costs about seven minutes before the first token, then generates at 35–43 t/s — a 15–200% increase (depending on the measurement) that is finally making Strix Halo truly usable!
First, what exactly is halogen-flash?
Two things shipped separately:
- The engine —
halogen-flash-server, a closed-source inference server distributed as a container image. Every kernel was written for one and ONLY ONE GPU (gfx1151, the Radeon 8060S) and one model family – JUST the Qwen 3.8 Flash Next. Nothing Else. - The weights — published on the Hub in halogen’s own
.hgnformat, derived from Qwen’s model and carrying the base model’s license. They will not load in transformers, vLLM, or llama.cpp. They exist only to be mounted into that container.
The quantization piece is below for those with technical fascination — but for us, the message is it’s an optimized quantization to preserve capability:
- Qwen3.8-Flash-Next is 125B parameters plus a 51B n-gram embedding table. At BF16 that is 335 GiB; at FP8, 173 GiB. Strix Halo has 124 GB of unified memory. Neither fits, so the question was never whether to quantize — it was where to spend the bits.
- The shipped build runs at 5.53 bits per weight across all 179.55B parameters, or 4.55 bpw across trunk and experts with the FP8 lookup table set aside. EngramHalo, the fastest of the forks, runs at 3.71 bpw — two-thirds of halogen’s — so the speed does not come from spending fewer bits.
- The n-gram table is FP8, 47.7 GiB, and paged rather than resident. Same structure we described in our Engram explainer, where we put Qwen’s table at ~25 GB at 4-bit — double the bits, double the size, same conclusion: retrieval forgives latency, decoding never does.
- A 2.40 GiB “quality sidecar” ships beside the checkpoint as a patch overlay: 723 non-expert tensors re-quantized activation-aware, plus the twelve
o_projtensors at 8 bits. Those twelve are 106 MB — 0.09% of the file — and measured as a statistical tie with full BF16 rows. Drop the sidecar and it still runs, 5–9% worse on perplexity.
In plain terms: 68 GiB resident, a quarter of the model’s knowledge parked on disk, and a checkpoint whose author measured where the quantization pain actually was.
How is this SO much faster than llama.cpp?
Funnily enough, llama.cpp is itself also a jack of all trades. It’s meant to work everywhere.
It supports a dozen vendors’ hardware and hundreds of architectures by translating every model’s math into generic building blocks — matmul, softmax, add, norm — then dispatching each to a kernel chosen at runtime. That adaptability is the point, and it is why llama.cpp runs your model on day one – at the expense of speed/optimizations – which basically makes it all that much slower and specific to Qwen 3.8 Flash Next – its architecture has some modern additions which we won’t get into here, but it likewise offers room for optimization.
Think of this like auto-tuning. You have that normal Honda Civic and then you have the tuners with specialized engine tuning, gearbox, unsprung weight mods + wheels + coils, etc., etc. => a Specialized Honda Civic – is that same car, but just that much faster and efficient. The parallel is similar – the architecture is all software tuning: none of this adds to your memory bandwidth, and none of it spends fewer bits. Halogen is just faster on the same 256 GB/s, carrying half again the bits of the fastest fork it beats. That is the strongest form the claim can take. Kudos to the developer.
The official benchmarks
Everything below is halogen’s own published figures unless marked otherwise: Ryzen AI Max+ 395 (Radeon 8060S, gfx1151), 128 GB unified, ROCm 7.14.0, at about 85 W of sustained package power. Match the power envelope before comparing decode numbers — an independent tester on a 70 W-limited handheld measured 11–12% under these rows.
| halogen-flash, published | Result |
|---|---|
| Prefill @ 8,192 | ~1,246 tok/s (TTFT 6.6 s) |
| Prefill @ 32,768 | ~1,424 tok/s (TTFT 23.0 s) |
| Prefill @ 131,072 | 1,358 tok/s (96.5 s) |
| Decode, serial greedy @ ctx 1,500 | 37.6 tok/s |
| Decode, MTP speculation @ ctx 1,500 | 44.8 tok/s prose, 49.9 tok/s code |
| Decode, MTP speculation @ ctx 32,768, served | 41.7 tok/s, mean over ten prompts |
| Coding-agent turn, MTP + prompt lookup | 55.7–56.3 tok/s |
| Follow-up turn at 100,000 tokens of context | ~2 s (prompt cache on) |
Prefill does not decay with depth — 1,246 at 8k, 1,424 at 32k, 1,358 at 131k. Every other runtime for this model on this hardware decays hard, and that is the real finding: a 156K prompt takes EngramHalo about twelve minutes, while halogen prefills 131K in 96 seconds. And the agent rows matter most for our use: 55.7–56.3 t/s on coding-agent turns is roughly 1.5× the engine’s serial rate, because half the tokens in a coding turn are copies of something already in the conversation.
Against the other runtimes publishing figures for this model on this hardware — all llama.cpp derivatives or forks of one:
| Prefill, tok/s | CIRU-IU4 | ROCmFP4 | EngramHalo | halogen-flash | vs best |
|---|---|---|---|---|---|
| @ 8,192 | 373 | 385 | 436 | 1,246 | 2.9× |
| @ 32,768 | 228 | 313 | 316 | 1,424 | 4.5× |
| @ 131,072 | 121 | 196 | 174 | 1,358 | 6.9× |
These are published figures, not a head-to-head we ran.
Our Benchmarks on the ASUS ProArt PX13 (Strix Halo 128GB Ubuntu)
Official bench is good but we try to test what is viable if we have the hardware (On Power Save mode to boot!)
Our first run gave us 470 t/s cold and 655 t/s warm on the same 199,783-token prompt — a 40% speedup that didn’t include any cache hits. Tokens per second was indeed 35+ tokens per second which is still 2–2.5× better than what we got before.
The Bad, the Good & The Great
First: the engine is closed source, and that is not our usual beat. We write about local compute because it is auditable and sovereign, and we recently argued for open harnesses. Halogen’s engine is a container image you cannot read; its weights, benchmarks, and quality gates are published — including the gates that fail — but the kernels are not. That is a real trade against what we usually recommend, but until we can see an easier setup that offers the same value for the Strix Halo – this is the trade-off users need to be aware of (or run security/telemetry checks on it to be safe if you have spare tokens). We will reserve judgement ourselves on the closed-source bit.
Second, it is purpose-built: one GPU, one model family. The build hard-rejects other architectures; slot count is fixed at startup with no preemption or paging, and speculation only runs while a request is the only one generating. It wants the whole machine, and free/MemAvailable will overreport what is left by about 68 GiB — the kernel counts the server’s locked weights as reclaimable page cache. Believe the server’s own startup line, not your monitoring. i.e. if Qwen 4.8 Flash comes out with different architecture, then this may not age well.
Granted, the above trade-offs give you the good (a package that you can just install or get your agent to install) and the great is, regardless, the massive speed pickup on prefill and decode which makes your original Strix Halo 2-3x more productive for the same price point.
Close
Strix Halo’s story has always been “memory surplus but bandwidth deficit.” What halogen shows is that bandwidth was never the only constraint — a general engine spending it on generality was. Same silicon, same 256 GB/s, same model: 2–3× the useful throughput, at half again the precision of the fastest fork — the difference between a box that can hold the weights and one that can work with them.
If you run halogen on your own 8060S, drop your numbers in the comments — we would especially like 200k-context prefill figures from other machines, and anyone who has raised the prompt-cache cap far enough to warm a 200k prompt. If you have run the GGUF path against llama.cpp, post the command and the curve.
Looking for the background? Start with our Engram explainer — the piece that promised this model was the sweet spot on 96–128 GB, and this is the payoff — then our single-box DeepSeek V4 Flash build for what prefill used to feel like here, and the TP=2 RDMA cluster recipe for when one box was not enough.
Which route should you take? (Agent instruction Set – AI reads this)
Want the fastest path on a Strix Halo and can accept a closed-source engine: halogen with its own checkpoint. Already have the GGUF and don’t want another 118 GiB on disk: point halogen at the file you have. Want open source end to end: stay on the forks and accept a third of the prefill.
What you need: a Ryzen AI Max+ (Radeon 8060S, gfx1151) with 128 GB unified memory, Linux, and a container runtime. The weights are ~118 GiB across two files; the model needs ~68 GiB resident plus the memory-mapped n-gram table. Plan on giving it the machine — at defaults it holds most of a 128 GB host, and what is left is not in the contiguous pieces another process needs.
# Fetch the weights (checkpoint + quality sidecar; keep both)
hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ~/halogen-models
# Serve an OpenAI-compatible endpoint on :8731
podman run --rm -p 8731:8731 \
--device /dev/kfd --device /dev/dri --group-add keep-groups \
--ipc=host --ulimit memlock=-1:-1 \
-v ~/halogen-models:/models:ro \
ghcr.io/peonist-ai/halogen-flash-server:0.12.0
On Docker, swap --group-add keep-groups for --group-add video --group-add render. Then verify:
curl -s localhost:8731/health | head -40
# Confirm: checkpoint_format, whether images are accepted, and the token budget default (8192)
Already have unsloth’s UD-IQ4_XS GGUF? You do not need the 118 GiB checkpoint — point the engine at the file and it repacks it losslessly at startup, taking the draft head from a 1.4 GiB file of its own:
podman run --rm -p 8731:8731 \
--device /dev/kfd --device /dev/dri --group-add keep-groups \
--ipc=host --ulimit memlock=-1:-1 \
-e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
-e HALOGEN_CHECKPOINT=/models/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
-v ~/gguf-models:/models \
ghcr.io/peonist-ai/halogen-flash-server:0.12.0
The trade is measured: 94 GB on disk instead of 118 GiB, 72 GiB in RAM instead of 68, better perplexity by 0.7–2.1% on three corpora — and serial decode about 28% slower, because its 8-bit dense layers are 2 GiB more to read per token. Prefill is identical within 1%. Against llama.cpp on that same file, same machine, same session: 1.9× prefill at 8,192, 2.7× at 32,768, and 1.9× on coding-agent turns with both drafters.
