Qwen 3.8 Flash Next has become the LLM of choice and daily driver for many local users. It looks big on the surface at 179B parameters but does come with a 96 GB engram offload — so in fact, you can run this not only on the Strix Halo, something we covered separately, but also on a 96 GB M3 or M5 Ultra Mac Studio at blazing speeds. Our tests and benchmarks below are based on the 96 GB Mac Studio M3 Ultra. Expect around 810 tokens per second on prefill and around 53 tokens per second at 200k context sizes — this is really nice speed and perfect for agentic use without it being too dumb.
With that, let’s dig in to see how to run this successfully and read further for our benchmarks.
First, what exactly is DwarfStar (ds4)?
As usual, our software stack needs no introduction. It’s again the familiar DwarfStar — antirez/ds4 on GitHub — which has continued to iterate upon Metal (Apple’s fix to CUDA). It’s a specific engine meant to run a smaller set of LLMs, with a good balance of selective quantizations to save space while maintaining the model’s bulk capabilities. In the past blogs, we used it to run — DeepSeek V4 Flash (plus its experimental vision model), V4.1 Flash, and for those with hardware the DeepSeek V4 PRO, GLM 5.2/5.3/5.3 Flash, and Qwen 3.8 Flash Next. The special thing about this model is the engram offload (which can be streamed via SSD), which lets you benefit from more intelligence/parameters for less memory.
In plain terms: what used to be impossible to fit without larger quantizations for 179B parameter models on the Mac Studio, now fits. For more on engrams, read What the hekk is an Engram?!,
What you need, and the memory math (Agents can use this)
What you need
- A Mac with 96 GB of unified memory or more. The same build covers M3 and M5, with hardware fast paths selected automatically.
- ~165 GB of free internal SSD for the Q4 checkpoint — the n-gram reads are random, so keep it on the internal drive, not a USB enclosure.
- macOS with the Xcode command-line tools (
xcode-select --install), thenmake. Nothing else: no container, no CUDA, no Python stack.
The memory plan
| Component | At 260,000 context |
|---|---|
| Resident weights (Q4_K gate/up, MXFP4 down, MTP included) | 69.7 GB |
| KV cache (6.5 GB raw + 1.8 GB compressed) | 8.3 GB |
| Runtime buffers | 11.2 GB |
| Total | 89.20 GB of 96 GB |
The 95.4 GB n-gram table is not in that table, and that is the entire trick. It lives in the same GGUF file, on disk, and the runtime pulls rows as it needs them.
The project’s own sizing puts resident weights at 69.7 GB for the Q4 build and 41.73 GB for the Q2 (165.11 and 137.10 GB downloads). Its memory guide points a 96 GB Mac at DeepSeek V4 Flash Q2 instead; we pointed ours at Qwen’s Q4 build.
Step 1 — update ds4 before anything else
DS4 needs commit 97ba327 or newer to include the engram reader.
cd ~/Documents/ds4
git stash # if you have local patches
git pull
make -j8
Verify: git log --oneline -1 should show a commit newer than 97ba327.
Step 2 — download the Q4 checkpoint
./download_model.sh qwen38-q4k --token <HF_TOKEN>
One 165.1 GB file containing the main model, the MTP weights and the original BF16 n-gram table; re-running the command resumes an interrupted download. The smaller qwen38-q2 target is the one for 64 GB machines.
Step 3 — raise the GPU wired limit
sudo sysctl iogpu.wired_limit_mb=92000
sysctl iogpu.wired_limit_mb # verify — prints 92000
The default cap is 84 GB and our plan needs 89.2 GB, so prefill dies with kIOGPUCommandBufferCallbackErrorOutOfMemory if you skip this. That is the ~91–92 GB figure you will see quoted; it leaves macOS roughly 4–5 GB (not great but workable). It also does not survive a reboot — re-apply it before every launch, which is what the verify line above is for.
Step 4 — launch headless
./ds4-server --ctx 260000 --mtp --port 8000 --host 0.0.0.0
--ctx 260000 sits just under the model’s native 262,144-token window, --mtp turns on speculative decoding with the MTP weights bundled in the GGUF, and --host 0.0.0.0 makes the server reachable on your LAN — the point of running headless, since it becomes an appliance you talk to from your laptop.
One caution before you copy that line: --host 0.0.0.0 exposes the server to every device on your network, and it has no authentication — anyone on the same network can use your model. Keep it on a network you trust, and put authentication and TLS in front of it before you ever expose it beyond that.
Verify the server is up and serving the model you think it is:
curl -s http://127.0.0.1:8000/v1/models | head -20
Then check the long-context recall path end to end, with the project’s own regression — start the server with at least --ctx 49152 for this one:
python3 tests/test_server_story.py --url http://127.0.0.1:8000 \
--model qwen3.8-flash-next --output /tmp/qwen-story
It plants sixteen facts in a 31K-token story, then asks a correction turn that has to reuse the prefix — and if your n-gram reads are failing, it tells you. So does tests/test_qwen4_ngram_state MODEL.gguf, which checks failed disk reads during prefill, decode and MTP, then exact recovery.
The numbers (not NVIDIA fast, but still very usable)
Our sweep, on our machine: Mac Studio M3 Ultra, 96 GB unified memory, macOS, Qwen 3.8 Flash Next Q4 (Q4_K gate/up experts, MXFP4 down), --ctx 260000 --mtp, 8,192-token prefill chunks, filler-text prompts, temperature 0, 200 output tokens per run.
| Context | Prefill avg | Prefill time | Decode | Total wall |
|---|---|---|---|---|
| 20k | 825 t/s | 24.6 s | ~55–60 t/s | 26.4 s |
| 50k | 846 t/s | 59.9 s | ~60–61 t/s | 62.2 s |
| 100k | 837 t/s | 120.9 s | ~54–59 t/s | 123.1 s |
| 200k | 810 t/s | 249.7 s | ~53–59 t/s | 253.3 s |
Three things are worth reading slowly:
Prefill is consistent even if context increases. 825 t/s at 20k, 810 t/s at 200k — a 2% decay across a tenfold context increase. Inside the 200k pass the per-chunk rate drifts from 936 t/s early to 766 t/s late, but the average holds: the table is on the SSD either way, so a deeper context does not make the reads worse.
Decode is flat too, at 53–61 t/s, and that is MTP. Speculation is what turns a bandwidth-bound model into something you can hold a conversation with, and here it does not degrade as the context fills.
A full 200k prompt costs about four minutes before the first token, then generates at ~55 t/s. The table below puts our other box’s numbers next to these.
To reproduce with the project’s own tooling — note that ds4-bench measures ordinary decoding and does not accept --mtp, so its decode rows will sit below ours:
./ds4-bench -m gguf/Qwen3.8-Flash-Next-Q4.gguf \
--prompt-file speed-bench/promessi_sposi.txt \
--ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128
One measurement note, because it fooled us for an afternoon: on filler-heavy prompts this model opens with “We need answer user’s request…”-style preamble text. Model behavior, not a server bug — but it costs you output tokens in a sweep like ours.
The same model on Mac vs. Strix Halo
We have run this model on two machines now, both ours, both at ~200k context, greedy — different engines and checkpoints, so read it as direction, never ranking — these are our own benchmarks:
| Qwen 3.8 Flash Next at ~200k, our numbers | Strix Halo, 128 GB (halogen; power savings mode) |
M3 Ultra, 96 GB (ds4) |
|---|---|---|
| Prefill, ~200k prompt | 470 t/s cold / 655 t/s warm | 810 t/s |
| Decode on top of ~200k | 35.1 t/s | 53–59 t/s |
| Checkpoint | halogen build, 5.53 bits per weight | Q4_K gate/up, MXFP4 down |
It may look like Strix Halo is somewhat slower, but again, that’s the memory bandwidth difference — and it is certainly usable.
Close
Engrams are a thing and will likely have more staying power in the coming releases of open-source models. The Mac Studios are now moving into pole position for inference, especially with the M5 Ultra series. In the meantime, those that have the 96 GB Mac Studio M3 Ultra, this is a great option of LLM to run with very decent capability. The memory you bought has “intellectually” appreciated again: same 96 GB, but more parameters!
If you run this on your own Mac, drop your numbers in the comments — we would especially like 100k-plus-context rows from other M-series machines, and anyone who has squeezed the qwen38-q2 build onto a 64 GB box.
Looking for the background? Start with our Engram explainer — the piece that promised this model was the sweet spot on 96–128 GB unified memory, of which this is the 96 GB end — then our halogen story for the same model on the other box we own, and GLM 5.2 at 4-Bit With MTP for the 512 GB end of the Mac Studio range.
