The Whale Fits: DeepSeek V4.1 Flash Q2 on Two Strix Halos

Sometimes you can’t help but wonder what happens when you start combining machines for AI. Today’s answer: DeepSeek V4.1 Flash Q2 — whale-sized at 340.6 GiB with close to frontier-class performance — running on two Strix Halo mini-PCs over a single direct-attach 25 GbE cable. In September we ran DeepSeek V4 Flash 0731 on this exact pair of boxes and published it as a cluster recipe; today is the sequel, on a newer ds4 build. Again, credit to Dwarfstar and kyuz0’s toolboxes for making this easy to run.

What our benchmarks found: a usable 404–448 t/s prefill at contexts up to ~100k, 297 t/s at 200k, and 13.3–15.4 t/s decode — all inside a 210,000-token context window, with no discrete GPUs. The tech stack has improved: that prefill runs 1.4–1.6× the old Q4_K build’s published 280 t/s headline, and long-context decode sits well ahead of our GLM 5.3 TP=2 reference run (5.9 t/s at 158k) — different harnesses and contexts, so treat both as direction, not ranking.

This build follows our setup: 2× Minisforum MS-1 MAX with 25 GbE NICs (Intel controllers) and an RDMA link (in theory you could do it over Thunderbolt too, with some agentic tweaking).

This is NOT a speedy LLM — it’s chonky even with that engram SSD offload — but there are occasions for it when you want more parameters (a larger knowledge base) than Qwen 3.8 27B or Flash Next. It’s a Q2, and Dwarfstar’s optimizations already make it usable in our runs: not the most accurate quant, but genuinely workable.

In this world of fast AI developments, this is another win for local compute users – you now have MiMo V2.6, GLM 5.3 Flash/Flagship, and Qwen 3.8 Flash Next/27B all at the fingertips of ONE Strix Halo machine — and two of them now give you either two LLM API endpoints for agentic work, or a cluster for higher-parameter models. As models get larger, the next logical move may be clustering Macs with NVIDIA GPUs, or Strix Halos with AMD/NVIDIA GPUs. That remains a longer-term project — the knowledge stack on this front is still thin — but it is worth exploring.

What you need (Agents Read This)

This post is the delta, not a from-scratch recipe: the BIOS carve-out, GTT kernel parameters, RDMA bring-up, and memlock work are identical to our TP=2 cluster post, and we won’t repeat them. What follows is what changed upstream, the two build gotchas that cost us time, the launch commands, and the fresh benchmark sweep. Let’s get to it.

• The cluster from our previous post: 2× Strix Halo mini-PCs (Ryzen AI MAX+ 395, 128 GB each) with the point-to-point 25 GbE RoCE v2 link already verified — ib_write_bw at line rate and memlock unlimited on both nodes. The two-node cluster guide covers the hardware underneath.
• podman, as before.
• The engine: kyuz0’s branch feat/rocm-deepseek41-halo (commit c77cb9c) of antirez’s ds4 — DeepSeek V4.1 support as staged for PR #1036.
• The model: DeepSeek-V4.1-Flash-Q2.gguf — a 340.6 GiB single-file GGUF (sha256 1ce6a8f8…6f42), staged on both nodes. The launch paths below are ours — swap in your own staging directory; what matters is that each node’s container can read its local copy of the same file.

What changed since the Q4_K build

Three things are different under the hood:

• New upstream. DeepSeek V4.1 support arrived in antirez’s ds4 via PR #1036, and we run kyuz0’s branch of it (feat/rocm-deepseek41-halo at c77cb9c) — the same packaging lineage as our single-box build, one generation forward.
• ROCm 10.0 container. The build image moves from the ROCm 7.14 toolbox to a ds4-build5:rocm10 image (ROCm 10.0 core plus the gfx1151 devel layer — see the gotcha below).
• The model itself. V4.1 Flash at Q2 is a 340.6 GiB file — more than twice the old 153 GiB Q4_K — which is exactly why this remains a two-box job.

Build gotcha: the ROCm 10.0 toolbox ships without HIP headers

The first build fails looking for HIP headers. The rocm-10.0 toolbox lacks them; add the devel layer in your Dockerfile:

RUN microdnf -y --nodocs --setopt=install_weak_deps=0 install \
      amdrocm-core-devel10.0-gfx1151 && microdnf clean all

That mirrors the one-line fix from the September build, bumped for ROCm 10.

Launching the cluster

Worker first (it retries until the coordinator appears), then the coordinator — both with GPU devices and the RDMA NIC passed through:

# ---- coordinator (node1, 192.168.50.164) ----
podman run -d --name ds41f-coord --network=host \
  --device=/dev/kfd --device=/dev/dri --device=/dev/infiniband \
  -v /srv/ds4-tp-v41:/work -v /srv/ds4-models:/srv/ds4-models:ro -w /work \
  localhost/ds4-build5:rocm10 ./ds4-server \
  -m /srv/ds4-models/v41/DeepSeek-V4.1-Flash-Q2.gguf --rocm \
  --tensor-parallel --role coordinator --listen 10.0.1.1 9911 \
  --transport rdma --rdma-device mlx5_1 --rdma-gid-index 3 \
  -c 210000 --host 0.0.0.0 --port 8010

# ---- worker (node2, 192.168.50.179) ----
podman run -d --name ds41f-worker --network=host \
  --device=/dev/kfd --device=/dev/dri --device=/dev/infiniband \
  -v /srv/ds4-tp-v41:/work -v /srv/ds4-models:/srv/ds4-models:ro -w /work \
  localhost/ds4-build5:rocm10 ./ds4 \
  -m /srv/ds4-models/DeepSeek-V4.1-Flash-Q2.gguf --rocm \
  --tensor-parallel --role worker --coordinator 10.0.1.1 9911 \
  --transport rdma --rdma-device rocep101s0f0 --rdma-gid-index 3 -c 210000

Read those launch lines carefully: the worker runs ./ds4, NOT ds4-server. The coordinator is the API-bearing process; the worker only holds its half of the model. Mixing them up is this build’s version of last month’s NIC-naming trap — the roles look symmetric and are not.

Also note the two --rdma-device values differ between boxes (mlx5_1 vs rocep101s0f0) — same kernel-version quirk as before, where identical hardware gets different interface names. Check ibv_devices on each node instead of copy-pasting.

When the coordinator prints that it’s listening, verify the API:

curl http://192.168.50.164:8010/v1/models

The API

One thing to know before plugging it in: this is a reasoning model. The chain-of-thought arrives in reasoning_content first and the answer in content after thinking closes — budget your max_tokens accordingly.

The numbers

Our measured sweep — this exact cluster, temperature 0, 56 generated tokens per sample, one sample per cell:

For calibration against everything else we’ve run on this hardware:

Two footnotes on those comparisons. First, the PR’s numbers come from its own benchmark harness at 100 GbE, ours from fresh-prefill sweeps at 25 GbE — direction, not ranking. The pleasant surprise is how little the cheaper link costs: ~5–10% on prefill versus the PR’s 100 GbE figures, because expert-transfer overlap hides most of the wire. Second, versus the old Q4_K cluster, our published numbers put V4.1 at 1.4–1.6× the harness headline and up to ~2.3× the server-side long-prompt figures — different harnesses, so direction not ranking. The shape of it is the point: a 340 GiB whale now prefills faster than the old 153 GiB build did at half the size.

The memory line is worth reading twice: at ctx 210000 with four cached sessions, each rank sits at 86 GiB of GTT out of ~124 GiB usable. There’s headroom left for longer context or more concurrent sessions — the KV economics that made the old build interesting carried forward.

Honest take

Decode is still not the reason you build this. At 13–15 t/s, generation feels like a batch job — you hand it long documents and come back for the answer. What TP=2 buys here is prefill (400-plus t/s through 100k context) and, above all, capacity: a 340.6 GiB Q2 file does not exist on one 128 GB box, full stop.

Two caveats, stated plainly: the sweep is one sample per cell at temperature 0 — directional numbers from our own runs, not an exhaustive study; and the PR’s published figures come from its harness at 100 GbE, so treat that row as context rather than a head-to-head.

One operational note from daily-driving this: these boxes are all-in. Bringing up V4.1 meant stopping our GLM 5.3 cluster (port 8001) to free memory — 128 GB nodes force you to choose what’s loaded. And per the PR’s own numbers, DSpark over two-node RoCE is not faster, so we skipped it.

If you need long-context prefill throughput on hardware you own, two Strix Halos and a 25 GbE cable now buy you 400+ t/s; if you need instant answers, the cloud API is still there.

What’s next

The obvious question is what this cluster does as a daily driver: concurrent sessions against the one server, or pushing context past 210k with the GTT headroom that’s left. We’re also curious how the V4.1 Flash quality trade at Q2 feels in real agentic work — the speed is one thing; whether the quant holds up over long coding sessions is another.

Running fewer boxes? Start with our single-box DeepSeek V4 Flash Q2 recipe for the ds4 basics, the two-node cluster guide for the hardware and network underneath this build, and our original TP=2 post for every BIOS, GRUB, and RDMA step this article skipped.