Running DeepSeek V4 Flash Q2 on Strix Halo 128GB

AIfinitee belives local compute as an alternative to cloud and that the most capable open models should be runnable on hardware you actually own. This post is the recipe for one of them: DeepSeek V4 Flash at Q2, running on a single AMD Ryzen AI Max+ 395 (Strix Halo) box.

V4 Flash is a 284B-parameter Mixture-of-Experts model with only 13B active parameters per token, a 1M-token context window, and FP4+FP8 mixed precision — and it posts numbers that matter for vibe-coding and agentic work: 91.6 on LiveCodeBench (higher is better), 79.0 on SWE Verified, and a 3052 Codeforces rating. The 13B active footprint is exactly why it fits on one machine and still allow for a sufficient token generation speed for agentic use. Our own use showcases ~50-110 Prompt Processing and 10-15 tokens per second.

The punchline up front: it works, it’s really good, but it’s not the fastest. The engine doing the work is ds4 — antirez’s purpose-built DeepSeek V4 Flash inference engine — packaged for Strix Halo by kyuz0 in the strix-halo-ds4-toolbox. Both get full credit below; but first, the recipe.

What you need

• An AMD Ryzen AI Max+ 395 (Strix Halo) machine with 128 GB of RAM (the IQ2_XXS quantized weights are ~80.8 GB; you want headroom for context).
• Ubuntu 24.04+ with a ROCm-capable kernel (6.18.5+ recommended).
• The unified-memory boot parameters so the iGPU can allocate ~96-124 GiB. Add this line to the options Linux loads at startup:amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856

  On Ubuntu, apply it in four steps:

  1. sudo nano /etc/default/grub
  2. Find the line starting GRUB_CMDLINE_LINUX_DEFAULT= and append the four parameters above inside the quotes.
  3. sudo update-grub && sudo reboot
  4. After reboot, verify: cat /proc/cmdline should show amd_iommu=off.

  The BIOS-side setting (often labeled “UMA Frame Buffer Size,” “iGPU Memory,” or “Shared Display Memory” in BIOS menus) is hardware-specific — our two-node cluster guide walks through it with screenshots.

The recipe

Six steps. We’re on Ubuntu, so this uses distrobox rather than Fedora’s toolbox — the container image is the same either way.

Step 1 — Install distrobox and podman

sudo apt update && sudo apt install -y distrobox podman

That’s the only host-side install. Everything else lives inside the container.

Step 2 — Create the toolbox container

distrobox create \
  --name ds4-rocm-7.2.4 \
  --image docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4 \
  -- --device /dev/dri --device /dev/kfd \
  --group-add video --group-add render --group-add sudo \
  --security-opt seccomp=unconfined

distrobox enter ds4-rocm-7.2.4

Once inside, reset your PATH so the ROCm and ds4 binaries are found:

export PATH=/usr/local/bin:/opt/rocm/bin:/usr/bin:/usr/sbin:/bin:/sbin

The container ships ds4, ds4-server, and ds4-bench already compiled against ROCm 7.2.4 for the gfx1151 iGPU. If you’d rather not use the TUI, the CLI is all you need.

Step 3 — Download the model weights

mkdir -p ~/ds4

HF_XET_HIGH_PERFORMANCE=1 hf download antirez/deepseek-v4-gguf \
  DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --local-dir ~/ds4

That’s the IQ2_XXS imatrix quantization, ~80.8 GB. It’s the recommended default — quantized with an importance matrix calibrated on code and reasoning data, so it punches above its weight on the tasks you actually care about.

If you have RAM to spare and want higher quality, the hybrid Q2/Q4 variant (DeepSeek-V4-Flash-Layers37-42Q4KExperts-...-imatrix-fixed.gguf, ~97 GB) is in the same repo — but it leaves less room for context, so pick your tradeoff.

Step 4 — Run interactive chat

ds4 -m ~/ds4/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --ctx 32768

That’s it — you’re talking to DeepSeek V4 Flash on your own hardware. --ctx 32768 is a sane starting context; push it higher once you’ve confirmed everything works. We have tried 300-450k context and it still works fine (Vibe-coders will be happz).

Step 5 — Serve it

ds4-server \
  -m ~/ds4/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --ctx 124000

ds4-server exposes both an OpenAI-compatible /v1/chat/completions and an Anthropic-compatible /v1/messages endpoint on port 8000. Quick sanity check:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

The dual OpenAI + Anthropic compat is the part that pays off in the next section.

Step 6 — Benchmark it

ds4-bench \
  -m ~/ds4/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --prompt-file prompt.txt \
  --ctx-start 2048 \
  --ctx-max 65536 \
  --step-incr 2048 \
  --gen-tokens 128

This sweeps context from 2k to 64k in 2k steps and reports tokens/sec at each. You’ll see throughput drop as context grows — expected, and we’ll talk about what that means for agentic use below.

Where ds4 comes from

The engine under the hood deserves no small credit. ds4 is the latest in a line of work by antirez — the creator of Redis — who has been building Dwarfstar, a minimalist C-based inference engine in the same design philosophy as Redis: small, fast, few dependencies. ds4 is the DeepSeek-V4-Flash-specific build of that engine (source on GitHub, antirez’s blog).

What we’re running here is kyuz0’s packaging of ds4 — the main branch, compiled against ROCm 7.2.4 and dropped into a container tuned for the Strix Halo iGPU (strix-halo-ds4-toolbox) which helps users get up and running faster (note: ds4-cockpit install also provides a GUI if you need it)

Hooking it up to your tools

This is the part that actually sold us. Because ds4-server speaks both OpenAI-compatible and Anthropic-compatible APIs on port 8000, it drops into agentic harnesses with no template wrangling:

• Claude Code — point it at http://127.0.0.1:8000 as a local endpoint and it just talks to V4 Flash through the Anthropic-compatible /v1/messages route.
• Hermes Agent — same story; OpenAI-compatible /v1/chat/completions, no extra config.
• Any other client that speaks either API: Aider, Cline, Roo Code, your own curl script — all work out of the box.

The contrast worth naming: LM Studio is great, but getting a fresh model talking to an agent often means hunting down the right Jinja chat template and tweaking stop tokens / context settings until the model stops hallucinating system-role tokens. That’s real dev time, every model, every time. ds4-server’s native dual compatability skips that step entirely — the model is already templated correctly server-side.

Honest take

Really good. Not very fast.

The reason it fits on one box at all is the 13B-active MoE — only 13B of the 284B parameters light up per token, so per-token compute is modest and the ~80 GB of quantized weights slide into 128 GB of unified memory with room for context. That same architecture is why the quality holds up against much larger dense models on coding and reasoning benchmarks.

But “fits” is not “fast.” Throughput is modest in absolute terms, and it drops as context grows. On Prompt Processing under Claude Code / Hermes, expect speeds of ~50-110 tokens per second (lowers as context grows) and ~11-15 tokens per second, thanks to MTP settings (ds4-cockpit TUI allows you see select or download these) — the same pattern we saw in the cluster post. For interactive chat you’ll feel it. For agentic and vibe-coding workflows, where the model is doing real work between your inputs and latency is tolerable, the quality-per-wait is excellent. Treat this as a partner that “reverts to your requests as an assistant” machine, not a “give me an answer right now” machine (you can always pay up on cloud API for that).

What’s next

That’s the single-box recipe. If you want more throughput — bigger context, more concurrent agents, faster prefill — the natural next move is to take this same ds4 setup and spread it across two or more Strix Halo nodes; ds4 has native coordinator/worker roles for exactly that, and our two-node cluster guide covers the hardware side. Next up, we will showcase the DS4 on a Deepseek V4 Flash Q4 which runs selective 8-bit layers allowing you to have a decently accurate model at your finger-tips.

For now: one box, one model, and it works.