That box on our desk has not changed since we started AIFinitee.
Same Minisforum MS-1 Max with Ryzen AI Max+ 395, same 128GB URAM. What keeps changing is what those bytes are cognitively-worth.
The latest post is about kyojin, an inference engine from Yamz Labs that runs GLM-5.3-Flash (320B total, 18B active) and MiMo-V2.6-Flash (309B total, 15B active) on ONE 128 GB Strix Halo — a class of model that had no business being run on this machine at all – until this month.
Halogen already was a huge step-up for most Halo users — ~470 t/s cold prefill at 200k context and 35–43 t/s decode was decent (our test on power reduction modes).
On kyojin, we did manage to run GLM 5.3 Flash and MiMo V2.6: 300B-class MoEs decoding at 23–34 t/s — below official numbers, but still quite surprising given how large these models are. This is the URAM cognitive-value thesis from our halogen piece playing out again, one software-vibe later.
This article talks about what this is, and why/when to use big brainz vs. faster/lower parameter models.
First, what exactly is Kyojin — and what is EXL3?
- Kyojin is Yamz Labs’ inference engine for gfx1151 (ROCm 7), built on ExLlamaV3 by turboderp, with the first AMD ports by vcruz305 and sdougbrown. MIT licensed, engine and packs.
- EXL3 has been a mainstay in NVIDIA land for some time and is ExLlamaV3’s quantization format. In plain terms: weights stored at a chosen bits-per-weight, mixed per layer — selective quantization that keeps the model smarter than a linear method would (different tech than DwarfStar). The GLM pack takes turboderp’s 2.05 bpw tensors and swaps in 3.05 bpw expert layers for 29–44 — the end result: a 320B MoE lands at 99.7 GB with its MTP head included.
- Fidelity is decent relative to size. Against the official FP8 weights: MiMo top-1 agreement 91.99% (KLD 0.0713); GLM 89.31% — 90.11% on the 126 of 129 held-out rows that fit the memory guard — at KLD 0.151. You can say “HEY IT’S 10% LOSS” — but you probably couldn’t run these models at all without a second Strix Halo in TP=2 (no, kyojin doesn’t support distributed inference; we tried).
The numbers
Published figures from kyojin’s benchmarks doc — one Ryzen AI Max+ 395 (128 GB LPDDR5X), not a head-to-head we ran. Prefill at 8K; speculative decode across chat, prose and code prompts:
| Model | Pack size | Prefill @ 8K | Spec decode |
|---|---|---|---|
| GLM-5.3-Flash | 99.73 GB | 634 t/s | 26.2–33.4 t/s |
| MiMo-V2.6-Flash | 105.8 GB | 760 t/s | 28.9–37.0 t/s |
GLM’s prefill barely leans on depth — still 567 t/s at 256K (close to the max we can get without OOMing) — while MiMo’s published curve stops at 458 t/s for 128K (it crashed for us above 128K).
Set that against the baseline we already live with: halogen on this same machine gives us ~470 t/s cold prefill at ~200k context and 35–43 t/s decode (our numbers). That was respectable. A 320B MoE decoding at 23–34 t/s on a mini PC is not something we expected to write this year. This is not halogen vs. kyojin — the point is that Strix Halo users just got another bump in mainstream open-source LLM choices.
GLM vs. MiMo in Technical Terms
The two packs are closer than their badges suggest — 309B against 320B total — so the decision lives in the details:
- GLM-5.3-Flash drafts with its own MTP head, plus an unquantized projection file that lifts decode from 26.7 to 31.6 t/s greedy on Yamz’s protocol. The pack carries a vision tower (+0.35 GB) — it is the one that takes images in the server today.
- MiMo-V2.6-Flash drafts with DFlash, a separate 4 bpw drafter that proposes up to 7 tokens and keeps only the confident prefix. Code gains most: 28.9 t/s plain against 40.8 speculative, a 1.41× bump. It is also the cleaner copy — KLD 0.0713 against FP8, where GLM sits at 0.151.
- The memory split differs: MiMo slides a 128-token window across 39 of its 48 layers, so KV stays around 2.8 GiB even at 131072 context; GLM instead loads at
-c 524288if you hand it nearly all the RAM.
When we reach for which when spoiled for choice?
So, now you have Qwen 3.8 27B and Qwen 3.8 Flash Next from Halogen. You also have GLM 5.3 Flash and MiMo V2.6 Flash from kyojin — what a time to be alive!
Our rule of thumb (it’s been debated, but this is our view): a larger parameter model houses more “tacit knowledge” — more logic, knowledge and workflow to reference. You trade some accuracy (think ~10% “wrong” statistical output) for creativity; it won’t be as code-accurate, and some tasks will need more tokens.
In the same tune, that extra logic/creativity makes it the better brainstormer. Use it for planning and thinking, READ that output (yes, you should still read it), then hand it to your Qwens to implement with higher fidelity and accuracy. (Plan => Implement => Vibe.)
Looking for the background? Start with our halogen piece — the post that promised dedicated inference would keep making this box more useful, and this is the payoff — then GLM 5.3 Flash’s two easiest routes, our Engram explainer for why these architectures lean on lookup tables, and the single-box DeepSeek V4 Flash build for what prefill used to feel like here.
Which route should you take? (Agent instruction Set – AI reads this)
One line per route: GLM = vision input, biggest active slice; MiMo = best fidelity, code-heavy drafts; halogen = deepest context, closed source. Everything below installs kyojin from scratch on Ubuntu (ROCm 7.0+ required — 6.4 has no gfx1151 code):
sudo apt install -y g++ python3-dev python3-venv git
git clone https://github.com/Yamz-Labs/kyojin && cd kyojin
python3 -m venv .venv && source .venv/bin/activate
pip install --pre torch rocm-sdk-devel --index-url https://rocm.nightlies.amd.com/v2/gfx1151/
pip install -r requirements.txt
rocm-sdk init # expands devel headers, ~12 GB on disk
export EXL3_ROCM_SDK=$(rocm-sdk path --root)
source tools/strix_halo/env.sh # run from the repo root; sets LD_PRELOAD/PYTHONPATH
./build.sh # expect 8-10 minutes
bash tools/strix_halo/env.sh --check
# pick ONE pack:
hf download yamz-labs/GLM-5.3-Flash-EXL3-Yamz --local-dir ./glm-pack
python tools/glm/serve.py --model ./glm-pack --port 8000 -c 131072 --num-draft 2
hf download yamz-labs/MiMo-V2.6-Flash-MOPD-EXL3-Yamz --local-dir ./mimo-pack
python tools/mimo/serve.py --model ./mimo-pack --port 8000 -c 131072
# verify:
curl http://localhost:8000/v1/models
Expected outputs and gotchas:
- Every new shell: re-activate the venv, re-run the
export EXL3_ROCM_SDK=...line, re-sourceenv.sh. - A warning that the system compiler is in use means prefill runs ~20% slower — fix the export and restart.
- First GLM launch tunes dense GEMM kernels for about 10 minutes with the port closed (progress logged every 30 seconds); later launches start fast.
- MiMo prints one
Overriding ...line per overlay tensor at load — that is the tuning overlay, not an error. - If you see
undefined symbol: hsa_ext_image_create_v2, the HSA runtime is mismatched; use the ROCm 7libhsa-runtime64.so.1per kyojin’s install doc.
