Two of the most interesting open-model releases of the quarter just landed, and they share a spec-sheet line that has a lot to do with benchmarks – but ignores the “Engram” part.
DeepSeek’s V4.1-Flash — a chonky 552-billion-parameter MoE with a 1M-token context — ships with something called Engram conditional memory that’s still a bit chonky for local folks like us: 196 billion parameters. And those of us who fit DeepSeek V4 Flash Q4 across two Strix Halos have opinions about what that implies for a single box. And Qwen’s 3.8-Flash-Next, the experimental preview of the architecture that will underpin Qwen4, loads up 51 billion parameters of N-gram Embedding: a 20-million-entry table of multi-word fragments.
That’s a lot of parameters doing neither attention nor expert-routing. So the starting point everyone is starting to question: What IS an Engram?? It’s Big and does what exactly?
So what the hekk is an engram?
The word is stolen from neuroscience, and was technically originated from Deepseek’s research papers, but it’s a good theft – intellectually.
An engram is the physical trace a memory in the “Brain” — a group of neurons that, when they light up together, re-invoke an experience. It’s why you don’t re-derive your phone number every time someone asks (or their birthdays, or that amazing date – once upon a time); the whole sequence is stored, and you retrieve it WHOLESALE.
In AI-land, it’s an LLM engram is the same idea in silicon: a stored fragment of language you retrieve instead of compute. Instead of every token passing through the full attention-and-experts machinery, some tokens — the ones where nothing interesting is being decided — get matched against a giant table and emitted from memory, nearly for free.
Which brings us to the part of this post we’ve been waiting to write.
Once upon a time — and why everything after “a time” is assumed
Consider: “Once upon…“
You already know what comes next. “…a time“. There is no creativity happening there. No reasoning, no weighing of alternatives, no 8-billion-parameter council debating whether the sentence should continue with “a time” or “a Saturday in 1997”. When “once upon” shows up, the next two tokens are decided by the English language, not by the model.
Did you really need to infererence “a time”? IT’s brilliant, and simple – those tokens cost exactly as much to produce as the tokens where the model actually thinks – but you didn’t really need it in the first place cause the ending tokens were always definitive (Yes, you can be creative – but you’d need to pay tokens for that).
That’s the insight behind the engram, and it generalizes far beyond fairy tales:
- “as a matter of” → fact — decided.
- “at the end of the” → day — decided.
- “in the middle of the” → night — decided.
- “it goes without” → saying — decided.
- “one of the most” → common / important / popular… — a short menu, essentially decided.
A huge fraction of any human-generated sentence is this kind of glue: pre-arranged collocations, idioms, connective tissue.
The words where a writer — or a model — makes a real choice are surrounded by words whose only job is to be grammatically correct and statistically inevitable. Human language is highly redundant, and that’s a feature: it’s why you can skim books (or e-books), why you can talk on a bad phone line and somehow the other person “gets you”, and why a lookup table of multi-word fragments can substitute for inference on a large slice of tokens.
So the engram design says: keep the expensive machinery for tokens where choice lives, and handle the inevitable ones with retrieval. If it’s static, it won’t need to be inferenced, it only needs to be pulled from a data-base of commonalities & assumptions of the language.
Why four words? The 4-gram sweet spot
Now the question in the title. Why or what is so special about this magical “4-grams” — sequences of four tokens looked up as a unit. Why four?
Watch what happens as the context grows:
| n-gram | Example | Predictable? |
|---|---|---|
| 2 (bigram) | “once upon” | Often, but not always — “once more”, “once you” |
| 3 (trigram) | “once upon a” | Warmer |
| 4 (4-gram) | “once upon a time” | Locked in — essentially certain |
| 5 | “a matter of fact is” | Also locked… but see below |
Statistically and in the history of Machine learning, Four tokens are roughly where common phrases snap into certainty. Two or three tokens in (How are you is OK), real ambiguity remains; by four, most fixed expressions of English have committed to their continuation.
And — this is the why — going deeper gets exponentially worse instead of better. If your vocabulary is ~100K tokens, the space of possible 5-grams is 100,000⁵. YOUR human-training corpus doesn’t contain every possible 5-word sequence; it contains billions of 2-to-4-word sequences. Push the table much past four and most entries would have been seen once, which makes them statistics of one — no better than guessing.
Four (or three-to-four; Qwen’s shipped table actually indexes bigrams and trigrams, DeepSeek’s does token-based lookup more generally) is the sweet spot between the two failure modes: long enough that the phrase is deterministic, short enough that the table is dense with learned facts. This is the same conclusion the pre-transformer generation of language models hit in the speech-recognition era, when n-gram LMs also settled in the 3–5 range — some math doesn’t change just because the hardware does.
The economics of rational-irrationality
Okay — so why does Deepseek or Qwen want this? Because inference is billed per token, not per insight.
In a vanilla transformer, emitting “a time” after “once upon” burns the same prefill and decode cost as emitting a line of poetry. Multiply that across the glue words — realistically a third to a half of any text — and you’re running a frontier model’s entire energy bill on tokens where zero decisions were made.
The engram makes the glue nearly free: match the n-gram, emit the fragment from memory, and save the active parameters (and the memory bandwidth they’d burn) for the tokens where the model actually has to think.
Qwen’s model card is refreshingly explicit about the motive: embeddings are “a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts,” making parameter scaling efficient “for memory-constrained accelerators without sacrificing quality.”
In plain terms: you can grow what a model knows without growing what a model runs. A lookup table can be parked in slower memory and consulted lazily, while MoE experts want to sit in fast memory and be touched constantly. DeepSeek’s version — 196B of conditional memory accessed sparsely via token lookup — is the same bet at frontier scale: knowledge you consult, not knowledge you execute.
If you’re reading this site, you know exactly which constraint that relaxes. Local boxes live and die by memory bandwidth. Every token that arrives from a lookup table instead of the expert layers is a token that didn’t have to stream GBs of weights through your iGPU. On a 128 GB unified-memory machine, that’s the difference between scaling knowledge and scaling your electric bill. (Which explains why Qwen 3.8 Flash Next is a sweet spot on 96-128GB URAM – coming up next)
Why the Engram was bourne to live on SSD and DRAM
Famous last words – we’re not entirely sure where this engram dynamic will end, but… FOR NOW: Here’s the part that should make 128 GB owners “hopefully – slightly – happier” in the RAMageddon times: the engram is the first big chunk of a frontier model that’s designed to be evicted from fast memory.
Think about what your URAM holds today. Attention layers, MTP and active experts get touched on every single token — a missed fetch stalls decode, full stop. The KV cache has to stay within arm’s reach or your context window turns into a disk-thrashing slideshow. Those seats are reserved; nothing but the fast tier will do.
An engram is the opposite — consider how differently it behaves:
| Active weights / KV | Engram lookup | |
|---|---|---|
| How often touched | EVERY token, EVERY layer | Only when a phrase actually matches |
| Latency tolerance | None — it’s on the critical path | High — one fetch serves 3–4 tokens |
| Access pattern | Random, unpredictable | Zipfian — the same hot phrases over and over |
| Natural home | URAM/VRAM | DRAM, and the cold tail even SSD |
That last row is the payoff. Language is actually predictable (we wouldn’t be able to talk to each other otherwise right?): a handful of fragments (“once upon”, “as a matter of”, “in the middle of”) account for the overwhelming majority of lookups, while most of a 20-million-entry table sits in the long tail, rarely — maybe never — read. So you keep the hot core in DRAM, park the cold tail on the SSD, and prefetch lazily in the background while the reasoning core keeps decoding. One slower SSD fetch that returns “a time” wholesale is still far cheaper than streaming four tokens’ worth of expert weights through your iGPU. Retrieval forgives latency; decoding never does. That’s the whole trick.
Now run the numbers. Qwen’s table: 51B parameters at 4-bit is ~25 GB — a quarter of a 128 GB box’s URAM handed back to your KV cache. DeepSeek’s: 196B of Engram at 4-bit is ~100 GB, which means at frontier scale it cannot live in the fast tier at all — it was never meant to. That’s what Qwen means by “more amenable to offloading than Mixture-of-Experts”: the engram is the part of the model you’re allowed to push down the memory hierarchy, because it’s the part that can wait for the bus.
And if you’ve followed our ds4 guides, none of this is hypothetical on our machines. This really is the incremental step of philosophy: SSD streaming for weights that don’t fit, KV spilling to a --kv-disk-dir when context balloons.
The engram is just the first model component born for the cheap seats — knowledge that lives on the SSD and DRAM, and shows up only when it’s called.
The Engram’s Future
Fast-forward a few AI-cycles, Engrams might be your Github repository of memories e.g. it remembers the stylistic grammar you prefer, memories or specific quotes that “Complete you” or that specific coding preference you may have. The pattern across both releases is simple: the way models scale is changing from “run more” to “remember more.”
The capability scale is changing from just simply “give me more chonkz and density” into that proverbial sweet-spot of chonkiness/intellect but with memory to boot.
For cloud providers, that’s a margin story. For the rest of us, it’s more than that (3-5-engram): it’s another round of “the hardware you bought last year just got smarter” — models that know more than last quarter’s, while running like something smaller. And if the lookup layer can live in slow memory, it’s one more reason that eventually, local compute will be the new homelab/gamerAI enthusiast benchmark. If you want to see that thesis tested today, our GLM 5.3 Flash on 128GB guide is as close as it gets.
