An infra-oriented diagram of the DeepSeek-V4.1-Flash architecture

DRAFT (unlisted study). Prose below is AI-drafted and awaits a human edit pass — see the disclosure footnote.¹

DeepSeek's V4.1-Flash (September 2026) is a 552B-parameter MoE, plus 196B of Engram lookup tables, that activates 8B parameters per prompt token and 16B per generated token. Three ideas do most of the work. A causal encoder–decoder split: the lower 20 layers are the encoder, and the upper 20 layers' global KV cache is projected once from the encoder's output rather than computed per layer, so prefill runs half the network. Compressed Sparse Attention 2: each layer's global attention is statically one of three modes — Full layers build a KV cache and an index, Reindex layers borrow the cache but pick their own top-512, Reuse layers borrow both — so only four of forty layers hold global KV. And Single-Pass mHC: the residual stream is four parallel copies, mixed by per-token coefficients around every sub-layer. Below is the same infra-oriented treatment the DeepSeek-V3 diagram got: the kernels you actually run, with their parameter counts. Primary sources: the V4.1 tech report, the checkpoint's config.json and reference inference/model.py, and the tensor shapes in its safetensors headers.

Reading notes, relative to the DeepSeek-V3 diagram:

• The attention enclosure has five tabs — one per static CSA2 role. The two SWA-only layers at the bottom of the stack see only a 128-token window. Every other layer attends over that window plus 512 selected entries of a global KV cache, and the tab decides where those entries come from. The plan on the left shows which layers wear which role; click a block to flip the diagram.
• There is one KV latent per token and it is shared by all 64 query heads (k = v, 512-wide — the "absorbed" MLA form made literal). Queries are low-rank (5120 → 1280 → 64×512), and the output projection is low-rank too, in 8 groups of 8 heads (4096 → 1024 each, kept in bf16), before a single 8192 → 5120 GEMM. RoPE touches 64 of the 512 dims, and is un-rotated on the attention output so the cache can stay in one rotated form.
• A Full layer's global KV is a second projection of the same normalized input. In the encoder (m=2) two consecutive tokens are pooled into one entry with a learned softmax gate; the decoder's single Full layer (layer 20) keeps one entry per token (m=1) and reads the encoder's final hidden state — this is the causal encoder–decoder: all twenty decoder layers share that one cache. The indexer's key is a 512 → 128 projection of the same latent, and the main cache is stored in FP4 (E2M1, one E4M3 scale per 16 channels).
• The indexer is DeepSeek-V3.2's lightning indexer: 32 FP4 query heads of 128 dims against one shared key per entry, ReLU'd scores summed with per-head weights from a 5120 → 32 projection, top-512. Reindex layers in the decoder only score a 16,384-entry candidate pool that layer 20 chose by blocks of 8, so their cost stops growing with context length.
• The MoE block is DeepSeekMoE with 384 routed experts (6 active, FP4) and one FP8 shared expert, √softplus scoring, and two correction-bias vectors — text tokens and image tokens are load-balanced separately. SwiGLU is clamped at 10.
• The mHC pre-mix / post-mix pills replace the residual add. Each sub-layer collapses the four residual copies into one input with a 4-vector of coefficients, and writes back with a 4-vector plus a 4×4 doubly-stochastic (Sinkhorn) mix of the copies; the coefficients are a 20480 → 24 projection of the flattened stream. Single-Pass mHC's twist is that the collapse uses the coefficients the previous sub-layer computed, so one fused kernel can do the update, the mix and the coefficient prediction in a single read.

Computing the parameter count

The tally in the margin sums the grey parentheticals; hover a row to see which boxes it covers. It has three positions: total, and the parameters touched per token in decode (all 40 layers plus the head) and in prefill (the 20 encoder layers plus the decoder's single KV projection — the decoder body never runs on prompt tokens). The routed experts are 543.6B of the 552B; the attention path of a whole layer is 127M, and a Full layer's CSA2 machinery adds 8–11M on top of that.²

The KV cache

The headline of the tech report is 890 bytes of global KV per token, and it falls straight out of the diagram. Four layers hold a main KV cache of 512 FP4 values with one E4M3 scale per 16 channels (288 bytes per entry) and an indexer key of 128 FP4 values with one E8M0 scale per 32 (68 bytes). The three encoder caches hold one entry per two tokens and the decoder cache one per token, so per token there are 2.5 entries: 2.5 × (288 + 68) = 890. The sliding-window caches are per layer but bounded (128 entries of 528 FP8 bytes each), and are never persisted.