DeepSeek-V3: from roofline to reality

A series of worked performance analyses of DeepSeek-V3

If you want to learn how to efficiently train an LLM on many GPUs, you may have already heard of resources like How to Scale Your Model and The Ultra-Scale Playbook, which teach you all of the foundational concepts necessary to achieve good MFU. However, it is one thing to learn out of a textbook and another to actually do the analysis in a production setting on a model you are planning to train in a month.

The purpose of this post series is to really dig into what exactly these analyses would look like for DeepSeek-V3, a historically important and influential mixture-of-experts transformer model. In my opinion, DeepSeek-V3 has stood the test of time as an unusually clean exemplar of the "platonic MoE model." It has very few anachronisms--I could count them on one hand--the three initial dense layers, use of global attention everywhere, the built-in MTP, perhaps where exactly the norms are placed. I don't think you will be poorly served from closely studying DeepSeek-V3. It also helps that MLPerf 6.0 added DeepSeek-V3 as a large-scale pretraining benchmark.

My goal for this series is that we start with a roofline model of DeepSeek-V3: an overly simplified approximation of the behavior of DeepSeek-V3 which tells us what the absolute best, "speed of light" performance we could hope to achieve on it is. And then, we progressively refine this model with more real world correction factors, until we end up with a model that is reasonably able to predict what a PyTorch profile trace would look like if you had actually run it. We should be able to understand why NVIDIA chose the performance settings they did in their MLPerf submission. We should be able to understand the tradeoff space if we're planning to try to pretrain DeepSeek-V3.

AI disclosure: The graphics and visualizations are 100% vibe coded with Fable, but I have been reviewing the widgets themselves carefully. The text can be assumed to be human-written, except when otherwise disclosed.

Posts

• A DeepSeek-V3 transformer block — An infra-oriented description of the DSv3 transformer block.

The performance model

All of the posts are built off of a shared model of DeepSeek-V3's memory and performance behavior. Below, I've placed all of the widgets for interacting with this model, in case you want to work with the model directly. There are still mistakes in the model. Part of the reason I'm writing the posts incrementally is that I'm using these posts to flush out bugs and omissions from various parts of the model. Remember, the model here isn't the important thing: the important thing is teaching you how the model works, so you can build your own for your own model.