DeepSeek-V2

DeepSeek-V2 improves upon the DeepSeekMoE architecture by introducing MLA, an attention mechanism equipped with low-rank key-value join compression. Empirically, MLA achieves superior performance compared with MHA, and meanwhile significantly reduces the KV cache during inference, thus boosting the inference efficiency.

Comparison of KV Cache

Attention Mechanism	KV Cache per Token (# Element)	Capability
MHA	$2n_h{d}_{h}l$	Strong
GQA	$2n_g{d}_{h}l$	Moderate
MQA	$2d_{h}l$	Weak
MLA	$(d_c+d_h^R)l\approx \frac{9}{2}{d}_{h}l$	Stronger

where $n_h$ denotes the number of attention heads, $d_h$ denotes the dimension per attention head, $l$ denotes the number of layers, $n_g$ denotes the number of groups in GQA, and $d_c$ and $d_h^R$ denote the KV compression dimension and the per-head dimension of the decoupled queries and key in MLA, respectively.

For DeepSeek-V2, $d_c$ is set to $4d_h$ and $d_h^R$ is set to $\frac{d_h}{2}$. Therefore, MLA requires an amount of KV cache equal to GQA with only 2.25 groups, while achieving stronger performance than MHA.

References

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model