Scaled Dot-Product Attention (SDPA)

Scaled dot-product attention is an attention mechanism where the dot products are scaled down by $\sqrt{d_k}$. Formally we have a query $Q$, a key $K$ and a value $V$ and calculate the attention as:

$$\text{Attention}(Q,K,V)=\text{Softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

If we assume that $q$ and $k$ are $d_k$-dimensional vectors whose components are independent random variables with mean $0$ and variance $1$, then their dot product, $q\cdot k={\sum }_{i=1}^{d_k}{u}_i{v}_i$ has mean $0$ and variance $d_k$. Since we would prefer these values to have variance $1$, we divide by $\sqrt{d_k}$. We then apply softmax to the attention scores so that they are transformed into a set of positive, normalised weights that sum to 1.

References

• Papers With Code
• Attention Is All You Need