Attention is all you need

references: https://arxiv.org/abs/1706.03762

Attention blocks

We call our particular attention “Scaled Dot-Product Attention” (Figure 2). The input consists of queries and keys of dimension $d_k$, and values of dimension $d_v$ . We compute the dot products of the query with all keys, divide each by $\sqrt{d_k}$, and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as: $Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}})V$