Multi-head Attention
In the Transformer, the Attention module repeats its computations multiple times in parallel. Each of these is called an Attention Head.
Attention Hyperparameters
There are three hyperparameters that determine the data dimensions:
- Embedding Size — width of the embedding vector (we use a width of 6 in our example). This dimension is carried forward throughout the Transformer model and hence is sometimes referred to by other names like ‘model size’ etc.
- Query Size (equal to Key and Value size)— the size of the weights used by three Linear layers to produce the Query, Key, and Value matrices respectively (we use a Query size of 3 in our example)
- Number of Attention heads (we use 2 heads in our example)
Input Layers
这个图是在input layer，看到5个sample下，seq是4个，emb为6.
这个就是塞给第一个encoder的Q、K、V。
The Input Embedding and Position Encoding layers produce a matrix of shape (Number of Samples, Sequence Length, Embedding Size) which is fed to the Query, Key, and Value of the first Encoder in the stack.
Linear Layers
However, the important thing to understand is that this is a logical split only. The Query, Key, and Value are not physically split into separate matrices, one for each Attention head. A single data matrix is used for the Query, Key, and Value, respectively, with logically separate sections of the matrix for each Attention head. Similarly, there are not separate Linear layers, one for each Attention head. All the Attention heads share the same Linear layer but simply operate on their ‘own’ logical section of the data matrix.
Linear layer weights are logically partitioned per head
This logical split is done by partitioning the input data as well as the Linear layer weights uniformly across the Attention heads. We can achieve this by choosing the Query Size as below: Query Size = Embedding Size / Number of heads
In our example, that is why the Query Size = 6/2 = 3. Even though the layer weight (and input data) is a single matrix we can think of it as ‘stacking together’ the separate layer weights for each head.
注意这个图的过程，其实开头和结果很简单，就是要把原来的Q给分组（按照head的大小）劈开，但是这个具体到矩阵运算（或者说是pytorch的库），就是先reshape大小从(4,6)→(4,2,3),然后为了能方便索引的取出一个head，把前两维进行swap（我认为就是转置transpose）→(2,4,3)
Compute the Attention Score for each head
Merge each Head’s Attention Scores together
(4,3) (4,3)→(2,4,3)→(4,2,3)→(4,6)
Multi-head split captures richer interpretations
An Embedding vector captures the meaning of a word. In the case of Multi-head Attention, as we have seen, the Embedding vectors for the input (and target) sequence gets logically split across multiple heads. What is the significance of this?
This means that separate sections of the Embedding can learn different aspects of the meanings of each word, as it relates to other words in the sequence. This allows the Transformer to capture richer interpretations of the sequence.
Decoder Encoder-Decoder Attention and Masking
The Encoder-Decoder Attention takes its input from two sources. Therefore, unlike the Encoder Self-Attention, which computes the interaction between each input word with other input words, and Decoder Self-Attention which computes the interaction between each target word with other target words, the Encoder-Decoder Attention computes the interaction between each target word with each input word.
Therefore each cell in the resulting Attention Score corresponds to the interaction between one Q (ie. target sequence word) with all other K (ie. input sequence) words and all V (ie. input sequence) words.