(Solved): Solve all parts of this question 2 Transformers-Self Attention [20 Points] In this question we wi ...

Solve all parts of this question

2 Transformers-Self Attention [20 Points] In this question we will compute the transformer self-attention covered in the lecture. You will be required to compute the value of different matrices as mentioned below and in the lecture notes. For all questions in this section, unless otherwise stated work must be shown in the form of matrix multiplications to receive full credit (i.e. $$C=A{B}^T$$ ). For performing the computations, using Numpy, Excel 2 Table 1: Word Embeddings or other software is recommended to avoid computation errors. When writing your answers please round to 2 decimal places. You may use scientific notation to represent your answers if necessary. $$W^q=\left[\begin{array}{cc}1& 1\\ ?2& 1\end{array}\right],W^k=\left[\begin{array}{cc}?1& 1\\ ?2& ?1\end{array}\right],W^v=\left[\begin{array}{cc}1& 0\\ 1& ?2\end{array}\right]$$ 2.1 In this question we will consider a single attention head. Given the set of word embeddings (Table 1), projection matrices ( $$W^q,W^k,W^v$$ ), and a normalization factor of 3 instead of $$\sqrt{d^k}$$, fill out this table with the normalized query-key score for each possible pair of words. Hint: Compute the value of $$\frac{Q{K}^T}{\sqrt{d^k}}$$ 2.2 Given the word embeddings, the previously calculated query-key values, and the value projection matrix, calculate the output embeddings of this attention head. Output embedding is computed using the Attention formula discussed in the lecture slides. Fill in the table with your results.