diff --git a/Transformer_walkthrough.ipynb b/Transformer_walkthrough.ipynb index bee7ba2..ce5f805 100644 --- a/Transformer_walkthrough.ipynb +++ b/Transformer_walkthrough.ipynb @@ -464,7 +464,7 @@ "\n", "A metaphor for understanding self-attention is a hash table. In a hash table, there is a list of keys, each of which is associated with a value. A query is sent to the hash table, and the hash table has to find the matching key and return the associated value. Except imagine that this hash table is a fuzzy hash table in the sense that it the query doesn't have to match any keys and the hash table will return whatever seems closest to the query.\n", "\n", - "The input to self-attention is a `batch_size x sequence_length x embedding_size` matrix. Ignoring the batching dimension, what we have is a sequence of embedded tokens. Self-attention copies this input, `x`, three times and calls them the \"query\" (`q`), \"keys\" (`k`), and \"values\" (`v`). Each of those matrices go through a linear layer. This linear layer is where the network learns to make scores. It makes each matrix different, and if it comes up with the right, different, matrices, it will get good attention scores. If it gets good attention scores and if it gets good attention scores, the loss will be reduced.\n", + "The input to self-attention is a `batch_size x sequence_length x embedding_size` matrix. Ignoring the batching dimension, what we have is a sequence of embedded tokens. Self-attention copies this input, `x`, three times and calls them the \"query\" (`q`), \"keys\" (`k`), and \"values\" (`v`). Each of those matrices go through a linear layer. This linear layer is where the network learns to make scores. It makes each matrix different, and if it comes up with the right, different, matrices, it will get good attention scores. If it gets good attention scores, the loss will be reduced.\n", "\n", "Attention-scores are generated as follows. First, we split the `q` and `k` matrices into multiple parts (called \"heads\"). This is called multi-headed attention. The reason we do this is so that each head/part can independently produce different attention scores. This allows each token to have several \"best\" other tokens. In implementation, we just designate chunks of each token embedding to different heads.\n", "\n",