Skip to content

Fix text duplication in self-attention explanation #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Josh-Archer wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix text duplication in self-attention explanation #8

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Transformer_walkthrough.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@
"\n",
"A metaphor for understanding self-attention is a hash table. In a hash table, there is a list of keys, each of which is associated with a value. A query is sent to the hash table, and the hash table has to find the matching key and return the associated value. Except imagine that this hash table is a fuzzy hash table in the sense that it the query doesn't have to match any keys and the hash table will return whatever seems closest to the query.\n",
"\n",
"The input to self-attention is a `batch_size x sequence_length x embedding_size` matrix. Ignoring the batching dimension, what we have is a sequence of embedded tokens. Self-attention copies this input, `x`, three times and calls them the \"query\" (`q`), \"keys\" (`k`), and \"values\" (`v`). Each of those matrices go through a linear layer. This linear layer is where the network learns to make scores. It makes each matrix different, and if it comes up with the right, different, matrices, it will get good attention scores. If it gets good attention scores and if it gets good attention scores, the loss will be reduced.\n",
"The input to self-attention is a `batch_size x sequence_length x embedding_size` matrix. Ignoring the batching dimension, what we have is a sequence of embedded tokens. Self-attention copies this input, `x`, three times and calls them the \"query\" (`q`), \"keys\" (`k`), and \"values\" (`v`). Each of those matrices go through a linear layer. This linear layer is where the network learns to make scores. It makes each matrix different, and if it comes up with the right, different, matrices, it will get good attention scores. If it gets good attention scores, the loss will be reduced.\n",
"\n",
"Attention-scores are generated as follows. First, we split the `q` and `k` matrices into multiple parts (called \"heads\"). This is called multi-headed attention. The reason we do this is so that each head/part can independently produce different attention scores. This allows each token to have several \"best\" other tokens. In implementation, we just designate chunks of each token embedding to different heads.\n",
"\n",
Expand Down