Hi and welcome back to this birth course, so we just have seen the general idea behind the transformer,
which is the base for the birth model.
But I think there is one last thing that we may want to explore before going into the birth explanation,
which is the attention mechanisms.
So.
I think it's really interesting to learn more about how it works, how it is implemented, the mathematics
behind it and all this kind of stuff.
Of course, it's not necessary to understand and to apply, but as we will not code the details of birds
who will only use it.
So if really you don't want to know about that, you can skip this lecture and I will see you in the
next one.
But if you want to know more about the attention mechanism and you haven't done the modern analytics
next course, then you are the right place.
And we are just ready to go into the ascension mechanism in more detail.
So what is the general idea behind that?
So we have two sequences or two sentences that can be equal in the case of the self attention, B will
be the contexts and A, will be the sequence that we want to really work with.
This is the sequence that conveys the information that we want to rearrange in a certain way.
So they can be different in certain cases, like, for instance, if we are working with the translator,
we could have one sentence that is in English and the other one in French.
And we want to apply the attention to those two sentences together, or they can be the same, for instance,
in the encoding.
But we have only the input sentence and we just want to apply this this attention mechanism to a single
sentence.
And in this case, A and B will be equal.
So what we do with those two sentences is that we first wants to see how each element from A is related
to elements from B essay elements and not words.
Because at the very beginning, of course, we have words because the input is a sentence.
But later, when we rearrange all matrices and we combine words and we do some mathematical operations,
we still have sequences of the same length.
But this time it's not really correct to say that they are words, they are elements, mathematical
elements that convey information in a more efficient way.
So that's why I say elements and not words.
It's still sequences and at the very beginning, its sentences.
So it would.
So we just want to see how each element from A is related to the elements from B, and then we want
to we combined all the elements from A according to those relations.
So we want to rearrange the information in a according to how each element of it is related to the elements
from B..
So if we do before after this attention mechanism, we have a beginning, a sequence A and A context
B that which tells us how we manipulate N and outputs.
We have a new sequence where each element I will be a mix of elements from a that are related to the
elements.
B I.
So generally speaking, B tells us how we will combine elements from A or in the case of the self attention,
A tells us how we recombine a so we can see that there are two important parts to two steps.
The first one being to compute the relations between elements from A and B, and the second one is to
use that in order to recombine.
So let's first talk about the first step, getting the relations between the elements we know that are
each words and later each element will be a vector because we will apply and embedding layer at the
very beginning of the model.
So each word is represented by a vector and a good way in math to compute the similarity between two
vectors is to use that product so that that product is a mathematical operations that take as inputs
two vectors and gives us a number.
We will say that it's between minus one and plus one actually is between minus the product of the norms
and plus the product of the knobs.
But in order to simplify this, we will say that the norms of our vectors are one.
So the outputs of the products will be between minus one and plus one.
And the idea is that two vectors will be similar.
If the product equals to one, they will have no correlation.
If the product equals to zero and there will be collated in an opposite manner if the other product
equals minus one.
And actually the way of computing those similarities focuses on the directions of the vectors.
I have a small animation for you to see how it works so we can see that the two vectors, you and V
and the direction of you is changing.
As time as time passes and you can see here how that product changes with the direction of you changing,
so when you, for instance, here is starting to be perpendicular with the this is what we call non
colation you and we are not correlated.
And you can see that the product was equal to one.
Two zero, sorry, now you go to the opposite direction and this is where the digits equals minus one.
This just means that the coalition is.
Negative.
Again, we go to a non correlating and none correlation phase, and now if you and we have a sense of
direction, we say that they are highly correlated.
So there is a huge similarity between them.
And that's why this what they call two plus one.
So that that product is a very useful tool in order to capture the similarity between two words, and
that's actually perfectly fits the Alpay context, because when we do imaging, we actually make sure
that's two words that have similar meanings, very close in all in meaning space, which means that
they have similar direction.
So a quick and very simple example of that.
This could be a representation of the merging of three words that would be joy, despair and tree,
for instance.
And we can see that joy and despair.
They kind of have purpose and meaning, and that's why they are vectors in the opposite direction.
So the DOT products between joy and despair would be minus one.
And if I pick another random words like tree, it is not really related to what is there.
So they are perpendicular with those two vectors and that will be equal to zero.
Well, it's true that we could argue that tree might be closer to joy than despair, but whatever.
And if we had another word like happiness, for instance, it would be very close to joy.
So Direxion will be very close to the direction of the vector Joy.
So the DOT predicts between happiness and joy would be really close to one.
So finally, the DOT is a very powerful tool in order to compute the similarity between two words.
So all we have to do now is to apply it to each pair of words for all two sequences and be.
There is a very simple way to do that, and in order to understand it, we will have a look at how that
for and matrix multiplication work.
So first, the DOT products.
We can see here that we have two vectors.
The first one, A, B, and the second one is why the first vector is horizontal and the other one is
vertical.
This is how we write our vectors when we want to apply that product.
And what it does is that it takes the first element from the first vector and the first element from
the second vector.
We multiply them together and we do the same thing for the second elements.
We multiply then also and we sum the total.
That's basically what that product does.
And what is interesting is that if instead of having a simple vector here we have the whole matrix,
the result of this matrix multiplied by the vector X Y is on the first elements, A, X plus B, Y,
and the second is C, E, plus Zwi.
And we can actually see that it really corresponds to the DOT product between A B and X Y here and the
product between KD and X.
Why he.
So multiplying a matrix by a vector is just to do multiple products at a time.
So each row of the matrix will be that predicted with the second vector.
I'm not sure that we could say that predicted, but we will for the sake of discourse and if we go even
further because we place the second vector by a matrix also.
So we will get our Matrix, ABCDE and WXYZ.
And again, knowing that the first element is A W plus B, Y and so on, we just cannot use.
That's actually each element is the product.
And so multiplying two matrices together is actually the same thing as computing all the products with
each row of the first matrix, with each column of the second matrix.
So if we want to compute many products at a time, we just have to multiply the first matrix, which
is made of all the first vectors of the they are vectors we want to apply the DOT product to and we
just make a second matrix made with the second vectors of the.
There are vectors that we want to apply to DOT product to.
The important thing to keep in mind is that we want for the first matrix to have all vectors horizontally
and for the second matrix.
We want our vectors to be vertical, but we will come to that Viter.
So we know that we want to apply those products to each pair of words or each pair of elements from
our sequences in order to compute the similarity.
And we know that we can do that with the whole sequences at a time just by using matrix multiplication
so we can go back to the formula in the paper and this figure that explains the whole process of the
scale DOT product and have a closer look at it so we can see here that we have three sequences as inputs.
That's because it's a very general way to consider the scale products.
But in our case, we have only two sequences.
As I said before, sequence A, sequence B, and that's actually because K and V, so the Ts and the
values will always be the same in our in our model in the transformer and Q could be difference.
So.
Q will be what I call the context is the sequence B and Q And we are the sequences that convey the information
that we want to rearrange, that we want to work with.
So and we are the sentence AI that I talked about earlier.
So the first step that we can see here is that we multiply.
Q Which is all context sentence with the sequence.
K And after the embedding, each sequence being a matrix, we get all the products that we wanted and
then we talked earlier and this t here just means transpose and it means that we simply flip the matrix
in order to transform any horizontal vector into a vertical vector and of course any vertical vector
into horizontal vector.
So rows become columns and cones become rows.
And with that we make sure that the matrix multiplication actually gives us all the DOT products between
all the pairs of elements from the two sequences.
Q and K all be an eight.
So this.
Q Times key is a new matrix that conveys all the information about how the Q is related to K, or more
precisely how each element or each words of Q is related to each elements of K.
Now the next step we can hear is this small scale.
It doesn't really have intuitively a meaning, it's just something we do to improve the model and it
stabilizes the whole process.
But there's nothing really to understand behind.
So we just divide by DKA, which is the dimension of.
Each element of each words from the keys, then we can see that we apply a softmax so very quickly,
Softmax is an operation that takes as inputs and outputs.
We get a vector of the same dimension, but each element will be between zero and one.
That's the first thing that is important.
The second thing is that we keep the relations between each element of the initial vector.
So if Elements two was lower, that element three eight will be the same after the softmax.
So we just keep half each element is greater or lower than the others.
That's the second point about softmax.
And the third one is that the sum of all elements from the output of the such max should be equal to
one.
And the reason for that is that it perfectly switz what we call weights in mathematics.
So we have just a sequence of numbers between zero and one, and there some equals one.
And having those weights is really useful because it allows us to recompose elements from vectors or
to apply products or to some elements from matrices of vectors or whatever in a very meaningful way.
Because it just tells us how much of each element we want to take when we make a time or when we recomposed
information and we all show that there is nothing that that gets out of hand just because the sum of
everything equals one.
So there is also sort of a stability in the operations we will apply later.
So this softmax is necessary in order in order to get valid weights.
So now we have a perfect and valid representation of how each element from two was related to each element.
OK, so how each element from our sentence A is related to each element from the context sentence B.
And finally, we will use those weights to recompose elements from our sentence, which here is called
the to get a new rearranged version of the information in a in the values.
Let's have a look at the first visual representation in order to make it even more simple and this will
be, what, two times Katy looks like.
So I said that with this matrix multiplication, we get a new matrix that tells us how each element
from Q is related to each element from K.
And here we can see an example of those mattresses we have here, each words of our sentence.
Q It's a French sentence.
And here we have each word of the sentence, which is an English sentence.
So of course this is the detention mechanism in a translator.
And in this matrix is square represents the value, the scope of the similarity between two words and
the whiter it is, the more important the similarity is between those two words.
And he, of course, as it is a translator and as the French language is very close to the English language
in terms of how a sentence is constructed, we have a straight line here because we know that first
words will most likely be the translation of the first words for the other sentence to.
And that's the same for the words at the end of the sentence and for basically every local parts of
the sentences.
But we can see some interesting behaviors.
For instance, here we can see a brewing, some sort of a square where we have some similarities that
are going on.
And that's normal because here we have in English, it should be noted, and in French we have to on
the note taker, which is the exact translation, but it's not a word by words translation.
It's just that the global meaning of it should be noted that corresponds to the global meaning of equal
violence.
And as it's not the word by words, literal translation, we can see that that there is more than one
square that is activated.
So words are related to many other words in order to get the general meaning of this group of words.
So that's for the representation of what these cute timescape means.
So now let's talk about the second main port and the second step in our attention mechanism, which
was to recombine the the the sequence values, which we called a before and is equal to K. as Venkayya
are the same ones doing all process in the transformer.
It's not as we get her.
Q And and so.
Q And we are related.
Let's recompose the according to that in order to see what happens.
I just showed here a visual representation of how matrix multiplication works and he on the left, this
matrix is the output of this operation.
So it's exactly the same kind of matrix that we had before.
He it's just the memorized similarity between the sequences and this matrix represents the sequence
V ulcerate, the dimension of the matrix.
So here the dimension on the left is the length of the sequence.
Cue the length of the sequence k because we made this operation two times the transpose of K, so this
is dimension of our matrix.
And here this matrix is the sequence V, so this dimension is the length of the sequence V and this
dimension is dimension of our model.
Of course K and V being equal length will be equal to the length, so the matrix multiplication will
be valid.
So let's say we want to know what we put in the second row of our results.
This is done by recomposition elements from our sequence V according to this rule, for the second row
of our results, we will only use the second row of our attention matrix.
And as each softmax has been applied real wise, we actually know that this row will be a valid set
of weights.
So let's say, for instance, that the second word of Cube was really related to the first and the third
words of K, which is also V, then it means that the coefficients a two one and a two three will be
very, very high.
They will actually be close to one and the others will be close to zero.
So when we do each vector multiplication here in this this row, we know that we will use a lot of the
first and the third row of the and we will not use the others because this coefficient two two will
be close to zero.
So we don't use this one, but the coefficient to one is close to one.
So we will use a lot of B one, two and third coefficient.
A two, three will be close to one.
So we will use a lot of B three, two.
So finally, C two, two will be made of a lot of B one, two and be three two.
And that will be the same for each other elements of this row.
So that means that this row, the second row of the outputs is made of a lot of the first row of V and
the third row of V, which makes sense because because we said earlier that our second element from
Q the second word was related to the first with a V and the third word of the.
So if we summarize, we have a sequence cue a sequence, we we will say the end, it's the same as Q
And the second word of Q is related to the first and the third word of the.
So when we to the attention mechanism at the end of it instead of Q we get a new sentence.
And in the second element we have a mix of the first and the third elements are words from the.
So this new sequence of sentence is a mix of elements from the according to how they were related to.
Q So Q is really a guide to how we want to we compose and rearrange information from the.
That was for how the attention mechanism works and now we'll see how it is applied in the transform
it so we can see that it appears three times in the transform architecture, it appears in the encoding
phase and it appears twice in the decoding phase, in the encoding phase.
And the first time in the decoding phase, we can see that the three inputs are the same and this is
what we call self attention.
So in this case, CU would be equal to K and V, so the initial sentence and the context sentence are
the same, which means that we will compose the sentence according to how each word of this sentence
is related to the other words of this very sentence.
And that's actually the main point in replacing Arnon's with a more global way of encoding sentences
instead of encoding a sentence, getting one word at a time.
We compute information globally with the whole sentence at once, and we repeat this process several
times in the paper to say that they do that eight times in order to make sure that we get the most out
of this recomposition of the information.
And in the decoding phase, we start with the same thing where we compose information from the sentence
that we give us input for the decoder.
So we apply of attentional layer.
And then there is this attention layer right here that doesn't use the same sentence for the three inputs.
And in this case.
Q So the context sentence will be the outputs of these?
Sibilia So it will be the sentence coming from the decoder and Shou and V which are equal will be the
outputs of the encoder.
So if we take the example of a translator, let's say a translator, that's transform an English sentence
into a French sentence, here we will have an English sentence that we want to recompose with the context
made of a French sentence.
So before this layer, we have an English sentence or an English sequence because we already applied
some mechanisms and math to the English sentence.
But so we have an English sequence here that we want to do.
We compose according to a context that is made of the French sentence that we have made so far.
So we are in the middle of a translation.
He we have what we have translated so far, like half of a French sentence, for instance.
We work with it a little bit and we get a context.
So all encoded version of the English sentence will see how it is related to the French sentence that
we have made so far element wise.
And so we will get a new version of the English sentence according to the work that we have already
done with the translation.
And that's how we get new information that we can use for the translation in order to produce the next
words in all translated sentences.
So the last elements we need to see in the mechanism is what they call in the paper lookahead musk.
And this is something that appears that he in this very attention subletter.
What it does is that it forbids the decoder to have access to future words.
And what I mean by that is doing the processing of the sequence in the decoder.
When we deal with the words or the elements, I we don't want to have access to the what I and the following
words to understand that let's simulate a training phase.
So during the training we will have a pair of text or a pair of sentences.
We will take the example of a translator.
So we will have an English sentence and the French sentence that is supposed to be the translation of
the English sentence.
So to train with this pair of sentences, we will use the English sentence as inputs for the encoder
and we will use also the French sentence as targets, as outputs for our decoder.
But we will also use this sentence, the first sentence to the output sentence as inputs for our decoder
right here, but as it is written here, shifted.
Right.
So what it means is that if you have a sentence of, let's say, 20 words, what we will use here as
inputs is a sequence of twenty one words, the first one being just a token that we will call the starting
token.
So it just says that the sentence starts.
And so what we want our transformer to do is to have as output here the exact same sentence as he but
shifted by one position to the left.
But of course, in order to predict the words I, we want the transformer to have only access to the
first AI minus one words from the French sentence.
So what we will do is that during the self-protection mechanism right here, when we compose the sentence,
that position AI, for instance, we will only use the first AI elements from the sentence, which are
actually the starting token and the first AI minus one words.
And the best way to do that is to multiply our attention metrics that we had before.
So with the few times transposed by a triangle, metrics that we see here, which is made of.
Once in the bottom left part of the mattress and zeros in the street, the writes parts of the mattress,
and by multiplication I don't mean a standard mattress multiplication.
This is an element Y's multiplication.
So we will multiply the top value of this matrix by one.
Then the next value by zero and so on.
So the top right value of this matrix by zero.
And that's how it works.
So by doing this, we keep all the attention values from these parts of our automation metrics and we
set to zero all those values and to see exactly what does.
Let's have a look back at the visual representation of the matrix multiplication right here.
So this is a few Times Square transpose matrix.
And if we multiply by the lookahead mask that we saw before, it just means that those coefficients
stay the same in the bottom left part.
But those coefficients now are set to zero.
So when we do the recomposition at the end, so when we multiply by V, which is again the same sentence
because we are in a self attention process, if we go to second row, for example, here we have only
the first two coefficients that are not equal to zero and all the others are set to zero.
So that means that when we do the recomposition, we will multiply all those elements by zero and we'll
only keep the first two elements of our initial sequence.
So our second elements in our output sequence will only pay attention to the first two elements of our
initial sequence, which means just the starting token and the first word.
So that's exactly what we wanted.
Element Number I, after the self-protection mechanism in the decoder, have only access to the first
AI minus one words.
And that's how we make sure that during the training, when we try to predict what number I, we only
have access to the previous words of the predicted sentence.
So that allows us to train with a pair of sentences in only one step by simulating the fact that to
predict the next words in our output sentence, we don't have yet the rest of the sentence.
So that was for this lookahead mask.
And finally, now we can see in the paper that there is this other figure that explained how they use
it in the attention Somalia, because they don't just apply the scaled products directly under sentences.
What they do is that's the first completely linear projections and that then do the concatenation.
So instead of applying the skills that predicts attention to the whole sentence, they split each factor
from the sentence.
So each element from the sentence into spaces.
So let's say each words after embedding has a dimension of twelve.
For instance, instead of applying the scale of the product to sequences made of elements of Dimension
12, we will split it into, let's say, four sequences, each element being of Dimension three.
And then we compute the scale that attention with that and we can get the results in order to get back
to our initial dimension for the model.
And the reason for that is well explained in the paper.
They say Neotel attention allows the model to jointly attempt to information from different representations
of spaces at different positions with a single addition head averaging inhibits this.
So what it means is let's get back to our example with Embling dimension of twelve.
We could say that among the, for instance, three first dimensions of our embedding, there is something
that we want to get between the first and the third words.
But it only appears in those first three dimensions then we would really like to have our attention
mechanism to be able to focus on those three dimensions so that the information is not faded into the
twelve initial dimension of the embedding.
So splitting the space into spaces allows the attention mechanism to attend to more information and
to be able to get more relations between the elements of a sequence.
And finally, how we do that, mathematically, it's important to notice that we don't split the space
into safe spaces and then apply many different linear transformations.
What we do is that we first apply a huge linear transformation.
So a dense layer, intense flow, and then we split the space into subspace.
Is the reason why we do that is that it allows the architecture, the model to learn more because of
Transformer could say that it's better for him to make, let's say, the first and the 8th and the 10th
dimensions work together.
So during the first big linear transformation, he will just rearrange all the dimensions of our embedding
to put together the ones that work the best together, actually.
And then we will split in two spaces, because if we split first, then it means that the first, second
and third dimensions will always work together and have access to the other dimensions for the scale
products.
And maybe that's not optimal.
So that's why we first apply the big linear transformation and then we do the splitting.
And of course, after updating the square, the attentions to all of those spaces, we can cut everything
back in order to get back to the initial dimension of our embedding.
So that was a huge part of understanding this mechanism, how it works, this composition of the elements
of different sequences.
Don't hesitate to go back to it to have a look at the figures.
But that was it for the hardest part, which was actually also the more interesting one, because in
my opinion, it's really interesting how this says a lot about how language works for humans.
My interpretation is that the transformer is getting closer to how we as humans actually process our
language.
I feel like that we process sentences globally and not one after the other, and that's what comes after
the other in a sentence.
When we talk, when we read, it's just the constraints of how our language is made.
But the way our brain gets the information and processing sentence is closer to how the transformer
deals with it, with the attention mechanism.
I feel like we do it more globally and then the language working would afterwards is just maybe a non
optimal way to convey information of the meaning of what we want to say.
Anyway, that was maybe a bit esoteric, but it's just that I find it really interesting how the sequence
to sequence models evolved to get to this one.
That seems closer, in my opinion, to how we really process language as humans.
So of course, we are not done with the explanation of the transformer because there are some other
details in the implementation that we have to deal with.
For the most part, this is done.
So we only have a few other details to see before starting to implement a transformer, hopefully excited
and using.