Hi, guys, and welcome back to this email because so right before talking about the transformer, we'll
see how we used to address the sequence to sequence tasks before.
So all the tasks where we need to predict the whole text or a whole sentence from a text or a sentence
that we are given, for instance, this could be text summarization or translation chatbot, all this
kind of stuff.
And we'll see how this led to the transformer.
So what we have mostly been doing to address those task is to use a pair of what we call encoder and
decoder.
So this encoder here is what is supposed to summarize all the information that we want to use from the
input sentence.
And the decoder will use as inputs the output of the encoder in order to create the right output.
And an actual way to do that was to use the Arnon's.
So recurrent neural networks and the way those Arnon's work is the following.
So we have a sale that takes as inputs the output of the produce sale and the next word from the input
sentence and it computes the next what we call hidden states.
The important point is that all of those sales are identical and they do exactly the same job.
It's just that we feed them with a new words from the input sentence each time.
And so it learns more and more about the whole input sequence.
So this is why we call that recurrent neural networks, because there is this cell that is repeated
as many times as needed, which is actually the length of the input sentence.
And we actually do exactly the same for the decoder.
We have a new cell that could be repeated until we reach the end of the decoding phase.
And this cell is different from the encoding one.
But of course it's the same through the whole decoding phase.
And we will have as inputs the hidden state from the previous cell.
The previous predicted would and we have as outputs the next word that we predict and a new hidden state.
So a hidden state is just a huge vector or huge matrix that conveys information.
And so we add information to those vector.
Each time we, for instance, add a new word from the input sentence or each time we predict a new word
for the output sentence and those hidden states, those are the information that we are carrying along
through the whole process.
So just to go through all the steps for those are announced, the first thing we do is to embed the
words into vectors so they can be used more efficiently by our computers, by our algorithms.
And so each word now is represented by a vector.
Then we applied what I just explained about the RNA.
And so we compute a new hidden states for our encoder with the previous hidden states and the new embedded
words that comes from the input sequence after this whole process has been done with the encoder.
We have a hidden state that is the output of the encoder and that is supposed to convey all the information
that we wanted to have from the input sequence.
And now we can apply the decoder, of course, in the decoder.
Each cell for the earnings are identical, but they are different from the cells, from the encoder.
And this is kind of like unrolling the information that we get from the encoder.
It's like in the encoder we made a package adding a new words.
So new information needs at each step and at the end we just retrieve a new information at each step,
a new word that will be a part of the output sequence.
So I won't go into the details of our networks, how they are computed, how we implement them, because
you will see later that we want to need that for the transformer.
But it was important to to see that because this is the origins of how we do sequence to sequence models
and thinking of evolutions of models.
We must be aware of the weaknesses of this model.
And one of the most important one is that you can see that we add words one after the other in the encoder.
And so we will certainly lose information from the beginning of the input sentence for each word we
add.
So for very long sequences, very long sentences, the information will not be complete at the end of
the encoder.
And so when we want to predict the next word in the decoder, we could mis information that has been
lost through the encoding phase.
So what has been added to the islands in order to address this issue is what we call the attention mechanism.
So basically what it does is that during the decoding phase, we add a new input to our cells in the
Arnon and we call that the context vector, and that's actually vector that is supposed to convey a
global information about the whole input sequence and more precisely how each step of the input sequence
is related to our current status in the decoding phase.
So let's suppose that we are at this phase right here.
So we just predicted the word XI.
We have a hidden state that is G1 and what we would have done before, which just computes a new two
hidden states using as inputs G1 and the previous predicted Watashi and get the new word is.
But here we add a new input, which is this vector C, that is, as you can see, he actually created
some of all the hidden states from the encoder.
And so the important part is how do we send those hidden states?
And we can see how it's represented by the coefficients, a zero and one and and so on.
So the intuition behind that is just to assess how Jiwon So the current hidden state, the current state
of the decoder is related to each of the.
Hidden states, so how our current decoding phase is related to each of the encoding phase.
So if we want to get an interruption of that, we could say that he, the Hidden State's Jiwon.
Is the one that we get just after predicting the word sheep, so this state would probably convey an
information like we just declared the subject of the sentence, which is she.
So now we want to we want to know how these hidden states.
So just having declared the subject of the sentence is related to each of those hidden states and naturally,
having declared the subject would be related to the verb that is related to the subject in the input
sentence, which is a question in our example, it's the same process as when we speak.
Just after saying she we know that our current state of speaking is related to the verb that the other
person just used to talk about.
So if the person in front of you said, how is she just after saying she, you know, you have to react
to the verb is.
So that's globally how the attention mechanism works here in the Arnon's.
And if you want to go into mathematical details, we would just say that S.I that's the context factor
that we want to feed the cell number with.
So this being G I mean this one and he g so this is just a way to sum of all the hidden states from
the encoder.
So basically each number I thought I'd just say how much we want to emphasize on the hidden states ahj
when we compute guy so he g so the important part is to create those alpha IJA but basically say all
current state that is actually J one.
So J ie minus one is related to each of those hidden states.
And the way to do that is that we will apply a softmax function, so that's just a tool that we use
in order to create weights for a weight.
So the coefficients e being the similarity, for instance, between the state of our decoder and the
hidden state of our encoder being those coefficients, we just compute the Alpha in order to keep the
relations between the E.
So if one of these coefficient is higher, of course the corresponding alpha would be higher too.
But thanks to this operation, the softmax, the sum will be equal to one and each number will be between
zero and one.
That actually makes all the Alphas valid ways that we can use to sum all the hidden states h and the
most important parts this e IJA that conveys information about the relation between J i.e. minus one
and each of the J will just be a function that we can pick.
Actually there are different options for that's one of the most common one is to use the products because
the products predicts that later between two vectors is a good way to get how similar those vectors
are.
But the important thing to keep in mind is that A is supposed to be a similarity function.
So if you just repeat this backwards, we are at the stage AI for all decoder.
So we are about to predict Jai and have already predicted the words ie minus one and the hidden states
jieyi minus one.
And so in order to predict this DJI, we will just wants to know how the previous hidden states, which
is where we are right now, AI minus one, is related to each of the hidden state of the encoder.
So it's almost like if you are looking for how these words is related to each of the words from the
input sentence.
And the way to do that is to use a similarity function between our current hidden state and all of the
hidden states of the encoder.
And now that have those similarity weights, we just apply a softmax in order to have real weights that
we can use for a weighted sum.
And we apply those weights to all of the hidden states of the encoder to finally have a vector that
is mostly made of the hidden states from the encoder that are related to our current state in the G
coding phase.
So that's basically how the attention mechanism works in the R.
A. It's just doing the decoding phase.
We simply add a new vector that conveys more global information about the whole input sequence.
And this vector changes at each step of the decoding phase so that each cell of the R and then gets
information from the input sequence that is related to the current state.
So that's it for the attention and marginality for how we use to address sequence to sequence task before
now we have all the bases to talk about the transformer and that's exactly what we will do next.
So it's useful.