Hi and welcome back to this birth, cause we are about to go into the last step of our intuition part,
which is to understand how birth has been betrayed.
So what are the two types of training that I mentioned earlier and how they work and how they allow
birth to perform so well into all those NLB tasks?
So so we have two phases for the training, each face corresponding to a type of output that we saw
before.
The first phase is masked language model.
So the ID with that is, as you can see here, to get all the vectors for each input 21CN and the second
phase for the training is next sentence prediction.
This time this allows us to get this single vector that is an efficient representation of the whole
sequence that we have as inputs.
So we'll go into the details of both those faces.
We start with the Muskan language model, which is actually the part of the training that allows birth
to be bidirectional compared to open IGP that we mentioned at the beginning, which used Transformer
two but wasn't able to use bidirectional training.
So how to achieve that?
If we remember how we would train a transformer, for instance, the idea would be to have our input
sentence shifted by one to the right.
But that's a detail.
So and the idea will be to we compose the sentence, only having a look at the left contexts.
So the idea of training this kind of architecture originally was to get the same sequence right here
as inputs, as the sequence we have, as outputs just shifted by one and only allowing the information
to go from left to right.
That means that we want to get this was exactly the same as this would the E three only by having a
look at the wood or the defector's E one and E two.
That's the standard way to do that.
And of course, if we allow information to go in both directions, that doesn't work anymore because
the computation of this output vector will be able to use all the informations before.
And so all the input words and more specifically, it will be able to use this input words if we which
is exactly what we are looking for.
So that doesn't make sense.
But we want to have all those connections because it's way more powerful to compute vector representations,
because we have access to all the contexts.
So we have to change the way we train our model.
We don't want anymore to get exactly the same input sequence as we have here, which was a condensed
way to predict the next would is just that we were predicting all the next words at once.
So each time we were predicting a word, we were only using the previous words, so that was predicting
the next would, but for the whole sequence at once and now we want to do that, what we will do is
that we will mark the words we want to predict.
So we won't predict the whole sentence.
We will just pick some words in the initial sentence, apply a mask to it.
So we switch the talking.
We just use a homemade token, which is a mask.
And this will be the words that we want to to predict.
That means that now we can allow the information to go in all directions and to use the whole context,
the left and the right, without letting this part, these outputs note what was the original word or
have any information about that apart from the context.
So that's a really powerful way to create a language modeling system because we use the whole power
of the context.
So here are more details and a cool visual representation of what happens.
So if we go into the details, what they explain in the article is that they want to mask approximately
15 percent of the words.
So when you have an input sequence, you randomly pick 15 percent of those words and you want to do
mask language model training with it.
And actually, what they decided to do is that they will not always put the mask techniques.
And the reason is that it would be too far from the fine tuning phase.
So the real words use of boots, because the mask doesn't exist in the real world in real sentences.
So we decided to not always replace the word with the mask.
Dukan, they did that only 80 percent of the time.
They also sometimes would just replace the original talking with a random one approximately 10 percent
of the time and the other 10 percent of the time they left it unchanged.
So so that means that even with a real life situation, we are able to get a representation of the words
using its context.
So here is what happens.
We have our inputs.
So that's the tokenized sentence that we want to train with.
And here is the mask, the version of our sentence.
So we just picked one word here.
It's improvisation and we muskets, then we apply.
But so the stock of encoders and we get vector representations for each of our inputs.
So if we take the fifth one, which corresponds to the masked token, we get an output vector.
So for instance, with the third base, the dimension of this vector was seven hundred sixty eight.
And with this vector, we just apply a fit forward's neural network.
So that's just basically a dense layer.
And the number of output units of this dense layer is actually the total vocabulary size in English.
So that just means that from this vector representation of the words, we get a probability of being
one of all the English words.
We take the highest probability, of course, that gives us the predicted words.
And we want this word to be exactly the talking before the masking process.
So that's the idea.
We just musker words so we don't have access to this word anymore, but we want to guess it's vector
representation only using the context.
So we get a vector that is supposed to represent distortion and from this vector with a simple, dense
layer, we actually get a list of all the probabilities for all the English words to be the math token.
So that's it for the first phase, the first type of training that we do with words.
And it's really useful and really, really powerful because as we saw, it lands the relations between
words.
And it also gives a huge importance to the context, which is what we wanted to get.
And that's also how well I think we as humans use the language, if we will, booch.
We learn words that we don't know using the context, for instance.
So that's quite similar to how we as humans process the language, using the context, using the relations
between words, using the whole sentence at once, and not confusing one word after the other.
And I think that's why Burts is so powerful.
It because in my opinion, it gets closer to how we as humans understand and use the language.
Also, a detail that is actually quite important is that with this technique that is able to make a
difference between homonyms.
So if you take the example of the World Bank, if you go to the bank counter, you are not talking about
the same thing as the river bank.
And with previous word embedding tools like word to wait, for example, we could not make the.
Between two homonyms, because we had a single representation for each word here, that's not the case
anymore, because in order to have a better representation for awards, we take into account the whole
context and how the word is used, the meaning of the words with respect to all the other words of the
sentence.
So that's really important.
And that's another powerful detail of that.
Now comes the second free training phase or the second training technique, which is next sentence prediction.
The ID and the reason why we do that, or at least Google does that, is that we want to get a higher
level understanding of the language.
And instead of just trying to get the meaning of words, we actually want to get the whole meaning of
sentences and trying to get the relations between different sentences and the combined meaning of those
sentences.
This will also allow us to get access to many more tests and not just a single sentence classification
tasks like, for example, what I mentioned earlier question answering this problem where we have a
question and we want to get the answer from a paragraph.
We need to get relations between two different and separated sequences, but also for pair of sentences,
classification task, like trying to guess if two sentences convey the same meaning, even though they
are different, for instance.
So it will be really useful.
So the idea is, as we saw before in the input part of this lecture, to have the two sentences or the
two sequences as inputs separated by this homemade Metuchen set.
And this also appears in the embedded inputs that we give to the encoder layers.
If you remember, we added two different vectors, a vector E A, which is a vector corresponding to
the first sentence and a vector Ebbie, which is the vector corresponding to the second sentence.
So that's how we make the difference between the two sentences or the two sequences into the whole process
and the whole computation.
And the idea of this next sentence prediction is that we give two sequences to birds and we ask birds
to predict if those sentences are related to each other or more precisely, if the second sequence is
a natural continuation of the first one.
So quick example.
If the first sentence is Ecology's one of the biggest challenge of the century and the second one would
be reducing our energy consumption could help seven living beings, then it's pretty obvious that the
second one could be a natural continuation in a corpus or a book or an article two, the first one.
So that is widely expected.
And so for this training phase would be is next.
And if we take another example, we keep the same first sentence.
Ecology is one of the biggest change of the century.
And the second one is camels have three Ilija to help protect themselves from sand blowing in the true
story.
By the way, this is obvious that they don't really come together.
They having the second sentence right after the first one in a regular corpus would make no sense.
So the expecting answer here is not next.
Again, the visual representation of what it looks like.
As we said before, the CLIA stuck and that we added at the beginning is exactly made for that.
So we expect the corresponding vector representation to be used for classification task and of course,
to be used for this next sentence prediction training phase.
So with the same principle as the masked language model, pre training phase, we use this very vector
that corresponds to the sea Duquan, and we apply a dense layer to that with only two output units,
which corresponds to the two possible answers is next are not next's.
And we just do standard classification training with that.
And as you can see here, actually to make the whole thing more powerful and to make birds being able
to combine all the different features, all the different things that it's able to do, we can do both
pre training techniques at the same time.
So you can see here that have two sentences and that we actually want to make is next or not next prediction.
But some of the words from those sentences are masked.
So Bird has to guess the relation between those two sentences while having some missing information
and having to deal with the context and getting information from elsewhere.
So that's really kind of pushing Bird to its limits so that it can perform to the best later.
So that was it for the training of birds and actually for the whole intuition.
But now we are theoretical masters of birds and all we have to do now is to become practical masters
of birds.
So we are done with the interesting part.
Here comes the second exciting part of this course, which is to.
With the applications, so I'll catch you right away to write our first lines of code and see you soon.