Hi and welcome back to this birth course, so we are here to talk about the general idea of births and
more precisely, the fact that it is a pre trained model, that we then fine tune, which is called
transfer learning.
So we train the model in a certain way and then we use this model to tackle another task.
So this is a visual representation of what transferal learning is.
So we have our model right here, which is supposed to be a language model.
So to be able to produce model representation of words of text's.
So we train it in a semi supervised way, which means that we don't have a specific task, we that doesn't
come with labels, all we want to do is to use a huge corpus of text in order to learn to our model,
to our eye, to understand our language.
So, for instance, the most common way to do that is that we could take a huge load of books and the
whole Wikipedia corpus, for instance.
And what we do is that we take all those sentences or all those paragraphs, all those sequences.
And the idea is that we hide the last part of the sentence and we ask our model to be able to predict
the next word.
So we give it the first inwards and we want to predict the end plus one.
What if our model is about to do that?
That means that he understand how context works and how words are related to each other and how to give
meaning to them, because just by getting the first inwards, he's able to guess the next words I just
realized that I'm talking about and they are saying he so that's a bit creepy, but must be prepared
that I think I will do that a lot.
So so that's basically it's huge, huge amounts of copies, huge amount of texts, just trying to extract
meaning out of it by trying to predict words with the context.
And that's a pretty training parts.
So now we have an API that takes the text as inputs and gives us a vectorized representation of this
text.
So most of the time it would be one vector per words of the text.
And so those vectors are supposed to convey the most efficient and the most versatile information possible
about the words and its context.
Now that we have that, we can transfer what we learned to our model for a specific task just by adding
a small output layer at the end of it.
So if we take this example right here, we are trying to build a classifier and we can guess here that
it's probably a on the listserv for movie reviews.
So the idea is that we take the picture and model rahe.
We use it at the very beginning of our personal A.I. We add a small a small classifier and for instance,
with words that will just be one dense layer.
So it could be really simple and we train it to get the right output.
And as this part was called, the training, these parts is called fine tuning and we call it fine tuning
because actually it's well, it's heavy than traditional classifier training.
We will need a lot less data and also a lot less time because there is a huge part of the job that has
already been done, which is the parts about understanding language.
So all we have to do is to focus this understanding for our particular task.
So that's it.
For what terms?
For learning is.
Now, let's see how transform learning in l.P used to be done before, but so what were the algorithms
that were performing in this area before and how they were implemented?
So the first one was Elmau.
You can see that be researchers are huge fans of season pass.
But anyway, this model so this language representation model used LSM, which is kind of enhanced Arnon's.
We don't really need to go into details.
And the idea is that they wanted it to be bidirectional actually just pseudo bidirectional, but they
wanted to represent each word and to predict each would to have access to the context on the left and
on the right.
So what they used is those LSM, which is a sequential model and the process each sentence twice.
The first one is by going from left to right.
So this allows each words to have access to the information from the left contexts.
And then they also apply the architecture from right to left.
And this time this allows any words to have access to the information from the right, so to the right
context and then the same.
Or they concatenate those informations to get a representation for the words.
So doing the pre training phase, for instance, they will teach this part of the architecture to predict
a word based only on the previous left words.
And they will teach these parts of the architecture to predict the words using the right context.
So you can see why it's quite powerful, because it uses information from the whole sentence, but it's
still not perfect as it's pseudo bidirectional.
And the reason why it's pseudo bidirectional is that for the words, we can't use the whole sentence
at once.
We have access to the left context and the right context separately.
And that makes actually a huge difference with having access to the whole sentence at once.
So that's the first thing about L.A. and the second one is that it uses Arnon's actually it uses elastin,
but that's just an enhanced version of the Arnon's.
The idea is that it's a sequential model and we will see actually in the next section while Arnon's
have some big downsides in Alpay.
But the idea is that first we kind of lose information if we have long sentences.
The information at the very beginning, after many iterations, can be fatal in a certain way.
So with LSM, it's less important that with Arnon's, but it's still the case.
And anyway, just the fact that it's a sequential model is kind of disturbing to process text and to
do NLB because we feel like it's not how language works.
And we would really like to have access to the whole sentence, to all the words at once in the same
way to give the same importance to each word.
Or at least if we don't give the same importance to each word, we want to have a reason for that.
We don't want the architecture to decide for us, which would we have more importance.
So here is the second model that used transfer learning, which is open.
And this time it uses transform, actually, just like Berts, but it wasn't able to be bidirectional.
We will also see right after that what is transformed and how it works and why it's so powerful.
But very quickly, the idea is that's to compute each state right.
We use what we call the attention mechanism, which means that these words will have access to all the
other words and it will see how they are related to it.
So each word has equal access to all the other words, and everything is mixed together in a clever
and efficient way to have a great representation of the words and of the sentence and of course, all
of the language.
But here, of course, for training purposes, we can't allow the information to go everywhere if we
want to predict this next word.
So we will say the third word.
We can let information go in every direction, because if we also let information go backwards, that
means that the third words will be able to go on the right and back on the left and it will be able
to go everywhere.
So to predict the third words, we will have access to to the third words, even if we shifted by one,
as we usually do for training.
What I mean by that is that you may think that it's ridiculous to want to predict the next words.
Number two, as the information from the number.
Number two goes this way.
And so actually, if we say that here, we want to predict what, number two, that's ridiculous, because
we have one number two here that goes through here.
But actually what we do is that we add here usually starting talking, which means that everything is
shifting.
Right.
So this is actually not the second word is the first words.
And he we want to predict the first.
Would he?
We want to predict the second words.
But here is a starting talking, first word, second word and so on.
We will see that in details later with the transformer.
But it's just to show you that if you want to predict the third words with this technique, we only
have access to the starting talk in the first and the second words, which makes sense.
But if we let the information go in all the directions, the information for the third word, which
is here, the fourth Duquan will be able to go backwards, to go everywhere and to go to the third slot.
That is supposed to be our prediction for the third words.
So that is why we have this open, which is based on transformer.
We had to stick to the left to right context so we don't have access to the whole sentence, unfortunately.
So those were the two main transfer learning models that were used in MLP before it arrived.
And actually what words to do is to combine both principles so it is able to use Transformer and the
bidirectional and actually even more bidirectional that element before because Demo was pseudo bidirectional
and that is fully bidirectional.
So you can see that like for Elmo Orozco in all the directions and it's even more powerful because they
can go in all the directions, in the same architecture, in the same sets of neurons.
While for Elmau we had a set of Ewan's that was going left to right and instead of nuance that was going
right to left and it uses Transformer.
But you might be wondering about what I said earlier about the fact that with Transformer, it doesn't
make sense to let this happen because each prediction will have access to the wood it is trying to predict,
which doesn't make sense.
And actually this is possible with the training method that they used with Berts, which is quite clever,
which is called the masked language model, small ball.
Right now, the idea is that instead of trying to predict the next words while letting everything available.
We actually remove the words that we called mask, and we only try to predict this one.
So that means that to predict these words, that is the masked words so doesn't convey the same information
as before.
We have access to everything, but we'll go into the details later.
So let's have a quick look at this information flow summary, as I said before, we can see that this
seems to be the more efficient and complete way to transfer information.
So if we could be able to make this work to give sense to training boats, that could be used.
And of course, that's actually what Google did.
And so we can see here the flows in those systems, this one, which is not bidirectional at all, and
this one being sued bidirectional and using, unfortunately, elector's, which is less powerful than
Transformer's.
Finally, some numbers and further information about the training phase of birds.
So birds use the book's corpus, which is the corpus of books, of course, and it contains about 800
million words and the English Wikipedia, which contains two thousand five hundred million words.
And the idea that they need to use huge corpuses or at least corpuses that contain sequences of sentences,
just using some isolated and random sentences would not be as efficient and will not allow us to compute
high level information representation.
Also, the training contains two tasks.
The first one is to predict the word.
So that's what I just quickly explained before we make a word in the sentence and we are trying to get
it back to guess which word was.
And the next one, the second task for training was next sentence prediction.
So the idea is that we put two sentences, one after the other, and we try to guess if the second one
is the natural continuation of the first one.
So does it make sense to put one after the other or are we just randomly picked?
And last information, which is quite important is that with this whole corpus, that's actually not
everybody can afford to get it still takes about three days on 16 to be used to train.
So the Tipu is basically an enhanced GPU that was developed by Google.
And it's not better than GPU for all applications it spends, but for birds.
And actually, Google kind of developed CPU's for that's it's just more powerful.
And so that means that regular mortals as me and maybe you will not have the opportunity to create their
own birds and to train it.
But fortunately, Google made the birds models available so we can all have access to the pre train
birds models.
We can all have access to the weights and we can all fine tune it for our personal tasks.
And this, the fine tuning part is way less demanding in terms of competition, time and power.
So that is why we will not build the bird call and we will not train because that's just almost unfeasible
for us.
And last thing that is important to notice about this transfer learning thing is that Bird is a language
modeling system.
That means that it's kind of a super encoder.
It is able to encode a sentence into a list of vectors or a vector in a very efficient way.
But it doesn't come with a decoder parts and there is no equivalent decoder for that.
That means that that is not suitable for some tests yet for the moment, like, for instance, standard
sequence to sequence tasks like building a translator or chatbot in a standard way.
This will not be possible for birds, at least for the moment.
But it's still really powerful, actually, the most powerful algorithm for text classification, which
is a huge part of tasks and to get the meaning of sentences.
So, for instance, to compare two sentences to see if they have similar meanings.
It's really powerful and it works really well.
Also with question answering tasks.
More precisely, if you have a paragraph or a huge text with a lot of information and you just ask a
question too, but it will be able to retrieve the appropriate sentence in your paragraph that answers
this question.
So that's also really useful and but performs amazingly well for that.
So that was it for the general idea of birds and more precisely about transfer learning, how it works
and how we use it now, we will be able to go into more details about birds, its implementation, how
it works, and we start by the history behind birds.
So we will go a little bit in the past dealing with our this how Armenians were applied to NLB and how
little by little adding some small tweaks or improving the models and finally changing it.
We could get to the transformer, which was finally the basis of birds.
So let's go into it.
Let's see the details of all these models.
Let's see the foundation of birds.
And finally, we will be able to understand the architecture of the bird model.
See you soon.