Hi and welcome back.
This birdcalls.
So now that you have seen how the transformer works, we are ready to go into that architecture so to
know how it has been built based on the transformer.
So let's go into that right now.
And actually, you can see that it's pretty simple.
The most important thing about that was actually to get the transformer because birth is simply a stack
of encoder layers from the transformer.
I just put here a quick reminder of what the encoder is made out from Transformer.
So this additional layer followed by a IT forward, Sibilia.
So that's basically what it is.
But it's just a stack of encoder layers which allows it to encode the sentence, encodes a language
basically in the most effective way.
So be composing information between every word of the sentence according to the relations between each
other.
So, of course, there are other details to see.
And that's what we are going to do right now, mostly about the inputs and the outputs of birds.
But actually now you get the architecture of birds, you have it.
And while it's not the only thing that is important to it, but we will have also after that to deal
with the training phase to understand how it has been trained, because this is actually the second
important thing about it's about birds and this is how it achieved to be bidirectional and to get those
huge results.
But before going to the details for inputs and outputs, just we will just have a look at some small
numbers about the architecture.
So in the paper and also on GitHub, actually, Google talks about two different models that the choice
that they implemented, the first one that they called Burt's Bees, and the second one which is bigger.
And so, of course, more efficient that they call it bird's large.
So they use three different hyper parameters for that.
The first one is El, which is the number of encoder layers.
So that's just this number you can see on the stack of encoder layers here.
You can see twenty four, it's twelve.
So that's the first hyper barometer for the architecture of birds.
The second one is H, which they call hidden size and we can call it embedding dimension.
That's actually the size of the vectors that are used between and inside and the layer so that here
the size of the vectors for the inputs and the output of each encoder layer and the last number, the
last hyperbola into is a number of self attention heads.
So if you remember for the transformer architecture, if we go into details of the mode attention layer,
this Motin attention part means that each vector is separated into spaces and the number of those subspecies
is actually what they call number of attention heads.
And that's what this represents here.
So in the paper, the present two different models and the first one base is with L equals 12 H equals
seven, six, eight and 12.
And that's for a total of one hundred and ten millions parameter.
And that's a lot.
And that explains why it was so huge in terms of time and computation, power to train this model.
And this is the one that we will use in the course just because it's easier to train, it's easier to
build the model.
The training goes faster and for explanation purposes, it was easier for me to deal with this one.
But of course, once you have a fully performing model, you can go with the larger version of, but
it will just ask a little bit more time to train.
But you will just use exactly the same thing, just calling a different model and you will have better
results.
And so this large model uses twice as many and Cordelia's as the base one and the Heyden dimension is
a bit higher.
It's one thousand twenty four.
And finally, there are more heads in the self attention, Sibilia, and we have a total of three hundred
forty municipalities.
That's huge.
So that's what's for the general architecture.
As I said, we have to now deal with the inputs and outputs, how they look like.
So we want to have inputs that have some flexibility.
The first one is because we want to be able to represent both a single sentence and a pair of sentences
for inputs so that we can tackle many different tasks and not just classification tasks.
So if we take the example of question answering, so if you have a question and you want to find the
answer inside a big text, I hope we have, for instance, or a Wikipedia article, we would like to
separate our sequences into two different parts.
So that's one thing we want with the inputs.
And also we consider both as kind of advanced and better.
That means that.
We get four hour sentence, a sequence of vectors, its vector being the representation for the words
in its context, but we could also benefit from having a more specified classification tool.
So instead of having one vector per words, we would like to have a vector that could be directly used
for classification.
So a vector that could in a certain way summarize the whole sentence.
And so those are the two main points, two main features in the bird's inputs.
And so in order to achieve that, the inputs will actually look like that.
So let's say we have two sentences, the sentence, a sentence, be knowing that sentence B is optional.
If we just want to perform classification on one sentence, we will, of course, use only this one.
But let's say for the general idea that we have two sentences, we will at the beginning, the CLX,
which is classification talking.
So that's Duquan.
That doesn't convey any meaning.
It's just that at the outputs of birds of the birds model, we will have a corresponding embedding for
this token and that will actually represent the representation for the whole sequence that comes right
afterwards.
So that's actually a token that is made to create a classification vector.
At the end of that, we will see how it's made in the free training explanation right after that.
And the second important thing is that our two sentences are separated by another Home-Made stock and
which is set for separation, of course.
And that would just indicate to our model that the two sentences have a different role in the whole
process.
So that's what it will look like.
But more precisely, what do we want our sentence to look like when they enter our model?
Of course, we can't just give a string to, but because string is quite hard to deal with for computers,
at least of letters doesn't really make sense.
So we'll take this example like he likes playing.
Very simple sentence for now it is just a string.
So we want to use that's the first thing we will do is tokenization.
That means that we will split our sentence into a list of individual tokens that we will deal with individually.
And that's quite an important phase, actually, because each of those tokens will be related to a number.
So, for instance, he will be the world's number one hundred twenty five likes will be the world number
two thousand eight hundred and seventy six, so that we actually get numbers that our AI, our algorithm
can deal with.
So we need our tokenization to be clever.
Like you see here, we see that playing is of course the verb to play with the energy form.
So saying that playing is play with the IMG talking at the end is actually pretty clever because it
conveys more meaning and it's more flexible.
You don't have for each verb to have two different tokens, one for the original verb and one for the
in the form.
You can just have one token for the energy form and then added at the end of every verb that's much
more powerful and more flexible.
And also it allows us to deal with words that we might not have seen in our corpus, splitting words
like that into known smaller words makes our tokenization less stiff.
We'll see that into details later.
But the important part is that we had a string and now we have a list of tokens.
Now that these have been done, the token ISO comes with a token to ID convertor.
So that means that for each of those tokens in a string form, we have a corresponding number that we
can use later.
And of course comes the next step, the embedding, which means that each of those token so we see strings,
but it can also be considered as a single number will become vectors.
And now we have really something meaningful that we can deal with and then we can work with with our
birth model.
So this is the whole process that comes before applying the stack of encoders for.
But we have a string or several strings.
Our sequence, we make it into tokens.
So a list of numbers and then this list of numbers becomes a list of vectors.
And then those vectors become what we deal with in each and called a layer.
So let's talk about that.
Took Annisa, the Google research team decided to use what Pitstick ENISA, which is quite famous one.
So some numbers about this took Annisa first.
It uses thirty thousand words, more or less.
And you can see that it's not a lot actually compared to the number of words in the English language,
which I don't know, we could argue that there is more than two hundred thousand words.
I don't really remember the numbers.
I think they are more words.
But let's say that just by reading and living normally in an English context, you will see more than
two hundred thousand words without considering the conjugations or things like that.
So it's not huge, but it's enough to deal with the most common words, or at least to have a single
token, a single ID for each regular word that we could encounter most of the time.
And so the idea, as I said before, is that it can deal with new words by combining known words and
it will try to use the biggest word possible in order to decompose an unknown word.
And if a word is just a bunch of nonsense, random letters, one after the other, it will split the
world into single letters.
So basically the idea about making it nicer.
So about what piece, for instance, it says to have a trade off between the vocabulary size.
So the number of words that are perfectly and personally represented by the Conesa and the art vocab
would so the capacity to deal with words that are not really fully into our vocabulary.
So what piece is one of the organizers that do that the best?
And finally, as I said before, the idea of the tokenized is that it comes with a convertor from Dawkins'.
So those small individual strings that we saw before and Idiz, which are just numbers that are then
usable by computers, by algorithms.
So let me show you examples of what the organizer can do.
I just wrote this code using the tokenized used by Google.
And if we try to you took an ISO to get the tokens for the sentence.
I love transformer.
Of course, the word transformer will not be into our thirty thousand words because it's not a common
words.
It makes sense that it's not part of the standard vocabulary.
So the tokenized would be stuck in either recognize that there is the word transform at the beginning,
which is a common word, and that exists in the tokenized vocabulary.
So it is transform and then it's used the suffix e r, which also seems to be pretty common.
So that shows how these two can give meaning to the word transformer without knowing it.
It knows that is something that is made to apply transformation to something that's pretty powerful.
And another example, if we are facing a world that makes no sense, or at least not that I know, like
if we take the sentence I love coffee or whatever, you would pronounce that he would recognize that
there is code which exists in English or at least as a prefix like cooperation or co executive or whatever,
and then it couldn't give sense to something with the F. So it's just the V by itself.
And then it seems that's Effy as a suffix, maybe has a meaning or exists in English.
I don't know.
But you can see here how it had to split these words or this expression.
So that's basically what the tokenized are do.
So after the whole process from a string, from a sentence, we get a list of vectors, each of those
vector corresponding to a word more or less, or at least to the tokenized version of the sentence,
but as from the Transformer, but uses different inputs or at least advanced inputs, not just the vectors
that we saw before.
So first, it needs the embedded words.
So that's what we just saw before each test.
It needs to have vectors for each word as inputs.
Then it needs to have indication about whether the word is from the first or the second sentence, because
as we saw before, we use SEP talking between both sentences.
But Burt wants to add information to those embedded words to see and to follow along the process if
the word comes from the first or the second sentence.
And the third thing that we need is a positional embedding.
That's exactly like the transformer is just to give importance to the position of the words in the sentence.
Otherwise we could just switch words and it wouldn't change anything.
And that's not what we want.
We want to be able to give importance to the place of the words in the sentence.
So that's the visual representation of what we get.
This is just the tokenized version of our sentences, so this is the output of the world's best organizer.
And from that, this is exactly what will be given to the and Cordelia's in the birth architecture.
So the first one or the Duquan and beddings.
So this is just the vectorized version of the second one is Sigmund and Beddings.
So those are also vectors of the same size.
And we only have two vectors, actually, one vector that is made to pinpoint the first sequence, the
first sentence and another vector that is made to represent the second sentence in our inputs.
And finally, positional unveilings.
Once again, it's a vector of the same size as the beddings and it changes for each position in the
sentence.
In the sequence we sum everything and that gives us new vectors.
That will be the final inputs for birds for the and Cordelia's.
Finally, let's have a look at the outputs so that we have two types of outputs.
The first one is this single talking right here.
Well, it's not a token.
It's actually a vector, which is called C, and which actually comes from the CLSA classification token
that we added at the beginning of our inputs.
And the second type of outputs is this whole sequence right here that is Nottie.
And that's actually the best representation of each of those words.
And of course, it's really advanced.
It's what birds has been made for.
So it's not a regular representation.
It takes into account the context a lot of learning.
So that's the advanced word by word representation of the input sequences.
So see, this one will mostly be used for classification.
So basically, if you want to process a classification task, a sentimental analysis or a spam detector,
for instance, you will put one or two sentences depending on your task and you will just use this very
vector.
And by applying to it's a dense layer with as many outputs units as classes that you have for your classification
task, you actually get the classifier.
That's as simple as that.
And if you are dealing with a task where you need to go into more details, into the individual words
of your input sentence, for example, I was talking about the question and solving task where you have
a question and you have a paragraph and you want to find the answer into that, then you want to have
a look at each word of the paragraph or the detailed sequence.
And for that you will use the sequence of T case vectors and those two type of outputs, all trends
in two different faces, actually, or they represent the two important parts of the training of birds.
So C comes from the next sentence prediction.
So the idea is just to guess if those two sentences, the first sentence A and sentence B are related.
And those talks just come from the mosque language model, which is the standard prediction task that
we do when we do for training.
And that's actually the topic of the next lecture.
Now that we have the whole architecture and we know what bird is made of, all we have to do is to learn
how it has been trained and how we gave meaning to all the stuff that we have built.
So let's do that right now.
And so using.