Hi and welcome back to this bird course.
So now that we have made all the changes that we needed to do for our inputs in order to use birds as
an invader in our CNN, let's go into modifying the model and evaluating the new performances and see
how better it performs compared to our previous version with the homemade embedding layer.
Just before doing that, I think it's important to have a look at what we can get with a bird, because
this is actually the first time that we create a bird layer and that we have the tools to use it now
that we have created three different kind of tokens that we need to give it in order to have the bird
outputs.
So let's do just a quick test and see what we get when we call a butterfly.
So what are the outputs?
In order to do that, I will create a tokenized sentence.
So my sense equals we start, of course, with the C.
L.
S token, then we will tokenized a sentence, tokenized that tokenized and our sentence will be, for
instance, roses are red that sits.
And then we, of course, add the sap.
Well, the sap talking like that.
So this is our tokenized sentence, and what we have to do is to create the three different types of
of tokens of input that we need to give to birds and then we will be able to call the fly.
So let's do that by calling bird player and we need to give in the list the three different inputs.
And there needs to be Tensas, of course, because we it's just potential flow modulations of layer.
And it needs to add like a batch.
We need to simulate the much.
So we will need to add a dimension at the very beginning to pretend that we are using batteries.
So for the first type of inputs, we need to call T.F. that expand DMS and we will expand the dimension
of 12. costs.
So we make our list of tokens into a tensor and this list of tokens is just gets ideas.
So the idea for our sentence, my sense.
OK, so we are starting to see that in 32.
So this will be integers and the dimensions we want to add is the first one.
We want to simulate the budget along the first dimension.
So that's our first set of inputs for the bird layer.
So let's simply copy paste in order to have the second set of inputs, which this time are not the ideas,
but the mask for the putting the padding steps.
So get mask.
Everything else is, is good is what we, what we need.
And finally the third one gets segments.
So here we change it to get segment.
That's it with an S..
So here we have the input for the bird flyer and let's run the sale to see what we get.
So as expected as we saw in the intrusion, but the input is made of two elements.
The first one is a Tensei of shape one seven sixty eight.
So one being for the simulated match and seven sixty eight.
That's the dimensions.
So kind of the embedding dimension.
So this is just one element and that corresponds to the CLX talking.
So this is what we would use for a classification task.
This is supposed to represent our whole sentence and our whole sequence.
And the second element that we could see is right here and now, the Tenzer.
And this time the shape is one six seven sixty eight.
So one again for the batch and six is for the number of tokens in our our inputs.
And again, this is the amazing dimension of the hidden side of the model.
So this time we see that this behaves like a real invader.
So we get a vector for each of our inputs, tokens, and we can see that six is the right number because
we have kleiss, roses are red dots and SCIP.
So those are the six tokens that are embedded in a certain way or that are represented by birds.
Hey, so that means that each time he wants to use birds, we have access to two different types of
outputs.
And depending on the application that we want to build, depending of what we want to do, we will use
the first one or the second one.
And as I explained, the first one will mostly be for classification task.
And the second one, we will use it each time we need to go into a Dukan level of specification.
And that's actually what we want to do in this application because we want to use but as an embed.
So we want to get a vector for each of our words, each of our tokens, and that's exactly what we will
do right now.
So let's have a look at our new new class for the classification task so we won't write it again, because
there are a lot of things in common, the things that change, of course, the name of the class that
are changed.
So it fits its purposes.
And same thing here.
I just removed the parameters that we do not need any more, like, for instance, the vocab size of
the embedding dimension.
Those were the parameters that we wanted to give to our embedding layer so that it was built as we want
it.
So that's just for the definition of our class.
And so now we can see the first important thing that's changed is that instead of creating a Twitter
cross, the players start embedding.
We just call what we did before for the Annisa.
We call it a cross layer from Hub, which comes from this you are URL, which corresponds to the birth
layer we wanted to use.
So of course, I used the once again, the base version of birds.
So with only 12 encoder stacks, if you want to have something even more powerful, you can use the
24 versions.
So the both large and when we call it the very important thing to do is to mention that trainable equals
force, because as I said in this application, we don't want to fine tune, but we only want to use.
But as it is like in a frozen way so that we get a number that we can use in a full MLP model.
So here in our CNN, so we freeze the layer, the both layer using trainable equals force, then everything
is pretty much the same for the definition of the following layers.
And important thing here is this function that I wrote in order to well, to just call the embed using
words.
And that's exactly just the thing we wrote in the cell before.
I just called the layer.
And the important thing is that our inputs contained the three different types of tokens.
And so we need to access them here using all the batches.
Hey, this is the first one corresponding to the IDs, the second one corresponding to the mosque in
here corresponds to the segments.
So whether it's sentence AOB and here we get, of course, all the values, so everything else.
So that's just the small thing that we need to to do in order to manipulate our inputs in the right
way.
And we call both layer.
And as we saw before, the first element is just one vector that is mostly used to represent the whole
sentence.
But here we want the representation of words individually.
So we only really use the second part of the outputs.
So that's why I leave this blank with an underscore and only keep the second part of the outputs.
And this is what I return in this function.
And so this function is actually exactly the same thing as a regular embedding layer.
So that's it.
We can just use it by calling it here.
So instead of being self-taught and bedelia, this is just stuff that's embedded with Berts.
So we call this function that we call the birthday and then the rest is, of course, exactly the same.
So as nothing changes for the code, for the training, let's have a look at it.
Right now, let's see what changes.
So, of course, there is almost nothing different except from the class that we call that is different
and we don't need the vocab size of the invading dimension.
And this is just how we create our new model.
Then the compile phase is exactly the same because we train in the same way and we assess the performances
exactly the same way.
We do the checkpoint once again and here we will see the results.
And again, this is really similar to the previous application.
We just called the fit methods of our model and this is what we get.
So first thing to mention is that the training was way faster than before just because we did not have
to train an AMBEDKAR, which is actually quite a huge layer.
So if I remember well, training this, I was approximately twice as fast as training the previous one.
And you might be afraid to see that the accuracy on the training set seems to be lower.
But that's not a problem because you can see if you go to the evaluation when they use evaluates on
the testing sets, that I got a higher accuracy than before.
So without using birds as an invader, we had inaccuracy of eighty four point five, if I remember well.
And here we have more than 85 percent accuracy, only changing the embedded with bits.
And we have to keep in mind that the rest of the network might not be optimized for this and better
because I decided to keep exactly the same thing that was optimized for the previous version of the
eye just to see how it would improve things with this small change.
And it was twice faster to train and we still had better results.
And in my opinion, what is really, really important to notice is that we seems to have way less of
a fitting with that solution.
And that makes sense because we changed a huge part of our eye that was made specifically for this task
with these data sets, with something that was trained with a more general and a bigger corpus.
So that's normal that we have less of a fitting.
And that's our I now seems to be more flexible and to have less bias.
So that also means that we if we were to use another testing sets, that would not come from Twitter.
So that would not be similar to our training sets for sure.
The second eye using that as an embedded layer would perform way better because it would generalize
in a better way.
So things to remember about these spots.
The first thing is that we need to give three different types of inputs.
So we need to create the IDs, the mask and the segments for both layer.
That's the first thing to remember.
The second one is that with a birth layer, when we call a bird, we have two different types of outputs.
The first one will be one vector representing the whole sequence, and the second one is a list of vectors
representing each word individually.
So according to our needs, we will need one or the other here for a number.
We use the second one.
Third thing that is important, if we want to use that as an embed and plug it to a full MLP model,
we need to specify that trainable will be false because we don't want to train it.
We don't want that to be the core of our model.
We just want it to be a tool that we plug at the beginning and that we want to be frozen.
And finally, fourth thing that might be important to remember is about the results, of course, is
that without making a whole new optimization of the hippopotamuses, we still got better results on
testing sets with the training being twice as fast.
And finally, we get in the eye that is less biased.
So that was it for the second way to use.
But the second type of improvement that you can add to your analytic models with books.
And now for the last one, we will apply both in a fine tuning fashion like it was intended to be.
And so I hope you're excited to finally see how to use Batar, Google Thouret Plus in order to not be
stuck with the same problem forever, we will tackle another task, a question answering system.
And so I can't wait to present you that and to show you how we will train birds in this manner.
Popularity.
Brace yourself and see you soon.