Hi and welcome back to this birdcalls.
So now that we have all of our tweets that have been cleaned and that we have our labels that are clean,
too, we are ready to go to the tokenization phase for our tweets.
So this is where we will use for the first time a birth tool.
And so we will create the tokenization tool that comes from bots.
So the way to do that is to create full tokenized, which is just birds.
So we call the birds module and from that day with the birds tokenization that exists.
And in this they are different things that exists, but the one we use is full to organize it.
So now we will be able to create our tokenized.
Just calling this function, we will.
This will be an instance of the class photo Kenenisa.
So now that I have done that, we need actually to call and to create a bird layer.
So a huge birthmother we need to call it because there is information about the tokenized in it.
And more precisely, we will have to use the full Capsis and an information about whether we apply lowercase
to our inputs or not.
So let's create how exciting is it our first bird layer by calling head?
That's what I explained before.
So 10th floor hub is where everything is stored, all the tweets and models are stored and we call from
Hub across layer.
So we will call birds as a layer and then we need to give it the address, the web address for it.
So I want to type it by hand.
Of course I will cheat and just to copy paste from the place where I stole some chits for that.
So that's it.
That's the Web link for the model.
Of course, you can find it if you go to the tenth floor hub website and you explore it, you find birds,
you find the different models.
And here is the one that we use for this course.
Of course, I use the little one because it's really easier and faster to train.
And so we can see here, as I explained before, that we have only 12 stacks of encoders while the large
version has twenty four.
And that's it.
You can see all the others hyper barometer's.
So that's actually the version of Bird that we load and we need to specify whether we will train it
or not.
So will we use it for fine tuning or not.
And here, of course, trainable equals false because we only used to have access to information about
the tokenized.
We don't want to change and to fine tune the weights of birds so we can use trainable equal force so
that it's that will give us the best player.
And from that we can get the information we need for the organizer.
So the first one is vocab file.
So that will give us all the all the vocabulary for to Conesa and that's just Berthot underscore layer
that resolved objects that vocab file that ascites path and we call it as Nampo.
So nothing really to understand that it's just the way to have access to the vocab file for the tokenized.
As an MP, this is how you call it.
And then we need information about lowercase the text or not.
Saidu Lowercase equals again that the player resolved objects and do lower case.
That's by.
No, but here we are.
And finally, with all the information we are able to create our tokenized, so tokenized will be an
object from the class full to Conesa and we will use vocab file and do.
Lowercase, that's.
Let's read that it takes a few seconds, actually, to load the birth layer because as we saw before,
there are a lot of waits for this version.
We saw that it's one hundred and ten million Parmeter.
So, of course, it will take some time.
But let's just see if everything went perfectly.
Actually, it works better if I don't do a typo.
OK, so while it loads well actually that showed that this this has been lowered, it's just the lower
case that felt that it's OK.
So before tyrannizing our sentence, let's just see what we are able to do with the token either.
So that's, for instance, called the most standard tool from the organizer.
So tokenized that tokenized and applied to a sentence like.
My dog loves strawberries or whatever.
And we apply it's OK, we can see that it has tokenized our sentence and here we can see the speed between
zero and there is so strong, there is a combination of 12 and Bury's.
That's what we talk about, the tokenized and how it works efficiently in that way.
And second thing we can do actually with the organizer is to call took Annisa.
That converts actually tokens to ID and we apply it to the tokenized sentence and we get a list of IDs.
So it's taken from the organizer is related to a number.
That's what we saw before and that is what is used by that.
So that was it, for quick example, to see how we can do with it.
Of course, I used converted to it, but we can do the thing the other way around so we can convert
ideas to Dawkins to do the same thing backwards.
But for now, let's define our function and code sentence that take sentence as input and return.
What I just wrote before tokenized that convert tokens to ID applied to the Conesa that's tokenized
our sentence.
So this will just apply what we did before to every sentence, but running a loop of everything so data
inputs will be the released.
So the sequence of encode sentence applied to a sentence for sentence in data clean.
So that's it.
That was it for the tokenization phase we have called our first birthday.
We have used it to have information about our tokenized.
And finally, we are here in the process of tokenized every sentence so that we can create valid inputs
for our module.
Talking about the inputs, that's exactly the subject of the next session.
So we'll go into the final data situation right after this.
This shouldn't take more than a few minutes once again.
And so once this is done, let's create data sets and Tucson.