Hi and welcome back to this birdcalls.
So we are here for this second application where we will really start to use birth and more precisely,
we will take exactly the same task as before with the same data and the same based models.
So this CNN, but instead of using a homemade embedding layer, we will use both as an embedding layer
so we will not train the weight or the variables of boats.
We will just use it how it comes, but we will use it as a vector representation for our words and we
will see how it improves our model.
So this is a technique that you can use in any NLB personal model that you already have, you can plug,
but as an embedded layer at the beginning of it and get a more powerful and more efficient model with
that.
So, of course, we want change of all things in this application.
This will be quite similar to the previous one in terms of dependencies, the data processing.
There is a lot of things in common, mostly the cleaning pots and the model.
We will just change a little thing.
Of course, the embedding layer and the way we train and we evaluate it will be the same.
So there are still two main things that we need to go through if we want to apply those changes.
And the first thing is the inputs, because our first layer for our model will be a layer.
So we need to make the inputs suitable for this layer.
So I will explain to you how to do that and what exactly both needs.
And then, of course, we will go into the model in order to know what we need to change to have a performing
embedding with that.
So let's go into tokenization phase to see what we need to to change.
So as I explained in the theoretical parts, we need to add some tokens to the input of, but we can
just give the tokenized version of our sentence to.
But we need to add at the very beginning of seamless token, which is the token that is used for classification,
and then we need to add a sceptic into separate both sentences.
So in our case, we have only one sentence.
Of course we are performing classification on only one sentence.
So we will just add Decepticon at the end and then nothing.
And this is actually exactly what we will need as input for.
But if we do a single sentence classification, so let's write our new encode sentence function.
So that takes a sentence and this time it will return.
Our Duquan.
S. s.
That we can coordinate with the took ENISA, so this would be the list of the things that technologies
applied to sentence.
And finally we add this stock in the SEP.
And here we are.
So this actually gives us pretty much the same tokenized version of the sentence as before, but we
just add this element at the beginning of the list and this second token at the end of the lists.
So now we do the same thing as we did before.
Data inputs would be equal to encode sentence applies to each sentence for sentence being in data clean.
So this doesn't change.
So the important things here was just to add the tokens that we talked about in the recent past.
And now while this is winning, we will talk about the inputs that we need.
So a butler takes three different inputs.
First, it will take, of course, those elements.
So the tokenized version of the sentence that we created there, and it will also need two additional
things.
The first one is a list of mask.
So that's actually just something that indicates to.
But where are the values in our sentences?
So this will just be a one wherever we have standard tokens and as you will when we added a padding
token.
So that just says to birds, which are the tokens that he must not use in order to process the data
and to make a vector embedding, in our case, we did not use the but token the purging phase will come
after.
So that's not really important.
So basically at this stage, we will only have once with the masking function.
But that's OK.
We don't care about that.
And finally, the third input that we need is what we call the segments input.
So this is just once again, a sequence of ones and zeros and zeros will indicate that we are currently
in the first instance.
So this will just corresponds to the tokens of the first sentence and once will correspond to the tokens
of the second sentence.
So let's write those functions right now.
So Def gets IDs tokens.
So that's the standard function that will return the IDs for each token.
So the number instead of the string.
So get IDs equals return to Conesa.
Can I say that convert's?
Talking to ideas.
That's the first function that we need.
So the first type of input for the third layer, then the second one is the mask that corresponds to
the padding mask.
So.
For Dawkins, what we return is and B, that chart does not equal.
So that's what we use when we have a list of strings in order to make comparisons between a list of
things and a single string.
So is that right now we compare each element of our tokens with the string pad.
So string that.
So this will tell us whenever a token is the talking.
Well, actually, because we are not equal, it will tell us the opposite.
So it will be equal to one whenever we are not using a bodging talking and zuo when we are facing a
talking.
So we do that.
And this is a list of booleans and we want to take us.
So we just do a stipend.
And so now we just have to write the third function, which will indicate whether it belongs to the
first sentence or the second sentence.
And so those are the SEP tokens that will help us doing that.
So until we get the first Sheptock and we will use zeros as those segment IDs and right after the first
token we will get one.
And an important detail is that's the first step, must be a zero.
So let's write this function right now with def get segments.
We take tokens.
And we will create the empty lists of our tokens that we will return, then we will say that the current
idea that we will use zero because we start with zeros for the first sentence.
And now let's look over the tokens with for talking for talking tokens.
We will just segue I.D. that up and and we add the current SEC ID.
And then what we want to do is that if we have passed the first sentence, we want to switch the and
so we will say Doug equals equals.
If talk equals equals the token step, then current say it equals one minus currence sake.
Heidi, this way we turn as you go into one and one after a zero.
So that will also handle the padding because after the second step, because at the end of the second
sentence, you must add the second step.
We don't use to sentence here, but if you use the sentence option from that, you have to use a set
at the end of the second one.
So this will also handle the padding because after the second step, we will get back to zero.
And so that will be perfect for the padding.
But in our case, that's not important because we will pay later.
And finally, we can return that with return sig ideas and that's it.
We have our three functions that will create the three different types of tokens.
And finally, while we have to do is to slightly change the sorting process that we made before, because
we want to have only one sequence of tokens as input.
We will have three of them.
So let's slightly change that with data with Len equals.
So this doesn't change since data labels I and Len sense.
So we keep our sentence, the label that corresponds to its and its length for I sense in enumerate
enumerates data inputs.
So that's exactly what we did in the previous application.
Now the next step is to optimize that so that we mix together the positive and the negative sentences
so random that shuffle data with Len.
After that, we want to sort it according to the length so that Saud's key equals Lambda X and we use
lowercase and we use the third element of X in order to sort our lists and finally sorted.
All this is where we need to change something equals.
And here we will use our three functions.
So first, we want to give the ideas for each token.
So since we apply get ideas to Central Hub zero, which corresponds to our sentence, we also need the
mask.
So get mask since lub zero.
And get segments.
Also applied to the sentence, so since lub zero.
Well, central to this four sentence labor, and that's what we are looping over, so finally we have
defined the inputs inside those brackets and we, of course, need to add the label of sent lab one
that it we have finally all the elements for one item of our data sets.
And so this is for science lab in data with plan and only if sent lab two to the length of our sentence
is greater than seven, for instance.
OK, so that's it, we have just made the changes that we needed in this cell, which were to not only
use the sentence that we had before, but to apply the three functions that we wrote here in order to
have a complete set of those three sequences of tokens as inputs for our birthday.
So that's it for the modifications we had to apply to the inputs when we use a birth layer.
So now we are ready to make the changes for the model and of course, to see how it affects the training
and the accuracy and how we will perform better using that option.
Need to see all that.
Let's get into it and use an.