Hi and welcome back to this because so let's start building our squad birth model by creating the squad
layer, so the small area that we will add right after birth in our model.
So as I explained, this will be made as a simple, dense layer, but we will need to use it cleverly.
So for each of our output vectors, we will apply a dense layer with two output units.
And so we will get two numbers as final outputs for each Dukan.
The first number will be as cool for being the beginning of the answer and the second number will be
the score for being the end of the sentence.
So for each input word, we know the score for being the beginning and the end of the sentence.
So right after having all that, all we have to do is to reshape everything so that instead of having
so let's say, for instance, for the paragraph of two hundred words, instead of having two hundred
pairs of numbers, so to hundreds of scores for being the beginning and the end of the answer, we want
to have two lists.
The first list being all the scores for being the start of the answer, and the second is being all
the scores for being the end of the answer.
And so that will make the training quite natural and more precisely, the last competition more natural,
because our data set right now is quite certain that it's actually really close to what we did for the
sentimental, nicer for each element of our data set.
We had the three different types of tokens, the IDs, the masks and the segments, ideas and outputs.
We have two numbers, which are the positions of the start of the answer and the end of the answer.
So basically our targets are two numbers, beginning and end of answer.
And so we want to return to list of scores, the first one being the scores for the position of the
start of the answer and the second list, the second output element of our model being the list of scores
for being the end of the answer.
So let's start doing that right now by creating a layer the birds squad.
Layer, which inherits from T.F. that cross, that layers, that layer, and we will need, of course,
to define in it that we take self, that's it.
And to call super with.
But.
Squads, liar and self, and we use it to call in it from the lower class and we thought the are what
we get is to create our final dense layer.
So self that final dense will be a WTF that carus that layers that dense.
And as I said, we will have to output units for each word.
We will finally have two numbers as output and the initialise we use the initialise that's ISO Google
using for that, then Slayer's for fine tuning, but which is Curnoe in Socializr equals T.F..
That's Carus.
That's initializes.
That's and here comes the truncated normal and the standard deviation.
For that we will use a point or two.
So that's what Google seems to think as optimal for a dense layer in frightening.
But so quickly.
What is a truncates normal?
That's actually the normal function, also called the Gorshin function.
Let's have a look at that.
It's just this kind of bad shape function that is quite standard in mathematics and computer science
and whatever.
And we say truncated normal because we get rid of the lowest and higher values.
We don't want to have very, very high or very, very low values in our initialization.
So we just keep the middle parts and the standard deviation just indicates how we spread our functions.
We can see different values for the standard deviation.
So that's what truncated normal means.
So let's get back to our layer, we have defined everything that we need now we can def call that will
get self and the inputs, and so we will get our logits by calling self that final dence inputs.
And as I said, with that, we have four lithographer of land to hundreds.
We have two hundred pairs of numbers.
But what we want to do is to have a pair of lists of size, two hundreds.
So we just want to do the bit of reshaping and to split the scores corresponding to the beginning of
the answer and the scores corresponding to the end of the answer.
So we start by just doing a bit of tfw that transpose.
So that means that we will invert the order of the dimensions we will apply to Logits.
And what we do is that we do two zero one.
So what this means is that the dimension to that was the last one.
So the third will come to the first place and then the zero, which was the first dimension.
So the one that corresponds to the budget will come into second place and the dimension number one.
So that's the second dimension will come into the last place.
And this one represents the length of the sequence.
This one represents the sequence of all the words.
So now let's put this into two huge vectors, the first one corresponding to the starting scores.
So that's the first number that we get out of our final then layer for each vector.
And the second one is the list of scores for ending the answer.
And so that's the list of the second numbers we get out of this final dance applied to each output vector.
So we do that by defining and starts logits.
Logits equals T.F. that and such.
So that means that we split all objects.
With respect to the Axis C. So our new first dimension here, which corresponded before to the last
dimension when we called our slayer, is off sites, too, because we had to output units from our Denselow.
So here are calling that we had.
We will write this something of Dymaxion batch size, so that's four.
But whatever the length of the sequence and two, because this is the number of output units, he after
the transpose, we have to much size and sequence length.
And so after and stacking logits, what we have is we have two times.
Tensas of size, buche size, cyc length well, Tensas of shape but size segment, so now all we want
to do is to return a tip or to return two elements, the first one being inducts.
Logits.
And the first element of it, and we also want, of course, to return and tax logits, the second elements
of it.
So this represents this cause for starting the answer and this represents this call for ending the answer.
So that was it for creating birds, qualia.
So the small area that we will add right after birds in our whole model that we are about to build right
now.
So let's get into it.
And Tucson.