Hi and welcome back to this both course.
So now what we have, all our data is ready to be used and our model that has been created, let's actually
create the A.I. and create all the tools we will need in order to train it.
So we'll start with the hyper parameters.
So I had to look at the size of the training data sets, so I just will write it here for the sake of
it.
So let's say that train data size is.
So it's eighty eight thousand six hundred forty one, that's the size of our training data set.
So that's not the number of patches, that's actually the number of training data that we have.
So we have four times less patches because the batch size is four.
And actually, as it's quite heavily trained for the sake of making it easier to use, we will not use
the whole data set for training because we just want to see if it performs well, if it seems to be
fully working.
So we will define no number.
But she's trained to be the number of matches that we want to keep.
So let's use two thousand batches.
So that will be approximately a tenth of the batches that we have, because, as I said, we have four
times as much as the number of training data.
So that's approximately 20 thousand.
And we will only keep the first two thousand, but we will still keep these entries for that.
The idea was just to see if it works well and then we define the batch size.
So that was for just to remember it then the number of epochs.
So I used three a box.
That's actually I think that two are enough.
I don't think there is a lot of difference between going through a two or three, but that's the one
I went for.
And then now we have to create the parameters for our optimizer, because as I said earlier, we will
use an optimizer made by Google that is just a slightly modified Adam, and it will need two elements
first and initial learning rates.
So let's define it right now.
And for that purpose and for actually almost any time, we will fine tune, but we will use five, eight
minus five.
Actually, this is maybe something that you can try to improve when you are working on a different task
and you are trying to train.
But maybe this is not optimized for any task, but for squad, this is the one that Google used.
So that's quite optimized for this one.
And we will also need a number of steps for the warmth of our training of our optimizer.
And we will define it as being one up steps equals int those white space that want to remove.
And this will be no, but she's train times zero point one.
So actually for approximately a tenth of all the elements that we have in our training data, we want
to be in a warmer configuration for our optimizer.
So that's it.
For all the hype of amateurs and all the variables that we will need.
And the first thing we want to do, as I said, is to create a lighter that sets so that we don't train
with all the data so that it's way faster.
So from train that says we will take no Butches train.
OK, that's it then.
We are ready to create our AI by calling, but quad equals that squad that sits.
That will take just a few seconds because we have a lot of weights to create and a lot of elements to
call.
But once we have it, we can create our optimizer as a set so we don't use a standard.
Adam, we use something that is provided by Google in the official models package that we imported at
the beginning.
So it will be up to my station that we imported before.
That creates optimizer and it will need the initial learning rate.
So it was in it at all then the number of training steps.
So that's the number of steps that we will have per epoch.
So in our case, that's exactly the number of batches that we have.
And the watches train and finally the number of walking steps.
So that's what we declared just above the steps equals up steps.
That's it.
So now we have created our optimizer and the last thing we need before combining our module is a lost
function.
So what do we want to create as a lost function?
We saw that our model returns two vectors per inputs, for example, and each vector is actually at
least, of course, the first one being the scores for being the start position and the second one being
the score for being the end position.
And that really looks like a classification problem.
That is actually almost as if.
For each example, we had a classification problem, so our whole inputs should, for instance, while
trying to find the start position, get the right talking.
So among the, let's say, three hundred words of our initial paragraph, we need to find the right
position.
We need to find the right number between one and three hundred.
So that's almost if we were classifying something.
What I mean by that is that we get a score for each of the three hundred positions to be the start talking.
So from the scores we get actually probabilities.
So we get probability for each of the three and the tokens to be the starting token.
So a good way to apply a loss to that is to apply exactly the same thing that we do for classification.
We know the right position that we need to have and we have the probability for its position as output
of our model.
So we will apply for both our outputs.
So for the outputs corresponding to start positions and the output corresponding to the end positions,
we will apply a sparse category called cross entropy, which is the last that we usually use for classification.
So if, let's say the start position should be in an example, the position fifty, if we have a high
score at the position fifty and very low scores for all the other positions in our output corresponding
to the start positions, then we will have a low loss.
So the idea is for the two different outputs for the start positions and the positions to compute a
classification loss separately.
So as a categorical course and B separately.
And then we take just the mean of both losses and we get our total loss.
So let's write this right now by death squads loss function, which takes labels and the model outputs.
And first, we want to get the real starts positions, so we define starts positions as being labels.
And we take Staats positions from it, then we get the real end positions by calling labels and and
positions.
And we get, of course, the different outputs by start logits and and logits being the model outputs,
so we separate the two outputs of our model and now we can come to the first class, the one corresponding
to the starts logits.
So starts loss equals T.F. That's Kiraz.
That's back.
And that spa's category called Cross Entropy.
And we applied to start positions.
So that's the real position for the start of the answer.
The stats logits, that's our predictions, so that's the score for each position to be the start of
the answer, according to our model and from Logits equals two.
So that means that our staff logits are not probability yet.
We did not apply, for instance, a softmax or something like that.
They are just Rothko's now.
Of course, we do exactly the same thing for the end loss, so.
It's and laws, we apply the same function to the and.
Positions and the and logits, so that's it, we have our two losses and all we have to do now is to
take the mean of its total loss will be G.F. that reduce mean because we want to get the loss for the
whole batch here.
We have separated loss for each element of the budget.
So by doing we just mean we take the mean of the losses for the whole batch.
So we do that for start loss.
And we do the same thing, we just mean for the end loss and after sending them, we divided by two.
So that gives us the mean of the start loss and the end loss for the whole bunch.
So all we have to do is to return.
They'll tell us and we get our squad lost function as simply as that.
Finally, when we will build our custom training loop, we will want to keep track of the lost during
the whole process.
So let's define train loss.
T.F. that give us that metrics that mean name equals train loss.
So we will keep track of the loss doing the whole training phase.
So that's it for this, quote, lost function.
Finally, all we have to do is to compile words so that squads that compile and we give it to the optimizer.
And the lost function, so quite loss function, that's it.
Let's compile it.
So that's OK.
Now our model is really ready to be trained and I do not rewrite this because this is the same thing
as before.
We can create a checkpoint to store the evolution of our model.
So here we are.
Let's not go into the last step of the training, which is to really write our own training loop.
And after this step, we will be ready to create the outputs to finally evaluate our squad models.
So let's do that right now in Tucson.