Hi and welcome back to this both course.
So now that have tokenized our sentences and our inputs, we are ready to finally go to the last step
of our data processing, which is to create the datasets that we will use as input for our model, for
the training.
And usually an important part of that is the podding step, which is to add to token token at the end
of each sentence so that all the sentences have the same length.
But instead of doing that at the scale of the whole corpus of all our sentences are decided two parts
per batch.
So of course, during the training we will use batches.
So we will group our sentences by buc of, let's say, 32, for instance.
And I only want those first two sentences to have the same length inside the batch and not necessarily
have the same sentences from another batch.
Because the only important thing is that doing one training step, which is represented by one batch,
all the inputs have the same.
But we don't actually need all the sentences from all the corpus to have the same length.
And by doing that, I will highly decrease the size of our data sets and also the time of computation
for the training.
So the best way to do that is, of course, to make batches that have approximately the same length
for each sentence so that we barely need to parse anything.
And so that's exactly what we will do now.
We will do a little bit of sorting and reorganization of our inputs so that everything is the most efficient
possible.
So let's start by creating data with LENTH.
And that's actually just a list of all our sentences, the corresponding level and the length of the
sentence.
So it's a list of census data labels, I guess, labels I and length of of a sentence.
And this would be, of course, for I sense in enumerates in the rates data inputs.
So the enumerate here, it's actually exactly the same as going through the data input.
So all our data.
But we also have AI, which is the counter actually.
So we know that we are having access to the element number two or one hundred twenty eight and so on.
So that's just a way to iterate over our data inputs while having accounts here so that we can have
access to the corresponding label.
Right.
So now we have that and we actually want to shuffle it.
So we call random that shuffle and we apply it to data.
With.
Len, and the reason why we do that is because in our initial data file, the inputs were sorted according
to the law, according to the sentiments.
So the first half of our sentences, of our tweets were related to a negative sentiments and the second
half was related to a positive sentiments.
So we wanted to shuffle everything so that we don't have.
But she's with only positive tweets and matches with only negative tweets.
So that's the first part.
We do this shuffling and then we sort everything, but this time according to the length of our sentence.
So we use data with length.
So the sort method that is able to sort of lists and we need to give it a function that is applies to
each element and returns a number that is used for the sorting.
So in our case, the key.
So that's the function that we use is just lamda.
So that so that's a tool to create a function on the go.
So that's a really small function that we don't want to really create.
We don't want to define it.
This will just be used here and this from that function for X.
So that means for each element of our huge list here.
So X will be something like that.
That would be those three elements.
What we return from those three elements is X too.
So that means that's what the function we apply for sorting is.
For each element like here, so each list of those three elements, the sentence, the label and the
length of the sentence, we get access to the third element.
The last one, which is the length of the sentence.
So with that, we say to.
Python that we want to source data with length using as the sorting number, this element for each sentence.
That's the reason why we had to build this.
So now we have our data that is sorted according to the length of the sentence.
So let's just get rid of this information of the length and just keep the sentence and the label.
So let's call it sodded or for instance, and say that it will be the tuples sent lab.
Zero and sends love for sentence level one.
And so this will be for.
Sentence level in data with Len, and we let's well, let's say that we will only do that if sentence
led to which represents the length of the sentence is a seven, because we don't want to keep two short
sentences, especially after all the cleaning we did, removing all the mentions, the links, the smileys
and everything.
With the very short sentences, we might end up with something that doesn't convey a lot of meaning.
So we just get rid of those ones and we keep the longer sentences.
So now that we have all our pairs of tweets and labels that are assaulted, according to the length
of the tweets, we are ready to create datasets and usually we use the front end of slices of intenser
methods to create datasets.
But here we can't do that because all our sentences don't have the same.
So we need to call that from generator.
That means that we can get datasets with inputs of different length, but we need to give it a generator.
So a generator is just something that can give elements one after the other.
And that's perfect because at list is a generator.
So we can use these lists to create our dataset using from generator.
So all that sets will be equal to that data that that sets.
That's from generator.
And we need to give it as a form of lambda an iterator.
So that's just a function that can give one at a time the elements that will build our data sets and
at least works for that.
So let's say that you sort it all.
And the other thing that we need to give to this generator thing is the output types.
So the types of our data sets and this will be for inputs and for actually labels integers.
So we can just say that it will be made up to that in 32 A.F. That's 32.
So that's.
All right, it's just types and the type and here we have that right now.
If we have a look at what all that is, it looks like what we can do when we have the data sets is have
access to the first elements clean next to.
And then you called all datasets.
And with that, we have access to the first elements, which is here, and we see that that's exactly
what we wanted.
We have a tuple the first thing being just an array like this.
This is our inputs.
So it has to be dawkins' that makes sense because we got rid of all the tokens that way off size less
than three.
And here we have the corresponding label, which is zero.
So now that we have that, we are ready to go to the building phase, which will be made at the same
time as the patching phase.
So in order to do that, all we have to do is first to define what would be our batch size.
So let's set 32, for instance, and we will create all batched, which will be all data set that.
But it's buche and we need to give it the size.
So batch size and we need to give it the shape for the for the podding.
So did.
Shapes, so the dimensions that are used for padding are indicated with the known wood.
So this is the first element of our tuple that we wanted to pass because that corresponds to with equal
here, that corresponds to our input so that the input that we want.
But using this shape known followed by the dimension that corresponds to the real values of the inputs
and for labels, we don't want to use it for the matching process.
So we just leave it like here.
And of course, it's a zero dimension tensor.
You can actually see it here.
So by living it like this, that's exactly what we want to do.
So the labels will be matched according to the batching process, for the inputs, of course, and for
the inputs.
This will be a padded budgeting process and and the budget will be added along to the first dimension,
keeping the values for the next dimensions.
So that's what it is for, let's call it.
And that's it.
That's already done.
So now if we do the same thing as before going next, it's a and we apply it to all but what we get
here.
Well, that's a bit long.
But this time we get batches, so batches of size 32.
And so he gets hit 30 to input sentences.
They all have the same the same length, which is three.
So so actually there has been no padding added in this batch, but that's normal.
The padding will be when, for instance, sentences go from size of three to size of four.
And so abash could be made of half sentences of size, three 1/2 sentences of size four.
So for the budget like that, we would add, as you will, at the end of the sentences of size three.
So they all have the same length.
But that's not the case here, of course, because this is the first batch.
So we have all sentences of size three and right up to eight we have the corresponding labels.
And you see that we have a mix of ones and zeros, so that's perfect, we won't have any bias.
And finally, as I said before, we don't have a testing site right here because the failed test that
we contained an additional label.
So we did not want that.
So we will create one Vilsack now.
And what we will do is get the number of batches that we have right now with no Butches equals mathaf
that seal.
So SEAL is just a tool that gets the smaller integer that is higher than the number we pass.
So he wants to do it with the length of sorting all.
So that's actually the number of inputs that we have divided by the batch size.
So this gives us the number of matches and let's say that we want to get a tenth of it in order to create
our testing set.
So no Butches tests will be equal to no matches.
We divide it by 10.
So before splitting it, of course we will shuffle all our dead sets because now we have the shortest
sentences at the beginning and the longest sentences at the end.
So if you would keep the first one thousand batches for the testing sets, then we will have only the
shortest sentences and we really don't want that.
So we will first or best that shuffle and we need to give you the Burgess's.
Actually, when our dataset is not too big, we can get the buffer side to be exactly the number of
batches.
So that's the best way to do that.
That's if our dataset is too big to store it into one viable at once.
We need to use a smaller size that's less efficient, but that's the only way to shuffle it.
But in our case, we can actually use no Butches that works perfectly well.
And so everything is shuffled now and we can create tests.
Data sets equals all that and we use the take method which allows us to take the first number batches
tests.
No, of actually and of course, the train there, the set will be our previous all matched with those
first no matches, Test matches.
So we'll use all buffs and instead of take we use this time skip and we skip no Butches tests.
So here we are now.
We have a train data sets and test data sets and we are ready to train it once we have our met already.
And that's exactly the topic of the next box.
We will finally build our model, our CNN model, and we will see right afterwards how it performs.
So all the data processing phase is done.
Now we are ready to start coding our model.
So let's go to the exciting part and using.