Hi and welcome back to this course.
So as I said before, the data, I propose the scene will be quite fast because we won't do everything
by hand this time.
We only know how to preprocessed data thanks to the previous applications.
And if we had to deal with the Jason and all those very particular things that are related to squat,
it will take a long time, but we would not learn a lot.
So we will actually use the functions that Google wrote and that are provided by Google in order to
quickly go through the data processing.
So first thing we have to do as before is to mount our drive in folders so we monts contents, flash
drive, enter.
And so of course we follow.
The link we pick our Google accounts, although we passed the code.
Well, we could be the code first, we passed it now and now we have access to our Google, right?
OK, so first thing we have to do is to call the function generated every code from JSON file.
So that's an intermediate file that will be created from the JSON file containing the data sets and
the vocab, the text that are talked about in the introduction that contains all the vocabularies from
the tokenized.
So in order to do that, we will get the output of this as input meta data.
So that's just information about the data that we are having and the file that we are creating.
So this input data that will be created by generates T.F. record from JSON file and it needs three things
actually.
The first one is the path to our train, the and file and we will Cleverley have access to it that way.
So my drive projects then.
But I know that it's in data squad and what we want is this one train V1 Jaeson, so copy path.
And we posted the second thing that we need is the path to the vocab text file.
So we'll do the same thing.
We will just copy.
But he.
Posted that.
And finally, we need a path for the free file that we will create, so that will actually be at the
same place.
But we will name it strain, the one point one that G.F. Records.
And that's it for creating our Jaeson file and that, of course, there is a sloshy, so that's it.
I want to bring it right now because it takes some time and I already have the files in my personal
folder.
But the next thing to do is to write in a file the input metadata that we have right here.
So with G.F. that I owe that file that she filed, this is the way we have access to a file directly
if we want to write something in it, for instance, and this file, we will say that it's.
Contents, well, the path to the same for the once again, and we will call the file train meta data,
we want to write this file and we open it as writer.
And from this open file, we will call the right method.
And what we will write is in adjacent formats, where is content in input meta data?
And just in order to know how to write it, we will use an index or four and so that there is no issue
later with the code, we just have to add a backslash and at the end.
So that's it for writing the input metadata in our personnel for the last thing we have to do is to
really create our data sets so we will use this difficult file that we created in order to create the
datasets and we first need to indicate the batch size that we want to use.
This time we have to use a really small about size.
Apparently, Google said that's using a bigger batch.
Size would be complicated because here our inputs are much bigger than before, because before we have
small sentences.
And actually that was Tweed's so really short sentences for our classification.
But here it's much bigger because the first sentence in our inputs is a whole text.
Could be a Wikipedia paragraph, for instance, that contains information.
And then, well, the second sentence is shorter because it's just a question.
But the two sentences, the two sequences packed together, it's really huge.
So if we use a big batch, that could be a problem, especially if we don't have huge, huge use at
our disposal.
So we will just stick with batch size of four and we will create our train data sets by calling creates
squad data sets.
And we need, of course, to give it the path to the traffic file that we created.
So train.
The one point, one that G.F. records, so that's the file that contains the preprocessed information
about the squad that sets, then what it needs is the maximum length of the sequence and we can get
it with input meta data.
Max, secc, lenth, so.
That's it, actually, for your information in this case, this is three page four, so that means that
we don't have an input sequence of more than three eight four words, then the boxes that we just decided.
And finally, we have to specify whether this is for a training set or a testing set because this would
be different.
As for the testing, sets are actually different answers that are possible.
While for the training sets, of course, there is only one label.
So one answer a question.
So is training here equals true?
Of course, because this is a training set that we want to input.
So that was it for the detector processing, thanks to the functions provided by Google to process the
squad data.
It was really short to write this code and it's actually not that long to process.
So now we can go to the most interesting part, which is to build the model that we use, but as it's
called and that we will find you right after that.
So hope you are ready to dive into this and use an.