Hi and welcome back to this birth course, don't leave, don't leave.
I know that I promised to start coding with birth right now, and you might be wondering why I see here.
Quixey in an explanation.
Let me tell you, my idea for this application part in this course is to show you all the different
ways to use birth.
And, of course, you can use birth as it was intended to like using fine tuning.
But there are actually other ways to use it.
And they actually involve using tools from birth and plugging them to your own energy models.
So what we will do is that we will try to build a really simple and model in the standard way.
So what we used to do before birth and I will show you how to improve it with but without going with
fine tuning.
So without getting rid of your model, the first step will be just to improve the data processing using
bloodstock ENISA because originally in this application we used a homemade took Annisa that we created
on the go.
And just with that, we will already have an improvement.
And next it will get more interesting as we will use but as an embedding layer, because originally
in this model and most of MLP models, we use an embedding layer that we also train on the go with our
data.
But he will use Berts as an emerging layer.
We won't find units.
It will stay as it is, but it will give us a vector representation.
So embedding representations of vectors that we can use for our model and you'll see that once again
it will improve the results.
So that's why I want to quickly introduce a standard MLP model.
I want to go into the details of CNN, of course, because it would be quite long and that's not the
topic of discourse.
But if you are used to A.I., it will be enough.
And even if you don't fully understand the CNN for NLB, that's not a big deal because you can use what?
We'll explain later with any personal and model and you'll see that it's not that hard.
But if you want to go into more details and to know more about CNN applied to an LP, you should consider
taking the modern and LP techniques course.
Anyway, let's go into a quick introduction to a CNN model for LP.
So first, quickly, what is CNN, what is made of the most important part of CNN as it stands for Convolutional
Neural Network is the convolutional bots.
So what is the convolution?
The idea is to have a matrix.
So in our case, it's a sentence, each word of a sentence being a vector of the same size.
So if we just put everything together, we actually get a matrix.
Originally, CNN's were made for images because images are natural mattresses, actually.
But here we focus on L.P, so on text.
And this is how we get our matrix, which is actually a list of vectors, each vector being an embedded
representation of our words.
And the idea is to have a small filter, a convolutional filter that we can also call a feature detector,
that we go through all our matrix.
So this feature detector, he is of size three by three.
So we will apply to small patches of size three by three in our list of vectors in our original or input
matrix.
And the way we do the computation is that we just multiply each elements by.
So the top left number of our feature detector will be multiplied by the top left elements of our patch
right here.
And we do the same thing for each element, one by one of pair by pair.
And we set everything.
And this is what we will put in the top left part of our output matrix.
So if we take this example, we get to zero right here because zero is multiplied by zero.
Again, zero by zero, this one in the feature detector is multiplied by zero in our patch right here.
So we still get a zero.
Sending the zero from the patch in the matrix is multiplied by one in the feature detector.
And then we get still zero zero zeros.
Those ones in the feature detectors will be multiplied by those zeros.
So actually, nothing is activated in this patch.
So that's what we get to zero here.
Next step is just to shift the batch by one toward the rights and to apply exactly the same feature
detector to this new patch.
So we do the same process, the same computation.
We get zero by zero.
Zero by zero, one by zero zero.
Here we have an activation.
This one will be multiplied by a one.
So it's the first time that we that something happens, that we get one, and after that we still get
zero because we only have zeros here in this patch.
So if we send everything, we finally get one.
So as our patch is the top left, one shifted by one, we put the one just right in our feature map.
And the idea is just to do the same, to shift the patch by one once again.
And this will give us the results of the application of our feature detector in this new patch right
here.
So we can call this actually the activation of our detector and so on until the end.
So we can see here it's the application right before the end.
So this time, if we do a bit of computation, we see that we have this one that fits with this one
and same thing here.
This one fits with this one.
And otherwise we have zero zero zero zero by one.
So we have two small activations in this area.
That means that we put a two here and we finish with this one actually with only one activation, which
is the middle left part right here.
So what we did is that we had a feature detector.
So that's a small matrix that is supposed to be able to detect a specified element, a specified feature
in our matrix, and we make it go through our whole matrix in order to locally detect those features.
So each time we have a high number, that means that this feature that we were trying to detect has
been activated in this area.
And of course, what we train in our network, in our CNN is the spot those coefficients.
So this feature detector is what we train.
So our CNN, our network learns to detect useful and relevant features by tuning those coefficients.
So our CNN will have many different feature detectors like that, each one trying to focus on an efficient,
relevant, valuable feature that would add information at the end of the process.
And so if we summarize this process, we have our input metrics here, our lists of vectors, and for
each of our feature detectors that we train that we that our network learns we get a feature map.
So each of those output metrics actually corresponds to a feature detector that we have learned.
So that's how we end up with a list of smaller vectors that represents the activation or lack of activation
of a specific local feature in our metrics.
So that is that's actually the whole idea behind CNN and more precisely behind the convolutional mechanism.
So now let's have a look at the architecture we will implement.
So how this CNN, how this convolutional idea is used for our energy task.
So here is the architecture we will implement if we have a look at the different steps we can see here.
We have our input metrics are, as I said before.
So it's just a list of words, each word being represented by a vector and how they are represented,
the different feature detectors.
So in this representation, we have six different feature detectors.
They have different size and they are applied to this matrix.
Here we have the outputs of those feature detectors and the idea is that we will apply the marks to
each of these aspects.
So that means that all we want to do is to know whether the feature has been detected, because here
we have a list of all the local detections of the feature that is detected by this feature detector.
And so this is how much the feature has been activated at the very top.
And this shows how much it has been activated at the very bottom of our sentence or at the end of our
sentence.
And if we take the maximum of all these numbers, that just gives us an indication of whether the feature
is present in the sentence.
And actually, that's just enough.
It's enough just to know whether the feature appears somewhere in the sentence or not.
Then we basically just concatenate everything and we apply a dense layer to get our classification tasks.
Pretty simple.
So just some things to keep in mind in this model.
The first thing is that you can see that those filters have exactly the same with as our matrix.
That means that the width of our feature detector is exactly the imbedding size of our model.
That is the reason why the outputs of the convolutional face is a vector, another matrix.
So if we go back to our example before that.
Means that instead of having a three by three feature detector, we would have a seven by three feature
detector.
So the shift will only be made vertically.
And so that is the reason why instead of having a five by five output matrix, we will have a five by
one output matrix because we have this huge detector right here that is only shifting vertically.
So that's something important for an right here, at least for this architecture.
Our filter is only shifted vertically and has the same with us.
Our input metrics.
Intuitively, we can say that the reason for that is that splitting this horizontal dimension doesn't
really make sense.
Trying to apply if it's a detector among, I don't know, here, for instance, four consecutive words
or hear three consecutive words.
It makes sense.
You want to know if a specific feature appears in a small part of the sentence.
But this dimension, the embedding dimension, it doesn't convey the same meaning at all as this dimension.
This dimension just represents the continuity of the sentence, the fact that we get one word after
the other.
But this one is completely different.
It's just the vector representation of a word.
And it doesn't really make sense to separate the first half, for instance, of this representation
from the second half.
Trying to find a feature in just the first three dimensions, for instance, doesn't really make sense
because at least the order of the dimensions is actually random.
We can get actually the same results and the embedding conveys the same meaning if we just switch to
dimensions, right.
So that's an intuitive reason why the width of our feature detector is exactly the dimension of our
model, and that's why we get vectors right here instead of mattresses.
Second thing to keep in mind, as I just explained before, is that for each output right here, we
just take the absolute maximum of it.
I just explained before why we do that.
It just that it's important to keep that in mind, because usually when we apply CNN to images, that's
not how we do that.
We once again take local maximums.
But that's not important.
We forget about.
That's the important thing is just that now this step for each feature detector gets a single number,
which is the absolute maximum value of the activations of this feature in our initial sentence.
Then a detail of implementations for this architecture is that we will have three different sides of
filters.
So one type of filters that we go through, sequences of four consecutive words here, a second type
of filters that we go through, sequences of three consecutive words and hey, just a third type of
filters that will have a look at consecutive pairs of words that will allow us to capture different
scales of correlation between words.
Finally, he for representation purposes, we can see that they implemented two filters the size of
filter.
But of course in our implementation we will use much more.
I think we will use something like 50 different filters for each site.
So in total we will have one hundred and fifty feature detectors.
So that was it for the quick CNN for NLB introduction or explanation?
Of course, it's not complete.
We don't go deeper into the subjects.
That's quite shallow.
But the idea is just to get a basis to understand a simple model that we will implement and try to improve
by adding some good tool to it.
So now, for real, we will start coding.
Let's start to write our first lines of code right after introducing the task we will deal with and
getting the data for that, we will be ready to write our first line of code.
So, Susan.