Hi and welcome back to this birth course, so let's get into the pre processing phase right now by starting
by cleaning outwits.
So to do that, we will define the function clean to it, which will take a single to it and it will
do all the cleaning.
So basically what we want to do is to remove the mentions.
So the at sign followed by the name, then we want to remove the U.
All else links.
We also want to finally only keep the letters and the standard punctuation marks and we will then get
rid of all the additional white spaces that we did with this process.
So first thing to do, as I said before, is to decode our tweets that are included in the XML format.
So we need to do beautiful soup.
We apply to to it that is in L, X and L encoding.
So basically that means that the tweets right now that we get from our data from the are not usable
as standard strings because there are many features in the way it is encoding in the way it is stored
that are related to this particular encoding format, which is.
And so we need beautiful soup to get us to on that string out of it.
And once we do that, we need to call get text in order to really get the string that we wanted in the
beginning.
So.
Now, let's start to remove the mention so the outside and in order to do that, we will use that said.
So what does it do?
Basically, it do pretty much the same thing as replace polystyrene.
So that means that we can replace a substring with another.
But this allows us to use regex expressions.
Basically, regex is a sequence of characters that define a search pattern.
So that allows us to search for many different and flexible things.
For instance, here we will be able to ask to search for any at sign, followed by any number of letters
or numbers.
And actually, if you want to play with the big exhibits, you can just go to this website and you can
play with regex with how it works.
Here is the explanation of all the tools that you can use here.
There is a text and you can he make a request and it will print what you get from these requests.
He, for instance, you can see that they asked to have any capital letter followed by backslash W,
which is a word a digit or the word space, and plus means that this thing can be repeated as many times
as we want.
So basically, this whole thing means we are looking for a word that starts with a capital letter.
So that's really useful.
That's quite fun, actually, to play with it.
But for now, what we are looking for is to search for all the at sign, followed by, well, letters
or numbers.
And we want to replace it by a wide space.
So we will write our regex, starting with an R before the string, just to indicate that we are writing
a regex.
And we want, as I said, an outside, followed by any capital letter, any letter or any number.
So that means that we are expecting any of those elements and plus means that they can be repeated as
many times as needed.
So with that, we get the whole name or the whole mention right after the assign plus the sign, of
course, and then we indicate what we want to replace it with.
So just a single whitespace.
And finally, we, of course, want to apply it to the tweets.
Now we just repeat the same thing with the you are our links, so we call once again, Redock said.
We call the regex expression and we want to get rid of HTP.
S.
We add a question mark after the S, that means that the S can or cannot be here.
So we are searching for the TTP and the TTP is followed by Cullum's First Vash and then once again.
Letters like that.
And numbers.
And also, we can have points and slashes in you all our link and all those elements can be repeated
as many times as needed, of course.
So that will be the regex expression for you are our link.
And again, we replace it with the space into our tweet.
Then finally, we do a bit more cleaning up, just realized there a typo here, sorry, a bit more cleaning
by keeping only the letters and the standard punctuation marks.
So readers said, we want to get rid of anything that is not.
So this is how we do that.
That is not.
A letter of any kind, and we will keep also the point, exclamation mark, question mark, the apostles
also and that should be it this way we only keep the standard characters and letters for our text.
So we are sure that we get a text that has a meaning and doesn't contain any weird stuff that I would
have issues to understand.
And again, what is not like that?
We replace it with a whitespace and we do it to our twits.
Then with all those replacements, like replacing sequences of characters with white spaces, we might
end up with too many white spaces in a row.
So let's just deal with it with tweets equals read that some.
This time we just get rid of spaces that are repeated several times.
And we replace it by the white space, so actually the plus means that it should be at least one time.
So we take all the sequences of white spaces so it could be just one white space or more.
And we replace it with only one white space.
And we do that to the tweet.
And that completes our clean tweet function with, of course, the return tweets.
So that's it for our function.
And of course, what we have to do now is to apply it to all the tweets.
So we will define Delaplane now, which is equal to the sequence of clean tweets, applied to tweets
with a tweet being in data that texts.
So that's it.
Let's see if it works.
Is that with of course, it's for.
And now it seems to be working, so this process takes a lot of time, like at least a few minutes,
because we have like several hundreds of thousands of tweets.
So I won't wait for you.
I will just write he the last cell of the spots, which is to process the sentiment.
So the label column.
And the way to do that is just to create data labels, which is equal to DataDot sentiments.
So we take the sentiment out of our diaphragm and values to get it as a for so we get a list of values.
And finally, label's data labels equals equals four equals one.
So when adjusted with this line, is that a change?
All the values for the sentiments from four to one?
Because actually I don't really know why, but in our data, the labels are not zero for negative and
one for positive.
They are zero and four.
So, well, whatever.
We just have to change this.
So everywhere the label is for, we make it a one so that we get the standard zero and one for our labels.
So that was it for for the data cleaning.
So this will take, as I said, just a few minutes.
But once this is done, we have all our tweets and our labels that are clean and we need to go to the
tokenization phase, which is actually the first time we will really use.
But so let's do it right now and using.