The first project of web scraping with Beautifulsoup, and we'll be using customer reports also.
They also call it Consumer Reports, uh, website and see, you know, uh, try to scrape the information.
All right.
Uh, let's, uh, do the preliminary required libraries first, and then we can move on.
But before we move on, I would like to show you how the customer report, consumer reports, uh, website
looks like.
Uh, it looks like that normally, just like that.
And we can, you know, basically scrape anything.
So for this thing, what we'll be doing is we'll be basically scraping, like, the best deals, uh,
for under five, $50.
So we'll be separating these 2 in 2 parts.
So one is best deal under $50.
This is the first thing that we'll be doing.
And second thing is a bit complex.
We'll be going through all the products, uh, from here, uh, all products.
And we will ask the user to put, uh, the products they want, and then we will just go through that
and, you know, uh, list out basically all the things that are listed here.
So, uh, we can think how we can do it for.
But for the first part, let's try to just do it.
Uh, the best deals under $50.
That's just what came to my mind.
But we can reframe what we thought.
So let's try to go here inside.
Let's see what they have.
So here we can get the list.
This one.
And, uh, what else can we do?
Here are the, you know, names of the, uh, things and kitchen tools.
Okay, okay.
So let's do it.
Uh, then we can do it.
All the best deals.
Not just best deal under 50, but all the best deals.
Okay, let's do a necessary importation.
So we import HTML5 lib, we import requests.
And finally we from BS4 we import beautiful soup.
All right let's try to get the URL.
You can just go here.
So we say rest equals request dot get.
So if you remember we first want to get the detail.
And once we have this then we want to create a soup of beautiful soup with the rest dot content as the,
you know content and HTML5 lib as the parsing library.
Let's try to print the soup dot prettify once to just check everything works.
Everything is working all good.
So let's go into desktop, go into selenium automation course.
Let's try the python3 main.py.
All right.
So basically we can get the whole, um, Consumer Reports thing.
So what we can do now is we can find, uh, all the deals inside this here.
So see all the deals.
We can actually go to this website directly, this page directly, and we can find everything.
Maybe it would be a nice idea to just follow the links actually.
Okay, so let's inspect and see what the best deal under $50.
Uh, you know, looks like, um, so it's, uh, class layout, shopping image link.
Best deal.
It has a class image shopping link.
We have to think a little bit, uh, what that is, uh, let's try to come to this main part, which
is just which has the ID of swap deals.
So let's just, uh, get that diff first.
So div with an ID of swap deals.
So we come here, we say uh, div swap deals equals I can say swap dot find and I want to find a diff.
And our attributes attr is a ID of shop deals.
Is that correct.
Let's check.
Shop.
Shop that steals.
Yeah.
Let's try to print the div shop deals and let's see if that works.
Yeah.
So we can see that.
And, uh, looks pretty good to me.
All right, so inside that what we want is basically all the A refs and the links.
That's what we want.
Basically the first, uh, you know, top five deals we can get.
So what we need to do is basically get all the A refs inside that.
So what we can do is we can say div shop deals dot find all.
All the a tags.
So let's just go through for I in find all.
Then we want to print the I but then not everything but just h refs of I.
So let's try.
Yeah.
So we can see all the links right now.
Obviously the links are without the Consumer Reports, because if you see the link and if you click
on that then it opens Consumer Reports base URL then that.
So what we can create is like a base URL equals that.
We can actually pass here the base URL.
And then here we can say print the base URL plus that.
And then if we see this.
Now you can see it's with that.
Clear this again.
Now we can see.
We can see the whole, uh, link.
So if I go to that first link, then it will be about the best deals under $50.
So exactly how we want it.
All right.
Uh, looks pretty cool and looks like how we wanted it.
Okay, now, what we want to do is we want to open each of them.
So we have our list of links that we want.
So let's save it into a top deals list.
Let's say top deals equals a list.
And we just want to append into this list.
So top deals dot append this into.
All right.
So once we have the best deals list then what we can do is basically we can go on each of them and uh
get maybe the, you know, uh, we can, you know.
Maybe what we can do is we can get we can ask the user, we can get these things here.
What happens if we go to more deal.
Yeah, I think it would be an interesting idea.
I'm just thinking now to.
Ask the user if they want to go into countertop appliance or, you know, kitchen tools.
So we list out here, we ask the user what they want, and then we basically, you know, show the products
of that particular category category.
And to do this, what we can do is we can firstly try to find and firstly let's try to find if that
is uh.
In all of them.
So what I mean is, if that is relevant in not just $50, but other things also like best deals on home
and kitchen products.
Yeah.
So we can see that, uh, kind of, uh, pattern in all of them.
So I think we can do that.
Right?
Yeah.
And how about for TV?
Yeah.
For TV.
We don't have it so we can take out the TV.
I think we don't need to go through the TVs.
Uh.
So.
Let's see.
Best TV deals.
We don't need it.
So what we can say is we can simply say.
If best TV deals in the eye, then just, uh, continue, because we don't want that link because it
does not have the same, you know, sections, uh, as normal.
So let's try to see how that how they look.
And let's try to print the top deals and how they look on it.
So if I just run it and yeah, so we have that.
And if I see best TV deals it is there.
Electronic computers.
It's actually like that.
Because it should not, uh, have printed five, but four.
Let's see where I can change this.
We can see also if it starts with electronic.
This one here.
So if it starts with this.
So what we can say is if I dot starts with.
Electronic computer then we skip this.
Let's try.
I is none.
Try this.
Except.
Continue.
Mhm.
Okay.
With that.
Let's try to fix this like it was before.
News.
So what we are doing is.
Squirting a bunch of different.
Electronic computers are okay.
Um, yeah, I think I know what that was.
With two c ts I rf because the link is not I, but it is actually I rf.
So if it is in I RF that's the thing.
Yeah, exactly.
So we don't have now our five, but now only four, I think.
Is that correct or not?
Let's see.
Self-promotion.
Hundred dollar.
Yeah.
So we don't have the TV thing here now.
All right.
Awesome.
So we did that.
Now what we can do is we can come here, right here, and then we can see we can get the section links
here.
So what are the section links.
Let's see inspect get the section links basically.
So it is inside the intro section.
Okay.
It is inside the section intro, so let's try to get that first.
So what we do is for each link in the top deals.
Then we go through all these links just like how we did in the response Restoule is gonna be link.
And then same thing.
Sub two again would be sub two would be rez two content.
And then we say intro.
Intro section equals we get the soup two and then find.
But now not the diff.
But what do we want to find.
We want to find the section.
So section with an ID of intro.
Simple.
So we just go here ID of intro.
So we have our intro section.
Let's try to print this intro section.
And I'll just break it so that it does not.
Now go through the loop.
Just gets the first uh, one link first.
Yeah.
So we have our intro section.
We just need the ref of it.
Okay.
Now what do we want is all the lists of these things.
And this is, this seems to be inside.
These are all the A tags into this thing.
So let's try to get all the A tags.
So same thing again.
Find all a.
So what we want is from the intersection find the all of the A's and print.
In those.
So let's try.
So we have our A's.
Now what we want is the text.
So what I can say is I can say for I in find all print the I dot text.
Okay.
Now, uh, I think the last one we don't need because, uh, it's not, uh, I think, uh, related.
So what we can do is we can say find, uh, a.
I think it should be noted that more deals.
Within that section.
Are there any other attacks?
Maybe from here.
It got it.
So?
So action.
It should not be.
It should be within this p tag.
But we don't know what that p tag is.
So what we can simply say is then we can simply say just.
We don't want that.
So we can just take it out later.
Okay, now we can ask the user which one he wants to see.
Countertop appliance.
The first and last I think we don't need.
And so on.
So what we will do is we'll say section text, text links equals not links but section intro list.
And we can append into this intro list.
And we have another list called.
Well, let's create a dictionary and just have it the links as their values so we can say intro dict
and intro dict I text would be equal to I ref.
So by doing this what we do is we get.
Finally if I print the I.
Not I sorry uh print the intro dict and we can see it is it will be a.
A dictionary containing, you know, all the.
Options and their links, so that's good.
Now what we can do is we can simply say for.
I in interdict.
Interdict?
Um.
Maybe we can remove the first, uh, you know, to let it be.
We can just ask the user so I can see.
Print I.
We can have a counter variable also.
C equals zero.
So that uh it has always you know c.
And, uh, we can also say c plus equals one.
And we can ask the user choice equals input.
Enter which deal you want to see.
So let's try this.
Yeah.
So it says okay, into which deal you want to see.
And what we want is if it says three, then it should take the link and open the kitchen tools.
So let's try that.
So.
Now what we can do is we can say.
If.
The.
Maybe it would be, actually.
Good to just put the name, actually.
No, no number, but just asking the user for name into which deal.
Case sensitive.
So, for example, what we can do is we can say link to open is going to be intro dict.
And then the choice that we have.
So let's print that link to open.
So we start.
So it says okay kitchen tools.
Then it says that right.
So our base URL we need.
So our base URL again is that link which we had.
So what we can do is again uh rest two to point one equals.
Requests.get.
So it all.
It is always that we always want to use requests and get and requests and get.
So we just because we cannot automate the things right, we have to always request and get the HTML
and then work with that.
And another would be soup.
Soup.
Uh 2.1 let's say and rest two underscore.
One is the content.
And once we have that, then what we can say is let's say we have it in the counter top appliance.
Then we can say that okay, we would need then Mr..
The this name and then this name, then maybe the, you know, let's see.
So let's see what we can get.
So let's just uh, for now, let's just say the name of the, uh, thing.
So what we can do is we can get the.
ID.
Uh.
So what can we get then?
I'm just thinking.
Intro is not.
This intro is up div is not this also.
Mr. coffee I think countertop appliance.
So it has ID of uh countertop appliance.
So always with uh low cases and uh hyphen.
So I can come here and say ID is going to be the choice.
Choice.
But then all in lower.
And what we want is I can show you also.
So for example if there is a which is someone says kitchen appliances or kitchen tools, then what how
we want it is kitchen dash tools because that's how the IDs are.
So what we can say is I can say a dot lower first, and once we have a dot lower, I can split this.
And once we have that splitted into two words, I can join this with a, you know, hyphen.
So I can say hyphen dot join, join with that thing.
Then I can get that.
So I will use that particular thing here.
So I'll just say instead of a I will just say choice.
Then we have our ID that we want to search for, and the id is of a div.
Right.
So div.
Yeah.
So let's try with sub 21. find.
It's a div and its attributes are id is that ID.
All right.
So now we have our soup.
That is awesome.
Of uh second thing also.
So we can call this a div.
Div of product.
Product.
Div products.
So now let's try to go ahead and print the div products.
And look if everything looks good.
We come here quit this thing and try this again.
Okay.
Uh, now let's try for kitchen tools.
Kitchen tools.
Um, link, we want to open the link.
Link to open is that.
But not with just.
Oh okay okay.
It should be link plus link to open because it is just slash uh after the hashtag.
So that's why.
Let's say we want, uh, kitchen tools.
Yeah.
So we have our kitchen tools here, and, uh.
So that's how we can get it.
Um, now, I think, uh, the next things, uh, we could do would be, you know, go through the next
step after this, you know, and stuff like that.
Uh, so I would leave that here because this would be too long.
And I gave you a tentative overview of how you can do it.
Now, uh, your part would be to use this and just, uh, you know, display the cases so you can put
this into the projects also.
And people can see and get inspired.
So the next project would be on creating a Craigslist, uh, scraping site.
So we'll be working with Craigslist, Craigslist and, you know, trying to scrape out some information.
So, uh, today we tried to scrape out some information from Consumer Reports and we were able to get
the links, click use that links to get other sections and stuff like that.
So you use the same concept on other things also.
All right.
So I'll see you in the next project on Craigslist website.