One of the concepts that is rarely discussed in the ASIO world is the term crawled budget.
That is a value that represents the number of pages that a search engine scans on a site generally with
reference to 24 hours since Google does not like to waste resources of its servers.
It will prefer to spend its time and performance to scan content that it believes to be of quality.
Evidently Google bot will assign a larger crawl budget to a site that it likes while it will reduce
it.
In the case of spam or problematic Web sites.
So how do you increase the crawl budget trust.
If we get new quality links to the site we will increase the general trust towards our content and we
will be more taken into consideration update of content if a site is constantly updated with new articles
or with modifications to existing ones.
Frequent scans will be necessary to keep up with the updates upload speed if the server responds quickly
to the bot in a fixed time it can scan more pages then in a slower site in order to view the pages crawled
by Google bot.
You can see the charts directly in your Google search console account.
Discovering the number of pages visited daily.
The bandwidth used and the loading speed.
Looking at these charts you will be able to tell if it is the case to upgrade the server resources or
take another action to improve the accessibility to your site.
I personally believe that crawl budget is a concept to be known but with some clarifications referring
to e-commerce or larger sites with thousands of pages real indexing problems can be treated especially
with a frequent update of products prices and variance.
In these cases the crawl budget must be taken into consideration to ensure that all sections of the
site are indexed and checked periodically.
On the other hand whoever has a blog with a hundred pages will probably never suffer from these problems
but can just monitor the situation to make sure that all the pages are equally scanned and everything
is correct so far.
We have talked about aggregated data but how do we know what specific pages or categories of the site
are scanned for this purpose.
We are helped by log files which we will see in the next chapter.