We already know that your site receives requests from users and robots to access content.
But how do we analyze all this traffic in detail.
In addition to tools such as Google Analytics which monitors human users there are the so-called log
file analyzer tools.
We are talking about the special files generated by the server which contain the list of requests received
with date IP address time path user agent and other useful information.
According to the W3 3C consortium format for each new request to the server a line will be written inside
the log file and day after day.
These records will accumulate to be analyzed.
The insertion is progressive and consequently the last access will be found at the bottom based on your
hosting specifications such as Apache and Jinx ISIS etc..
The path will be different as you may have noticed if your site is particularly visited you may have
log files with thousands of lines and in this case it would be very complex to do a manual analysis
to facilitate you in this operation.
There are specific software specifically designed for this purpose such as integrations of elastic search
which respond to the needs of an enterprise or get MCO fixed log analyzer for a more handy use that
you can find on get SEO fixed dot com slash log dash analyzer but what is the purpose of these tools.
Will security bug fixing correcting technical malfunctions and ASIO analysis in terms of security.
It is important to monitor all received requests so that we can find out malicious bots that visit hundreds
of pages a day.
Pages visited that do not exist accesses to private pages and elements identifying intrusion attempts
after discovering these accesses.
We can decide to block them to cover security holes or simply to prevent malicious bots from indexing
our site and using our resources.
The block can be executed via the robot start text file or with rules in the firewall for the user agent.
On the other hand the log files also tell us how Google bought access as our pages and how it proceeded
with crawling operations.
Let's see some practical examples of what we could discover through the logs number of requests in a
period of time indicates the average load for the server error codes for 0 4 and 5 0 3.
If we find that there are many errors we should do a technical action or upgrade the server dates from
the last crawl.
How often does the board visit us the most visited pages for Google board.
They are particularly appreciated content which pages or areas of the site are never visited.
We should discover why and improve these sections to get better results and waste of the crawl budget.
If the board visits dozens of useless pages duplicated or with little value it will not have enough
crawl budget to index new content or view the updates of really important sections.
After identifying this data we will be able to proceed by making optimizations to facilitate Google's
crawl perhaps thinking about the structure of internal links or making other improvements and then checking
if something has changed.
Here is an example of a recent log file belonging to one of my sites about health.
You will notice that there are hundreds of hits by famous ASIO analysis bots.
I could decide to block them to save server resources or simply not allow them to offer interesting
data to my competitors.