Prometheus setting up alerts.
Prometheus setting up alerts demo.
In this section I will show you how to install Alertmanager,
create config for the Alertmanager, configure the mail,
configure slack, alter the Prometheus config
to use the Alertmanager, setup an alert
and you will see how the notification
will come in, into slack via mail.
First we gonna log into our DigitalOcean server.
And in the beginning of this course,
you cloned this course directory.
So let's check Prometheus course, scripts.
As you can see, there are a lot of scripts here.
Right now it will be focusing on script four and five.
Let's see what it does.
As you can see here,
we are setting the Alertmanager version,
which is at this time 0.15.0.
Gonna download it, unzip it,
gonna navigate into that directory,
we gonna add a new user called alertmanager,
we gonna create unnecessary directories,
we gonna touch the configuration file so that it's there.
Then we gonna change the ownership
and set it to alertmanager.
And then we gonna copy the binaries,
also set your ownership correct on those,
we gonna create a system de-script
so you can easily stop and start the alertmanager,
as you can see here,
so it's enabled also when you reboot your machine.
And we gonna also make sure that Prometheus is started.
Then we have to add it to following config.
And then we add this to the Prometheus configuration.
So this is needed to establish the communication
between Prometheus and Prometheus Alertmanager.
All right, the second script,
script five is actually to install a mail server,
this is needed if you want to receive emails from this host.
So I tuned the configuration so you are only able
to send emails from this machine from localhost.
This is needed else your box will be exploited to send mails
or become a spam server.
So make sure you use this configuration so you're safe.
We also gonna test this using this echo commands.
And then we gonna check our spam folder
to make sure that we received that email.
All right, let's start with the first script.
All right, now let's edit this configuration file.
So we're gonna copy this.
Let me copy the path first,
if I'm using Vi but you can also use Nano or whatever.
I'm gonna copy it, insert it.
So the SMTP server is gonna run on port 25 on localhost.
This is the email address the SOD alerts are coming from.
It's best to use your own domain if you have a SMTP server
with a username and password, you can enter it here.
This is a template, it's gonna send an email
to use a correct email address, you're at in4it.io.
And I'm gonna alter this domain as well.
So in4it.io, now we configure to email.
Now let's create a WebHooking slack.
So, you're going to your domain
in my case in4itio.slack.com/apps.
Gonna open it.
Now we gonna go to Manage, Custom Integrations,
Incoming WebHooks.
I already got two defined.
I'm gonna add a configuration, choose channel,
I'm gonna use the Prometheus course
and Add Incoming WebHook Integration.
So it's now been added, you can find the WebHook URL here,
gonna copy this one.
Now let's head back.
I'm gonna replace this with the actual value.
Before we gonna save this file,
you should add something to the SMTP config.
So in this case, we need to smtp_require_tls,
to put to false.
Also, when you're trying to send an email via Alertmanager
is gonna use SSL by default,
which we've not installed on our server.
So don't forget to put this here.
Are gonna write quit it.
So this is done.
Now I'm gonna do service alertmanager restart.
I'm gonna check the status.
As you can see, it's running
so the configuration is correct.
If it's not working for some reason,
and you find it difficult to debug using system D,
you can always use this command to run it
on your shell to see what is going wrong.
All right, now this is done.
Let's add this to the Prometheus configuration.
So this means that is gonna look for an Alertmanager
on localhost on this port,
if your Alertmanager is remote on a different server,
you need to adjust the localhost
with the appropriate IP address.
So this is the default configuration.
As you can see, it's already here.
You just gonna remove this comment
and alter the words alertmanager by localhost.
Right, write quit.
And we gonna restart Prometheus service,
prometheus restart, check the status.
And that's also running, great.
Now let's check if we are able to reach it.
So let me open a new tab here.
Okay, as you can see,
I serve to the IP address using port 9093.
And this shows you the actual Alertmanager
which we just installed.
Now let's see if Prometheus is still running correctly.
And yes it is.
So now if we've start this up,
let's head back to the machine and install the mail server.
So in the scripts, you got script five.
And let's run it.
So it's now installing the mail server.
There you go.
Now let's see if this is working.
Copy this, quit it, and then your add in4it.io.
Let's search the syslog, I already feel my phone vibrating.
So I know the email is there but it's just to show you.
So connected to, network is unreachable.
That's not a problem, it's a issue at Google
and then they actually replied, status sent.
So the email is actually sent.
So that's a good thing, so it's working.
Let me put it up here.
As you can see here,
you've just successfully installed the mail server.
The date is correct, and the time is correct as well.
So yay, we received an email.
So that's working already.
Now, this is all configured.
Now we're gonna set up an alert.
So let's head back to our machine,
and then we close the tail,
and now we gonna go to vi /etc/prometheus.yml
and then we gonna see here, you got the rule files.
Gonna to delete those ones and replace it with mine.
So here you have the rule files.
So, this is going to be called etc/prometheus/alert.rules.
Gonna copy it for later re-usage.
Make sure the formatting is correct.
I'm gonna create the alerts file,
are almost there for our first alert.
So we're gonna paste our alerts here.
So it's gonna be called cpuUsge is an example,
it's in a group which named example
and the rules are CPU usage,
so it will report when the CPU is higher and 95%
for more than one minute, the severity is critical.
And it has a summary of machine under heavy load.
Let me save this.
I'm gonna restart Prometheus.
Let's see if it's running.
Prometheus is running.
So let's head over to Prometheus
to see if it sees the alert.
So I'm gonna go to alerts.
Oh, you can see our alert that we add is here,
and it's not active.
So now let's try to generate some loads
and make sure that this alert gets fired.
So let's head back to our machine.
Gonna type yes to dev/null and put it in the backgrounds.
Right, let's see how much loads you're generating.
So as you can see running at 99%.
So we're gonna leave it here till we get an alert.
So we set the threshold to one minute normally,
let me check, so as you can see,
needs to run for one minute over this percentage.
Let me take this query so we can follow it up.
Let's head back to the Prometheus server, Graph
gonna execute this query.
Gonna graph it oop, as you can see,
it's already seeing that the load is quite high.
So it's more than 100%,
so it should be firing in less than a minute.
As you can see here,
it's running for more than one minutes above 100%.
Let's check the alert.
One alert is active, let's check it out.
Yeah, that's the one we defined.
Now let's head over to the Alertmanager to see the alert.
Here it is, can see some info, machine under heavy load.
Source, that's the actual domain where it's coming from.
But since I didn't configure DNS, it's not reachable.
Now let's check our email and slack.
Let's first start with slack.
Here we go, so the alert came in.
Let's head back here and do killall yes,
so this is gonna kill all the processes, yes is running,
so the CPU should get lower now to 0%.
Let's get out of here.
Let me check the email to see if we actually received it.
As you can see here we got an alert,
we can view it in Alertmanager,
the alert where it's coming from, in this case localhost,
the severity and the summary, the annotation we added.
Now let's wait a moment for the alert to be resolved.
Kinda go here, check the alerts.
So none are active.
Let's like, check slack so it's fired.
And send back to our Alertmanager
to see if it's still active.
And no alerts are found.
So that's a good thing.
As you can see here, the alert got resolved
right after we killed the yes command.
So I hope this gave you a better overview
of our setting up alerts,
will continue to add some alerts in the future.
So for now, this gives you a firm basis
to experiment on your own.
Thanks for watching.