Prometheus Alerting.
Prometheus Alerting-introduction.
Automated alerts are essential to monitor.
They allow you to spot problems
anywhere in your infrastructure,
so that you can rapidly identify their causes
and minimize surface degradation and disruption.
Metrics and other measurements facilitate observability
and alerts draw human attention to the particular system
that requires observation, inspection and intervention.
In this section, we're going
to discuss alerting in Prometheus.
We gonna see how alerts work and what the Alertmanager is.
Finally, we'll see how to set up alerts,
get notifications when those alerting rules are met
using email and Slack.
Alerting in Prometheus is separated into two parts.
You got the alerting rules which are defined
into The Prometheus server, and
then you got a separate component called the Alertmanager.
As you can see on this diagram,
you got a Prometheus server where the rules are defined,
and then the alert got pushed to a route.
This route defines where the alert should go to,
via the receivers.
In this case, it will send a notification
via email or slack.
Prometheus alerting rules.
Rules live in Prometheus server config.
Best practice is to separate the alerts
from the Prometheus configuration.
You can do this by adding an include to
/etc/prometheus/prometheus.yml,
and you add the rule_files.
And the Prometheus configuration is reloaded.
It will also include this alert file.
Alert format, like this.
So you get the alert with an alert name.
Got the IF that's an expression.
This expressions were used in the query section.
So you can define a certain thresholds here.
Let me show you an example to get it more clear.
So we got an alert CPU usage.
And we got an IF,
which shows us the percentage of CPU used.
And then at the end of the line, we got the evaluator
that shows us that data needs to be more than 95%.
FOR, defines a duration, just can be like five minutes,
one second, you can choose there
and then you get a label.
The label set will be used inside the message template.
Use these labels carefully.
So you can set the appropriate alerting severity.
You can define other receivers for these alerts.
Think of a disk which is 85%.
So in my case, the severity will be warning
and send out an email.
But when the disk reaches 95% you should page somebody.
So let's say it's 2 a.m.
And we need to wake somebody up.
You don't need to wake anybody up for 85%.
But you need to do that for 95%.
These labels will help you reduce the noise on pages
and only wake you up or alert you
when human intervention is needed.
Alerting rules allow you to define the alert conditions.
Alerting rules sent the alerts being fired
to an external service.
The format of these are alerts is
in the Prometheus expression language
Let's see an example, of a group named
on important instance, the rule,
and the alert instance down, expression
is up equals zero for five minutes.
The label severity page and then add some annotations.
Summary instance with the label of the instance is done.
And then description to give more context
on the alert like it hasn't been done
for more than five minutes.
Prometheus Alerting-Alertmanager.
Alertmanager handles the alerts fired
by the Prometheus server.
It handles deduplication, grouping and routing of alerts.
Alert manager routes, alerts to receivers,
like pagerduty, Opsgenie, email, Slack, a custom wipebook.
whatever you define it.
The Alertmanager configuration lives in
/etc/alertmanager/alertmanger.yml.
And it kind of looks something like this.
So the global config shows you know SMTP server
which will be sending emails, you got a template,
like how the alert should look like.
Then you've got a route like repeat
this interval every hour.
So if no response gets in and you get to receivers.
The receivers define on which channels
you would like to get notified.
So in this case, you got the name operations team,
you got an email configuration with an email address,
and you got a slack configuration with a channel.
And it will also send a confirmation
when the alert is resolved.
In order for the Alertmanager to work Prometheus need
to know that the Alertmanager is there.
So by default Alertmanager is not installed
and you should alter the Prometheus configuration in
/etc/prometheus/prometheus.yml,
in order to be able to reach the Alertmanager.
Here we can see in bold
this configuration is very straightforward,
but it's just to give you an example.
So in this case, you got alerting the alerting-manager
got a static config.
These are the targets.
So now I only defined one targets
but you can add multiple targets there.
Alertmanager has a few concepts.
So grouping, groups similar alerts into one notification.
Inhibition silence other alerts if one specified alert
is already fired.
Think of this like the site is down,
but the CPU is still quite high.
So you want to get notified that the site is down,
but the alert that the CPU is high is a secondary alert.
So we're just gonna mute that for now.
As you already got alerted that the site is down,
you get silences, simple way to mute certain notification.
Let's say you're gonna do a maintenance window,
we're gonna upgrade my SQL so the site will be down
so you can silence those alert so you don't get paged
or get notifications whenever you're doing a maintenance.
Alertmanager can be set up highly available.
You can create a high available Alertmanager cluster
using a mesh configuration.
Now place Alertmanager behind a load balancer.
So instead of like placing the Alertmanagers notes
behind a load balancer, provide a list of Alertmanager notes
in the Prometheus configuration.
All alerts are sent to all known Alertmanager notes.
So this way you don't need to monitor.
Guarantees the notification is sent at least once.
Alert states.
So we got three alerts state, you got an inactive alert,
a no rule is met,
and a pending, this means the rule is met
but can be suppressed due to validations.
Let's say the site is down.
you got alerted that the site is down.
And you have an inhibition on for my SQL is not running
and the CPU is high.
These alerts will stay in pending
because the other one is fired.
Then you got another alert state called firing.
This means the alert is sent to the Configured channel.
So it just sees that it needs to alert
about the distance almost full and it will
send an alert to the Alertmanager.
And from that moment the alert state is firing.
The Alertmanager runs on port 9093.
You can see a small example of Alertmanagers UI.
As it can be quite useful to notify
multiple destinations at once.
You'll see a small example.
So this way you have the name operations team
with multiple receivers.
But this is quite useful when you want
to send a Slack message, a pager duty
and an email for example.
In the next chapter, we're gonna put
this theoretical knowledge to the test.
And we're gonna set up the Alertmanager
and all the necessary config to eventually
send the notification via email and Slack.