Prometheus use cases: EC2 auto discovery.
Service discovery is the automatic detection of devices
and services offered by these devices on a computer network.
So this is the general definition of service discovery.
We will leverage this feature in Prometheus
to auto discover EC2 instances.
In this use case we will create the prerequisites in AWS,
in IAM role security groups, and of course EC2 instances.
We'll alter the Prometheus config to scrape those
EC2 instances, we query the data in Grafana,
and you can find all our commands on the GitHub page.
So let's get started.
So first of all I'm gonna show you where you can
find the commands and all the config variables
that I'm gonna use.
So you go to use cases, EC2 auto discovery
on our GitHub page.
And then you see a file, lab.txt.
You can find everything that I'm going to enter
which is prone to error when you're typing it over.
Let's get started.
First we're gonna create an IAM role.
We go to services, security, identity, and compliance.
And you select IM.
Then you select users.
We will add a user, let's call it Prometheus.
We're gonna give it programmatic access.
And now we're gonna attach some existing policies directly.
So in order for this lab to succeed, you need to have
the AmazonEC2ReadOnly.
So we're gonna select it.
This is the manage policy, the AmazonEC2ReadOnlyAccess.
And we're gonna create the user.
Gonna download the CSV so we have the credentials
on disk so we don't lose it.
Like I said before, be careful with this file.
There we go.
And now we can start altering the config.
So we're gonna log in to our digital Ocean machine.
And we're gonna alter the configuration of Prometheus.
So we're gonna go all the way to the bottom.
And add this from our GitHub page.
So as you can see here you need to put the access key here,
the secret key here, but as you can see also,
I'm gonna relabel some stuff.
So right now, I'm only gonna scrape EC2 nodes
starting with Prometheus.
So it's very important that you set the name tag correctly
or you just leave this out if you don't want this.
But it's just to demonstrate the power of Prometheus.
What I'm also doing is relabeling the address
as we will be using the public IP
addresses of the Prometheus
machines we want to monitor.
So we need to relabel them, else Prometheus will report
the internal IPs by default.
But as our Prometheus machine is in a digital Ocean
environment, we need to do this.
If you're running your Prometheus machine,
which is actually recommended in the same VPC on Amazon
you can leave this out.
And only these five lines should be sufficient
to monitoring your machines.
So let me add my access key, there we go.
Now we're gonna add the secret key,
now we should be good to go.
Gonna write quit it.
Restart Prometheus, check the status to make sure
we didn't make any errors.
Prometheus is running, now let's check if it's
reporting something already.
Go to status, targets.
As you can see it's not reporting anything yet.
Now let's head back over to our AWS environment.
Go to services, EC2.
We're gonna go to instances, gonna launch some instances.
Gonna use the Ubuntu one.
Just take one.
We're gonna launch.
You can select this number as it's appropriate to you.
But I'm gonna launch five.
Gonna leave all this to default, but we're gonna add
some advanced details.
And I select it, and in here we're gonna paste this
user data script.
This will actually install the node exporter.
Let's add some storage, eight gigabytes should
be sufficient for this lab, we're gonna add a tag.
Name, and we're gonna call it
Prometheus.
Now we're gonna create a new security group.
We're gonna allow TCP traffic, we will be needing
port 9,100.
We're gonna add another rule, also a custom TCP port.
9090, and now let's check to make sure
our environment is secure.
We'll limit the source to our digital Ocean machine.
So on digital Ocean I'm gonna copy the external IP address
and back to the AWS console, and I'm gonna limit it on 32.
So 32 means we're only allowing from this exact IP.
I'm gonna do the same here.
And then I'm also gonna allow some SSH traffic.
Only from my IP.
Then we're gonna review it.
That all looks great, and let's launch it.
While this is launching, we go into services,
EC2, then we can follow up the progress.
As you can see there are five instances starting
with the name Prometheus.
In the meanwhile we can monitor on the Prometheus side
to see if there are already instances reporting.
As you can see, you've got five nodes already
which are all down because they're still initializing.
As you can see here, connection refused, or a timeout,
this typical timeout error.
So we just have to wait.
I'm gonna pause the video here until every
instance is online and the metrics are reported
back to Prometheus.
As you can see, it went quite fast,
and here are all the instances.
So they're up, the last one came also up.
And as you can see here, the endpoints are the external
addresses of the machines.
So now we can head over to Grafana
and try to draw some graphs.
So we're gonna create a new dashboard, graph.
We're gonna edit it.
The data source is Prometheus.
Let me copy paste the query.
And as you can see it's already reporting some data.
But as those machines aren't doing anything,
I'm gonna generate some load on one of them.
So gonna head back to here, just gonna take the first one.
I'm gonna SSH into it.
There you go, we SSHed into it,
then I'm gonna generate some load.
The command is also in the doc so this will just
generate some data, and as it's gonna use the input,
dev/zero and put it on dev/null, it's not gonna impact
the system but it's gonna generate a lot of CPU.
So let's do that.
And that's it, keep running.
Oh, let's head back over to our Grafana.
And reload our page.
Add a graph.
Gonna edit it.
As you can see, here is one instance reporting
at 100% CPU, so this is the one we're you're running
that command, all the other ones are idle
because they're not doing anything.
So this is actually what we wanted, so it's nice that
we can use this to auto discover EC2 instances
as it's a hassle and it's just super annoying
to update the list of nodes you want to monitor.
You just have to keep in mind that the node exporter
needs to be running on every node, and if you share
your accounts with let's say the testing environment
and production, you can always limit it
in the Prometheus configuration
to only allow certain instances.
So let me show that again to you.
As you can see here, we're just only keeping
every instance starting with Prometheus.
But another good example, let's say if all your instances
are tagged with a name starting with PROD,
you can do it just like this.
And it will only scrape the nodes with the tag name
starting with PROD.
I hope this was useful for you, it has been useful
for me in the past.
For other clients, this is useful and eliminates
the time consuming editing of the config files
whenever a node leaves or gets added.
Hope you enjoyed it, thanks.