Prometheus service discovery.
Having to manually update
a list of machines in a configuration file
gets quite annoying after a while.
One of the features of Prometheus is service discovery.
This allows you to automatically discover
and monitor your service or services.
The definition of service discovery
is the automatic detection of devices and services
offered by these devices on a computer network.
Having to update a file
whenever you add a server or a server leafs
or if you have a dynamic workload,
this is not maintainable.
So this is not really a service discovery mechanism.
However, Prometheus has support
for the major cloud providers AWS, Azure and Google,
for all the major cluster managers Kubernetes,
Marathon, Docker Swarm,
and a generic mechanism like DNS, Consul, Zookeeper
or even a file discovery.
Let me show you an example
of configuration file used by Prometheus
to how to discover easily instances on AWS.
So as you can see here,
you should add your access key and your security key,
but you also have to make sure the user,
which you provided the access key and the security key from,
has the appropriate IAM role.
In this case, the AmazonEC2ReadOnlyAccess.
This is needed for it to be able to
scrape all the EC2 instances in your account.
Also, make sure your security groups
allow access to port 9100 and 9090.
As you can see here, you can relabel configs.
What does that mean?
You can only monitor instances, for example,
with a tag name started with PROD.
So as you can see below,
you got a source labels
which looks at the meta data from EC2 tag name,
which is a field in AWS.
And you perform a regex on it,
like PROD and every value that comes behind it,
let it match.
And then action keep.
This means whenever an instance starts,
for example with DEV or QA,
it will not get monitored.
What you also can do is relabel an IP address
to an instance id for convenience.
So, as you see here to source label
which looks as a meta data of EC2 instance id
and it will replace the default target label
with the instance id.
Prometheus service discovery, example Kubernetes.
As you can see here,
you get a default configuration file
in order for Prometheus to be able to scrape Kubernetes.
It's a very straight forward configuration file
as it just uses the API server of Kubernetes.
It has to be able to authenticate to it.
In this example we used basic auth,
but you can also if you look up the configuration file
in the Prometheus documentation,
can also use certificates.
And then you give it a service name, the job name.
And then you make sure he scrapes everything
from those API services.
Prometheus service discovery, example DNS.
So this is also quite nifty about Prometheus
is that whenever you get an application
that registers itself to the DNS,
you can easily do this by adding the following configuration
to the prometheus.yml file.
DNS-based service discovery configuration
allows specifying a set of DNS domain names
which are periodically queried
to discover a list of targets.
So one DNS record can give back multiple services.
Let's say you got MySQL
and you're just registering all your MySQL service
under that DNS name which will get returned
and then Prometheus will look at the DNS query,
see what hosting got returned
and start scraping those metrics from that node.
Prometheus service discovery, example using a file.
So you can also add a file.
In this case we added target.json
to the prometheus.yml file.
So the use case of this is that
let's say you're matching your infrastructure
with Chef or with Ansible,
you can update this file whenever you deploy something.
So Prometheus will check this file in a regular interval
and then it will automatically pull those metrics also,
if they're defined in here.
So this way, you can easily work around
if you don't have anything set up that uses DNS
or whatever more.