In this demo,
I will show you how to do the consul integration
with Prometheus.
I'm logged into my Prometheus server,
and I already have Prometheus installed,
and it is running.
First thing that we need to do
is you need to install Docker,
because I will use Docker to launch consul.
If you haven't done that already,
you can use script/install-docker.sh
to install Docker.
I already have it installed to my system,
because I executed this in a previous demo.
So, I will just control c for now.
But, you need to make sure you have Docker
and Docker Compose on your system.
Then, I can use start-consul,
which will start the consul server.
Let's have a look at this script first.
It uses docker run, gives it the name consul,
uses my host network.
I processed data in /consul/data,
and I have a config,
where I saved as my datacenter eu-west.
This is the server; this is the consul server.
You can also run in just a client mode,
but this is server mode.
I'm going to enable debug.
This is version 1.2.1,
the agent in server mode,
I'm going to bind on my local IP,
and I only except one node.
So, typically you would run three nodes,
for simplicity I'm going to use one.
If you want to use this in production;
you have to change this into three,
and then you have to run two more servers,
and then the auto-consul knows have to join the cluster.
But, that's not relevant for us.
A one node cluster behaves in the same way
as a three node cluster.
So, we are going to keep it at one node.
And, once the install is complete,
I'm just going to echo the following lines.
Let's see if there's anything still running.
There's nothing running anymore.
So, I'm just going to start consul.sh,
and let's have a look now.
Consul is running.
Let's have a look at the logs.
Docker, logs, consul.
Seems to be okay.
And, then I need to copy, paste, these lines to
my Prometheus,
Prometheus.yml.
So, job-name consul, this is the consul server,
should be able to reach at one.
Let's check that.
Yes, it says consul agent, so this works.
Let's reload Prometheus with kill -HUP.
And, then we also need to register a service,
otherwise it will not do much.
Let's check whether...
it will successfully start it, yes.
Loading configuration file, no errors.
And, now we have to load service.
So, let's start a service first.
In the Prometheus course,
I have this flask Prometheus that we can use.
It's from a previous demo.
So, I'm going to do docker-compose up -d.
And, this will start the flask server
that runs on localhost
8000 for Prometheus metrics and 5000 for the HP server.
And, you have to try a few times,
because it takes some time to start-up.
Flask is up and running,
I'm just going to do, probably a few times,
so that I have some metrics.
And, then I'm going to register this service.
So, normally, like I said in the previous lecture,
you will use some auto discovery mechanism,
not going to do that
because that's a lot of configuration work.
And, it has nothing to do with the Prometheus course.
So, I'm going to manually register,
it has a script,
scripts/register-service.sh,
and this script is going to register a service
with an ID named python-flask.
It's on the address IP,
port 8000, that's where I'll scrape my metrics from.
And, we're going put a health check in place
on 4, 5000 to use whether are service is running or not.
And, this is going to be sent to this url.
This is the consul API.
So, do curl -X put,
script/register-service.sh.
This registers the service,
we can also have a look
whether this was successfully registered.
Using the same API,
curl localhost:8500/v1/catalog/services,
and we have consul, and we have python-flask.
So, the service was registered.
And, now if everything went okay,
we should be able to see this same service,
or actually both services,
consul itself and python-flask
in our Prometheus interface.
So, this is Prometheus.
(mumbles)
Let's have a look at targets.
This consul is not gonna work,
because this is a service that doesn't expose
the Prometheus metrics.
But, here is our python-flask
that exposes the metrics and this one is working.
So, it's running on the instance Prometheus
and let's have a look at our metrics.
So, we have this flask_request_count for example,
that we can use.
Let's execute this,
and now we didn't have to add any targets,
we didn't have to say to Prometheus that we were
running this app, this app runs its course through consul
and the metrics were automatically added.
So, we see here that we have
a flask_request_count(endpoint/http_status "200",
instance "prometheus", job "python-flask".
Remap the service name to the job.
So, we also have the label on it.
We also configured a service check,
so this is the service check that we see.
So, our query is here,
we execute a few times,
the slash query and service check.
If you refresh a bit,
then you see that this line goes up,
because the service checks are running,
and then the count obviously goes up.
Let me show you the Prometheus configuration again.
So, we have a look at prometheus/prometheus.yml,
then we see here the relabeling config.
We have job_name: consul,
we configured the consul server,
and then here we do a relabel.
So, that consul service becomes the new job.
The consul node becomes instance,
and the tags, consul tags there.
Then, if they match this,
then we gonna relabel them to group.
So, we can have group dev four.
A service of the tags, dev for example.
If you want to only include node of a certain tag,
you can add a tag here,
for example to make sure that you don't add
the consul server.
Here, everything is running on one server,
so it will not really have affect.
You have seen that the consul server was in the target list
and maybe you don't want it there,
because it's not scraping any metrics.
So then here, you can specify a tag
and only nodes using that tag will be included.
So, that way you can make sure
that you don't include your consul server
or any services that don't have Prometheus metrics exposed.
So, that's it for automatic adding targets,
but there is also consul exporter,
the script install-consul-exporter,
will install the consul exporter.
And, this works very similar as the node exporter,
it's almost the same script.
But then, it is for the consul exporter.
So, the consul exporter will scrape consul
for some more metrics of the services.
So, let's install this one as well.
Scripts install,
then consul exporter
and then we just have to add these lines to Prometheus.
This is the consul exporter;
this will be running on localhost:9107.
Restart Prometheus again,
and then we can go to our targets,
and then here now we have the consul exporter.
And, the consul exporter will also expose
some interesting metrics like the node health
or the service health.
For example, one query
that you could execute
is show me
the consul catalog service node healthy
by service name.
Then we execute this,
and if you see service name consul
and service name python-flask,
and they are both healthy.
So, you can put alerts
based on whether a node or a service is healthy or not.
This exporter doesn't expose a lot of metrics,
but the health is one of the very useful ones
that you can use and put alerting on,
so that you get an alert when a service doesn't work,
for example.
So, this was it for the consul integration.
So, I installed the consul server,
then I enabled integration in Prometheus,
and then I also installed the consul exporter
to get metrics of consul and of my services in Prometheus,
so this is a very nice way to not have to config here
all your services manually,
but have them added automatically in consul
and then in Prometheus.
So, again I'm not really explaining how auto discovery
or how the integration of consul
within of info-structure works.
This is very different for everyone.
If you're interested in consul,
I shared some urls in the previous lecture,
then you can read about consul
and how to implement it in your info-structure,
then this lecture really explains you how to
integrate consul within Prometheus.
Try this out yourself,
and let me know if you have any issues.