In this demo I will show you how to monitor
a web application
with Prometheus.
I'm logged in to my Prometheus Droplets
and I have Prometheus running.
I also have a final running,
but I don't have anything else running.
The node exporter, for example, is not running.
You can have it running when you do this demo,
but you don't need it.
So I have my Prometheus course git repository checked out.
And here there is a directory, flask Prometheus.
And this is the application that we are going to deploy.
Let me go over the application first,
and then I will deploy it.
So there's a Python application
and all the initial code is in main.py.
First, we're going to start with importing our libraries.
We're going to import MySQLdB,
which is bundled with the MySQL client
going to import OS because we want to
get some environment variables.
Go to import flask,
and we're going to import some functions from
flask Prometheus.
This flask Prometheus is
a library that is in a directory flask Prometheus,
I also have to explain you this library.
After importing, we're going to start the flask HTTP server.
So we're going to do app equals flask.
And we're going to initialize
the connection to the database.
So db=MySQLdb.connect
using a host a user password in a database.
And I'm going to get the correct host user password
and database from the environment variables.
So MySQL HOST USER, password and database
can be passed as an environment variable
that I will do during deployment.
And then we're going to create treat routes,
you know, with a route for slash,
for slash query and for slash sleep.
So when we have the app running,
we can go to the URL slash query, sleep or just slash.
And then we'll end up in one of those methods here.
So if you don't pass such query or slash sleep,
then you will just get a string flask is up and running.
And otherwise, you will get in one of your route.
Then we also going to run the monitoring.
So the monitoring is gonna be one of our own functions.
We're gonna pause the app
and we're going to say please run on port 8000.
So if we going to go to the IP address, Port 8000,
then it's going to give us the Prometheus metrics.
And the app itself runs on port 5000 by default.
We use it here monitor and we also using here mysql_fetchall
those are imported from the flask_Prometheus.
So let's have a look at the flask_Prometheus
library.
These four lines here are defining our data types,
the histogram and the counter.
I explained this in a previous lecture.
Here we are importing the Counter histogram.
And we're also importing to start_http_server
to start our Prometheus HTTP server on port 8000.
So let's have a look what this monitor does,
I will do a split screen with the main.py.
So here we're calling monitoring.
So this monitoring, make sure that we are logging every hit
on our server on our HTTP servers
every time you make a visit, we want to log this
and also the latency.
So if you have a look at monitor,
we bind the app before request
and the app after request functions.
So we're going to say use these defined before request
and after request here
to make sure that we execute this code here
to calculate the latency.
So every time a request comes in, first this before request,
this method here will be executed.
And then after the request, this one will be executed.
So this will just lock the start time.
And here we will calculate the request latency
based on the current time minus the start time.
And then we define the flask-request-latency
and the flask-request-count.
So here we add the labels, the methods, the path,
and then we and then we passed the latency.
And then here with the current impasse, the method, the path
and also the status code, let me just increase a Counter.
So, this is to calculate the latency
and to increment the counter.
Then you're also going to start the HP server.
So when we call here monitoring,
we bind before request, we bind after request.
And we all start a server.
So this then runs the monitoring, this runs and app.
So what about the queries, we going to execute two queries.
And I have written a custom query method for that,
so that we can lock the latency.
So here, we have the slash query endpoint.
And we'll hit search query, we're going to do
a MySQL query, we're going to call mysql_fetchall.
So here, we have mysql_fetchall
and then we can calculate the latency
start time is now start time equals the current time.
There we go with the execute query.
This is just something MySQL client goes to execute a query.
And then we're going to lock the finish time.
So again, we're going to lock the latency
and also increment the counter.
So this particular here,
we only going to look at the first 50 characters
of a SQL query because they can be super long.
We only want to log the beginning of it,
then we're going to return the data,
in the query returned to any data.
And that we can then use right here,
res is our data, then we have the output,
we want to initialize it to an empty string.
And then this is just going to output the lines.
So whatever this query is going to return,
we'll output the lines, so that we will just see the rows
and then the values comma separated.
And then we're going to return the output.
What is the query going to be, we do a select one as ID,
and then we generate a random number between zero and 10.
And then we do a union, like two as ID
and then a random number between zero and 100.
So this will give us two rows,
one column ID, one column rand.
So the first row will have one as ID and a random number.
And then we'll have two as ID
and then another random number.
So this query is going to execute super fast.
will take no time to execute
and that is a query that we should not worry about
in our monitoring.
This on the other side, this slash sleep
is going to sleep between zero and 10 seconds.
So this is going to be a query that is problematic.
And we should be able to identify this query
in our monitoring system.
So this also as in mysql_fetchall
we don't do anything with results,
but is going to use the same function.
So let's deploy this application,
and I package this application in Docker.
Slash Dockerfile here.
What did I do?
You have to be a little bit familiar with Docker.
If you're not, don't worry about it,
you can just execute the commands to run this.
I will just briefly go over how I built this Dockerfile
for if you're interested.
And then we go into deploy it.
So this from Python:3-alpine,
we'll get Python as a base image Python three
I'm going to deploy usr/src/app
and copied requirements to my Docker Container,
requirements are the libraries that I need to install
to MySQL client, the flask and Prometheus client.
I'm going to install some packages that I needs like
the MySQL dev files,
GCC to compile my libraries and so on.
And then I'm also going to add a tool,
wait-for-it.sh that will just wait
until the database becomes available.
Otherwise, app will just immediately terminate,
because it takes some time before the database is available.
We're going to copy the app in the Docker Container,
going to export 5000 and 8000,
and I'm going to start the app using Python main.py.
It's also worth mentioning that
you wouldn't use Python main.py in production.
You still need a real web server in front of this.
If you're familiar with flask, you notice,
that this flask, this Python flask
is not really a web server it's a framework,
so we still need a web server in front of it,
that will then launch this python main.py multiple times.
So it can serve multiple clients.
When you do this, you can also use a reverse proxy
to bundle the Prometheus endpoint
together with your main endpoints
that you only have one port.
It's more work, but it's nice to have everything
over one port.
There's not really immediately a benefit over this
it's just another way of working.
You can have two ports exposed
or you can have one port exposed
and have your Prometheus on one URL,
it doesn't really matter.
We are going to use two ports.
So you will clearly see the distinction between our app
and the Prometheus metrics.
We don't really need to rebuild the Docker image.
I already did that for you, and then I put it publicly.
So then I have the docker-compose.yml
that you can use to launch these images.
The Flask-prometheus-example image
I'll expose Port 5000:8000 on my whole system
so that they can access it the environment.
So these are the environment variables
that we're going to pass through our app.
Host is MYSQL, User root, Password app and database app.
This is the image that I'm going to use,
I pushed it to Docker Hub.
So now it's available at this URL.
This depends on MySQL, which is defined here.
And the command that I will execute is wait-for-it.sh
wait for MySQL to be available on port 3306
and then execute the main.py.
So this MySQL is going to be a service
and we can reach this MySQL with using the same name.
So the host MySQL will become available
and point to this service.
That's why we can use the MySQL host MySQL.
We have image MySQL:5.7
we go to give it a password app
and it's going to create the database app.
So to run this I first need to install Docker.
So I have in scripts and installed docker.sh script
that will just install Docker from getdocker.com
and we'll install Docker compose.
So let me run this first.
This will install Docker it'll take some time
and then we will be able to run Docker compose
which will download our images to MySQL image,
the flask image and then run it.
The installation is complete, let's run it.
Let's go back to the flask Prometheus
and let's now enter Docker-compose up minus D.
So it will run the two services defined in Docker compose
and it will do that in the background.
So now they are running.
Let's have a look with docker ps.
So both of them are running
and it also exposes the boards as you can see,
now, I will be able to do a curl on port 5000
on localhost.
And this will give flask up and running
I can already execute this slash query.
So you see here, it just returns two rows
with random numbers.
And I should also be able to execute sleep,
which will then sleep for some time,
up to 10 seconds, and then it will say done.
And now you can see that this is gonna be
the problematic query, because it keeps a client waiting.
So this should turn up in the monitoring.
The monitoring is available on port 8000.
If I do a curl localhost 8000, I get my results,
I get my Prometheus data.
And here we can already see our histogram.
For example, mysql-query-latency,
like one as ID is the normal query we executed three times
And you can see here that it executed three times
in a very short time, so it was less,
it was less than 0.005 seconds,
still less than 0.1, still less than zero to five,
because they all have gone three.
Now, if you have a look at the other query,
that we see that we only executed once,
so the query was definitely not executed less than 0.005
does only execute less than five seconds,
but more than two and a half.
So if we execute another sleep,
then another sleep,
then we'll see some more data.
So this one's a long one.
So now we see for instance, there were two
that were less than five but more than two and a half,
but then there was one that was still less than 10,
but more than seven and a half.
And this, we can then put in a nice graph,
so that we can see what queries are problematic.
We can also see the hits.
For example, the slash query,
and slash sleep will also have brackets.
And this will give us information
but on a different level,
on the level of endpoint,
which doesn't include the query.
But it also notifies us that there is something wrong.
And if the problem was not with MySQL,
then we will see that all queries would go fast.
And this will still go slow.
For example, if there was a slip in the code,
or there was an issue in the code,
so that's why you have different metrics for database
and for your endpoints so that you can see
where the problem really is.
So you see four slash query, never gave an issue.
It always was three seconds.
But here done here at sleep you can see that this is
problematic endpoint that it took multiple seconds
to execute our
slash sleep endpoint.
So let's put this in a graph.
First, we need to configure Prometheus.
Let's go back to scripts.
And I have a script here, add flask-app.
And this will add to the Prometheus ylm file,
the job flask-app and it will say to the target is 8000.
So let's execute this.
Add-flask-app.
And let's reload Prometheus.
So I'm going to still help 1311 1311.
And this should reload Prometheus.
So let's have a look at Prometheus.
So here, I went to the Prometheus server.
Let's have a look at status targets.
And we have the flask-app is running.
Let's create a query.
And one of the interesting queries that you can do
is to calculate the average of the observed values.
And we do that.
Let's do it for the MySQL queries.
We can do that using the MySQL-query-latency seconds some,
and the MySQL-query-latency count.
I'm going to divide the MySQL-query-latency seconds sum
by the count, and it will give me the average.
I need to
use rate for that.
So I'm going to use rate.
And I'll calculate this average for every five minutes.
So I'll use rate all divides.
And here I'm going to use a count.
And then I'll execute this.
And then I'll execute a few queries.
So let's do curl localhost 5000 of query
just a few times
and then sleep
I will do this in a loop, while true do
time to see, let me see how long it takes
of this sleep query.
So this will just run forever.
There's one took 4.8 seconds.
The next one takes eight seconds.
So you see, it'll just keep on running.
Then we go back to Prometheus.
And now we see that here, we have one query
that takes almost no time is still zero.
And then here we have a longer query
that took us 4.8 seconds
and then 6.3 seconds,
and then it will just continue on running.
So here, once you have, let's say hundreds of queries,
you can then see hundreds of lines mind you,
you will then definitely see which ones take longer.
And those you can then start investigating.
So this is a very powerful way of monitoring
your application metrics to see
what part of your application is slow.
And this is something you can add to your graph
on dashboard to see the latency of your queries
or the latency of your endpoints.
And if something goes wrong with the application,
in just a couple of seconds, you can see
where you have to look for the problem.
So that's it for this lecture.
It was pretty long.
It's actually a very useful demo to try it out yourself.
And if you have a custom application,
you can implement this way of calculating the latency
for your app and your queries.