In this demo, I will show you how
to run Prometheus on Kubernetes
to get the the Kubernetes metrics.
First, you will need to spin up
two new droplets to install Kubernetes.
So I spun kubernetes-01 and kubernetes-02
with the tag kubernetes.
These are just two droplets
with two gigabytes of memory each.
It's exactly how you spun the Prometheus droplet.
In this case, we now have two droplets
to run a Kubernetes cluster on.
For networking, you want to setup a Kubernetes firewall.
So here we have a Kubernetes firewall exactly
in the same way that we did the Prometheus firewall.
We allow TCP to an ip address, which is my IP address,
and then we allow TCP to the tag Kubernetes,
so that the two droplets can communicate with each other.
Now we just have to install Kubernetes on the nodes,
and I'm going to use a script that I have written previously
that I use in one of my Kubernetes courses.
So if you're interested in learning more about Kubernetes,
I would suggest you have a look at those courses.
I'm just going to install Kubernetes
and not go over the details of the scripts.
I'm logged into kubernetes-01 and kubernetes-02.
I'm going to get clone
my on-prem-or-cloud-agnostic-kubernetes git repository.
It's a Git repository for my other kubernetes course,
and I'm going to do that also on kubernetes-02.
Once it's cloned,
I have the on-prem-or-cloud-agnostic-kubernetes directory,
and here I have scripts,
and in scripts, I have an install-kubernetes.sh.
It's also important to note that this is
the 16.04 Xenial Ubuntu release,
and that is currently the only release
that this install-kubernetes.sh will work with.
So if you use the more recent 18.04,
it will not yet work because the docker version
is not tested with.
In a later stage, it'll probably work,
but for now, I would suggest that you use Ubuntu Xenial.
I'm going to do the same on the other nodes,
but rather than executing install-kubernetes,
I'm going to run install-node.
So kubernetes-01 will be our master.
Kubernetes-02 will be a worker node.
This will some time to install,
so it's going to install docker in kubernetes.
It's using kubdeadm to launch the kubernetes cluster.
It will now pull the images, docker images necessary
to run the Kubernetes cluster,
and now the cluster is up and then you see this line here.
You can now join any number of machines
by running the following on each node as root.
So I'm just going to copy, paste this,
and I'm going to run this on kubernetes-02,
and on this kubernetes-02,
we'll try to join our Kubernetes cluster.
So our second node has joined.
There's one more script that I wanna execute here.
I wanna create a user,
so this will create an Ubuntu user
that I can use,
and if I then type kubectl get nodes,
then I can that my master is ready,
are ready for 47 seconds,
and the second node is now joining,
and now it's also ready.
Let's git clone our Prometheus course.
Git clone github.com/in4it/
prometheus-course.git,
and in this prometheus-course,
we have a kubernetes directory,
and here we have a README,
and in the README, I have the instructions
to Install Helm, which is the Kubernetes package manager.
I'm going to start Prometheus,
and then we're going to expose a Prometheus port,
so we can connect to Prometheus.
The reason that we're running Prometheus
within Kubernetes is that Prometheus will need access
to the internal service running in the Kubernetes cluster.
So it's best to run your Prometheus
within the Kubernetes cluster,
and we can do that easily using a package,
using the helm chart prometheus.
I'm going to run the Prometheus server without persistency
because at this point in time,
we are not going to configure any volume to save the data.
This is just a demo.
If you're running a real Kubernetes cluster,
then you're gonna have persistentVolumes available.
For example, in AWS you have EBS,
and then you can just remove these set lines
or put it do true and then you will have persistentVolumes.
Let's start with installing helm.
So I downloaded helm, extracted helm.
Let's move the helm binary too,
user local/bin/helm.
Now we should be able to execute helm.
Let's create our rbac rules for the tiller
account, so we're gonna create a serviceaccount tiller,
and then we're gonna create a rolebinding tiller.
This is just for helm,
so that we have a serviceaccount for helm,
so that it can deploy packages for us.
If you're running this on a production cluster,
you might wanna change this file a bit
to limit the access
because now we are giving it an admin account,
and there we should be able to help inits,
and now we have helm installed.
Now that we have helm installed,
we can start Prometheus, helm install, name prometheus.
We're not going to use persistentVolumes,
and we're going to use the stable/prometheus helm chart.
This will deploy Prometheus on our Kubernetes cluster.
So if we now do a kubectl get pods,
we will see that prometheus is launching
on our Kubernetes cluster.
Now we want to connect to it.
Let's have a look at the README.
We can use export POD_NAME to export the POD_NAME.
So our POD_NAME is prometheus-server
and then this string, which is the same as here
when we did kubectl get pods,
and then we can do a port forwarding.
We can go port-forward port 9090
to a localhost 9090,
and then we're going to use socat
to port-forward localhost 9090 to all the TCP IP ports,
so that we also can access it from externally.
This runs in the background.
If you want to bring it back to the foreground,
you can type fg, then type Control + C,
or if you just wanna see what's running in the background
you can use jobs.
If you log out, this will also stop working.
So now we have our Kubernetes up and running.
So let's have a look at our Kubernetes instance.
This is the IP address of my Kubernetes-01,
9091 forwards to localhost 9090
on that kubernetes-01 node,
and it forwards to the 9090 on the pods.
So this is just a normal Prometheus install,
but it comes preconfigured for Kubernetes.
So you will see that will have multiple Targets here
configured for Kubernetes.
So our Kubernetes has a kubernetes apiserver.
This is the master server,
so we can pull metrics from the master server.
We have the kubernetes-nodes two out of two are up.
So the first one, if you see here, is the kubernetes-01.
The second one has a host name kubernetes-02.
We're also scrapping from cadvisor,
which gives us docker metrics,
so we are contacting kubernetes default as we see.
This is a local host name used in kubernetes
to contact the API server,
and then on api/v1/nodes/kubernetes-01/
proxy/metric/cadvisor, we can scrape the metrics
to get the cadvisor metrics.
Then we can scrape the service-endpoints.
For every service that supports metrics,
we are going to scrape them.
So if you have a look here,
we have here a Kubernetes app kube-dns.
Kube-dns, and here we have prometheus.
So we can scrape from kube-dns and from prometheus,
but you can create more services within kubernetes
that can have metrics enabled
to easily enable metrics on your service-endpoints.
It's also going to scrape from prometheus itself,
and we have the prometheus-pushgateway,
which comes with the helm chart.
So how is this all configured?
You can check the Configuration here
to see how the configuration has been done,
but this doesn't have any comments anymore,
so if you want to have a better explanation right here,
from github prometheus, we have in the documentation example
prometheus-kubernetes.yml,
and here you have comments.
Actually, here you can read a little bit more
how, for example, the kubernetes-apiserver is implemented,
and we'll just highlight the most important parts.
So Prometheus is running as pod on our Kubernetes cluster,
and it needs access to the Kube apiserver
to pull those metrics.
To do that, it has a bigger token_file,
which is basically the token, the secret,
that it needs to contact this apiserver,
and this token is on var/run/secrets/
kubernetes.io/serviceaccounts/token.
So we have serviceaccount assigned
to prometheus and this gives us a token
that we then can use to contact the apiserver.
We also have the ca certificate,
which is the ca that is used by the Kube apiserver.
The scheme is http.
We have 10 out of 10 seconds,
and then are going to relabel some configs
to then get every apiserver that is running
on the Kubernetes cluster.
So if you want to know more,
you can just go to the prometheus kubernetes
and here relabel_configs
for the kubernetes-apiserver has some more explanation.
So keep only a default kubernetes service endpoints
for the https port.
This will add targets for each API server
which Kubernetes adds as an endpoint
to the default kubernetes service.
So default is the name space,
the default name space and kubernetes is just standard name.
This is the alerting, alertmanager config,
but you see then, for every config
you will have this bearer token.
So here we have the job_name prometheus.
Well for prometheus, we don't need this
because this is just localhost 9090,
but then all the other ones kubernetes-apiserver,
bearer_token, ca_file,
relabel_configs, kubernetes-nodes.
This is to get the nodes statistics.
Again, this bearer_token, relabel_configs
to build the correct URL to then get the metrics
from every node that exists in our Kubernetes cluster.
Same for kubernetes-nodes-cadvisor,
and then we have the service-endpoints,
and we also have kubernetes-pods,
which will scrape if we have annotation.
So if you look up at this in our config annotation,
so here we have an annotation example for our pods.
Example relabel to scrape only pods
that have the example.io should be labeled
to true annotation,
so you can add an annotation, the key value pair,
to your pods in a Kubernetes cluster
and only when this is true,
then the targets will be scraped.
And then for every target,
we then have the targets listed here.
Once you're done, do to metrics
that you will see you will have a lot of metrics
that you can use,
so let's for example have a look
at kube_deployments_created,
and if you done Graph this,
so we have a deployment for tiller,
which is for helm for the prometheus-server
pushgateway, prometheus-kube-state-metrics,
prometheus-alertmanager and our DNS server.
So whenever we're going to deploy something
to our Kubernetes cluster, it'll be here in the list.
So these are a lot of metrics,
and you need to be a little bit familiar
with Kubernetes to know which ones are relevant,
which ones you can use.
You can send these metrics again to Grafana,
and then with Grafana, you have then again
predefined dashboards that you can use
to show you some nice information
without a lot of effort.
So this is how you setup Prometheus in Kubernetes.
I would suggest that you do it this way
with the helm chart.
Makes it a lot easier,
rather than configuring everything yourself.
If you need to configure it yourself,
you can use the Configuration that's right here,
and the examples configuration
that is provided by the Prometheus docs.
You will need to know a little bit about Kubernetes
because you will have to configure these
bearer_token_files to make sure
that you can access the API server,
and you will also have to use these relabel_configs
to get the correct endpoints,
the service endpoints, to contact these internal services.
So it's a little bit complicated,
but as you can see, by using packages
all the configuration is done for you,
and then you can just start using it immediately.