In this demo,
I will show you how to use
mutual TLS for the targets.
So we are going to enable TLS encryption on the targets,
but mutual TLS,
so it will also would for authentication.
With mutual TLS,
your server needs a certificate,
but also the client need to a certificate.
And using these client certificate,
it's like a login and password you can authenticate
and then you can connect it to the server.
So, I will show you how it works with the
Prometheus server that is going to connect
to a target using encryption,
but also using mutual TLS.
So,
I spun up another droplet
there's also on the window machine,
one gigabyte of memory,
and we'll just install the node exporter here.
We'll do a git clone,
and I'm going to install the
node exporter.
Exactly as I did with my Prometheus server,
my other machine.
Even would like to scrape the metrics from this new node,
the Prometheus node.
So this is a new node.
Then I would have to add node exporter,
but instead of target local host,
I would have to use the IP address,
which is 1-3-8-6-8-1-3-5-9,
'cause it not local.
But if you use a remote IP address,
you actually want the target,
the node exporter protected.
You want it data is encrypted and authentication is used
using certificates.
So right now just copying and pasting this
and adding this target.
We are going to put a reverse proxy
in front of this one.
And this reverse proxy will expect a certificate from the
Prometheus server and that will be used for authentication,
and then also for encryption.
So first of all you need to make sure
that the Prometheus server,
the first machine that you bring up,
can contact this new one,
the Prometheus node.
And to make sure that this works,
in inbound rules of your Prometheus firewall,
you have to make sure that you add
the Prometheus tag for TCP,
I also use it for UDP,
but definitely for TCP.
Enter here,
add the tag or you can also use a droplet name.
So my droplet is call Prometheus node,
but my Prometheus node is tagged using Prometheus,
so that's why I can use a tag as well.
Although actually you can use just Prometheus node.
This will allow the Prometheus server to communicate with
the new instance.
Let's test this out.
On my first tab,
I am connected to my Prometheus server,
so,
curl,
this IP address,
and then I can contact a node exporter.
So lets install a reverse proxy in front of this node
exporter and then go to your "det port"
in the Prometheus server
so if you have an encrypted connection that we also need to
authenticate for.
So in my Prometheus course scripts,
I have a script
called
mutual TLS.sh.
And this mutual TLS.sh will create the certificate that
we need to configure mutual TLS.
Target IP address is the IP address of this new node
that you want to use.
So the,
here's the IP address,
you need to make sure that this matches.
Then what we're going to do is we're going to write and open
SSL config file.
We're going to add this IP address to this config file,
because we need to specify in this
config file this IP address
over this node,
and then we going to create certificates.
We're going to create our certificate,
authority certificate,
so CA.
Put key and not.crt.
And were gon to create a key for the target,
our target is this other machine here.
This target will use Prometheus config file,
that also defines this IP address.
So then we have a certificate that also has the IP address
in it.
And then we going to create a client key and certificate
for Prometheus server.
So our Prometheus server needs a client key and a client
certificate to then authenticate to our target.
So this one is to generate the key and the certificate.
Once we have those certificates,
we can then do a move of this ca.crt to
etc sl.crt Prometheus ca.
So now we have our ca certificate authority that we need,
we can put the key in private,
we going to move the client key in etc Prometheus Prometheus
key,
because Prometheus needs this.
Going to join this, change ownership to Prometheus
so Prometheus can access key.
And I going to move the client certificate to /etc/ssl
/certs/prometheus.crt
And then we're going to add the following lines to
Prometheus.yml.
So let's execute this.
Mutual TLS.sh,
is creating all certificates and then here are the lines
that we need to add to our Prometheus config file.
Node exporter SSL,
we refer to Prometheus ca certificate ,
Prometheus.crt,
and Prometheus.key.
And then we have the target on 443,
and that where our reverse proxy will run on.
So lets already put his in the Prometheus demo,
but not reload yet,
because we still need to configure our reverse proxy.
Make sure the indentation is correct,
job name and this is correct.
So our script already puts these files in place,
and then,
lets go back to our note,
scripts,
we have another script reverse proxy mutual TLS.sh.
And this script needs three files,
target.crt,
target.key,
and the Prometheus certificate authority certificate.
The target host and port is local host,
so this is gonna be a reverse proxy for local host
9100.
On port 9100 we have the
node exporter running.
So this very similar as the previous one,
the previous nginx,
so we install nginx,
and we need to find a server where we're going to say
here's the certificate,
the key,
the client certificate,
and SSL verify client is on.
So to connect to this endpoint,
to connect this reverse proxy,
you will then need a client certificate,
so Prometheus will need this client certificate
to be able to connect on this reverse proxy.
And this client certificate needs to be signed with
this Prometheus ca,
and that what we did in the mutual TLS script.
We signed our Prometheus client certificate with the
Prometheus ca,
and that's how it works.
If you just create a random certificate yourself,
it will not work,
because it wil--
it has not been signed by this Prometheus ca.
And to be able to sign it,
you need this Prometheus ca key that has been generated
and this key,
you need to keep secret.
So then If SSL works,
then location slash will proxy pass to host and port.
Last lines are going to enable nginx and then restart nginx.
So lets execute this,
but lets not forget to get the target.crt,
the key,
and the Prometheus.crt.
So I'm,
copy this one here so that target.crt,
target.key,
and the
etc SSL crt Prometheus ca.crt.
So I just get them,
so now they're all under each other,
and then this is the first one,
target.crt,
target.key,
that's a private key,
and ten the Prometheus ca.crt.
And that is the last one here.
Copy this,
And we can execute this reverse proxy in mutual TLS.
Installs nginx,
configures reverse proxy.
Let's have look whether this works,
curl,
certificate problem,
that's false -k
and we get
"400 no required SSL certificate was sent",
so we know have to do the correct certificate,
we can now not go to this url.
So let's test this on our Prometheus server with curl.
So we gonna copy this,
we still need -k,
but now we going to pass the key and the certificate.
So the key is etc Prometheus Prometheus.key,
and the certificate is etc SSL certs Prometheus.crt.
Those have been generated by our script,
and with that key and that certificate,
we can now or we should now be able to log into this
nginx server,
and yes,
that works.
So now we have the north exporter slash metrics ,
so also Prometheus should be able to reach it.
If you have another look at Prometheus config file,
Prometheus Prometheus.yml,
then we see here as well that we have the certificate and
the key configured,
but also the ca file so that Prometheus can verify
the certificate.
Target is correct,
scheme is https
Let's reload Prometheus,
let's check
journal detail,
loading configuration file.
Looks good.
We are now in here again on our Prometheus server,
status,
targets,
and here we have our node exporter SSL,
and its using this endpoint,
the IP address slash metrics.
And it is up,
we were able to scrape data points.
So even if I would click here now,
I get,
this certificate is not valid.
I do proceed,
and I get "400 Bad Request".
I haven't sent the SSL certificate.
So if you don't have the SSL certificate,
the client certificate,
you can not connect to this endpoint.
The only this that is left is that you need to make sure
that you only allow this port,
port 443,
and not 9100,
so that only this port can be accessed externally and
not other port,
otherwise your reverse proxy can be bypassed.
So you need to block the original ports.
So now only Prometheus can connect to this node exporter.
So that's it for this lecture.
It's a little bit complicated with the certificates,
if you don't really understand exactly what mutual TLS is,
you can also just google it,
it's a very generic concept,
you can use it for everything,
it's not only used in Prometheus.
The end goal really is that you have an endpoint
that is encrypted
and that you can do authentication using your
client certificate.
And that is exactly what I showed you.