Prometheus uses its own storage engine.
So, in this lecture, I'll explain a little bit more
about how this storage engine works.
You can use the default local on-disc storage,
or optionally the remote storage system.
Local storage is a local time series database
in a custom Prometheus format.
And with remote storage, you can read and write samples
to a remote system in a standardized format.
Currently, it uses a snappy-compressed protocol
buffer encoding over HTTP,
but this might change in the future,
to, for example, use gRPC or HTTP/2.
Remote storage is primarily focused
on the long term storage.
Currently, there are adapters available
for the following solutions:
For example, you can use remote storage for backup,
or to get all your data in a different system.
For example, PostgreSQL or an InfluxDB database.
This is for remote storage.
The next part of the lecture
we'll only talk about local storage.
Like, how is data in Prometheus stored?
Prometheus, starting from version two,
uses a new storage engine
which dramatically increases the scalability.
Ingested samples are grouped in blocks of two hours.
These two hour samples are stored in separate directories.
And these directories
are stored in a data directory of Prometheus.
Writes are batched and written to disc in chunks,
containing multiple data points.
So, if you would take a look
in the data directory of Prometheus,
you would find multiple directories.
So, here are three directories,
which each contain two hours of data,
and this does contain the chunks that have been written.
So, the writes are batched and are written in those chunks.
Every directory also has an index file, called index,
and a metadata file, called meta.json.
It stores the metric names and the labels,
and provides an index from the metric names and labels
to the series in the chunk files.
So, you have your samples on one side, your data,
but you also have the metric names and the labels,
and those also need to be stored,
and you also need an index
to provide the link between those names and labels,
to the data in the chunk files.
So here's, and again, our three directories.
So we have your chunks,
and next to it you also will see the meta.json
and an index file.
The most recent data is kept in memory.
You don't want to lose the in-memory data during a crash,
so the data also needs to be persisted to disc.
This is done using a write-ahead-log.
A WAL file.
So, here again are our three directories.
And you'll see we have,
next to the chunks, the metadata,
and the index, also WAL files.
So this data is kept in memory.
The recent data.
If the Prometheus crashes, you don't want to lose the data.
So the data is written in WAL files first.
Let's talk about the write ahead log, the WAL file.
It's quicker to append to a file, like a log file,
than making multiple random reads and writes.
If there's a server crash, and the data from memory is lost,
then the WAL will be replayed.
This way, no data will be lost or corrupted during a crash.
When series gets deleted, a tombstone file gets created.
This is more efficient than immediately deleting
data from the chunks files,
as the actual delete can happen then, a later point in time.
For example, when there's not a lot of load.
So, here, we then have our three directories,
and we can also have a tombstone,
that indicates that there is deleted data.
The initial two hour blocks are merged in the background
to form longer blocks.
This is called compaction.
So, for example, we have directory one,
which is two hours of data,
and then we have directory two and three,
which already got compacted at some point in time,
and now includes the data from directory two and three.
So, you take your two hours time windows,
and you put them together,
and then you have a four hour time window of data.
Let's go over the block characteristics.
So, a block on the filesystem is a directory with chunks.
You can see each block as a fully independent database
containing all time series for that window.
So, in my example, the windows are two hours.
In the future, that might even change.
Every block of data, except the current block, is immutable.
No changes can be made.
These non-overlapping blocks
are actually a horizontal partitioning
of the ingested time series data.
So, your data is horizontally partitioned.
This horizontal partitioning gives a lot of benefits.
When querying,
the blocks not in the time range can be skipped,
because you know exactly what time ranges
are in what blocks.
So, if the query doesn't go into that time range,
you can just skip it.
When completing a block, data only needs to be added,
and not modified, so it avoids write-amplification,
where you would have to do a lot of writes
to just complete a block.
In this case, you write only at the end.
So, data only needs to be added.
So, you don't need to rewrite anything else anymore.
Recent data is kept in memory, so it can be queried quicker,
And to make sure that we don't lose it,
we have this WAL file.
Deleting all data is only a matter of deleting
directories on the filesystem.
That means that nothing needs to be
rewritten to delete data.
So, deletes are very quick.
Compaction then.
When querying, blocks have to be merged together
to be able to calculate results.
Too many blocks could cause too much merging overhead,
so blocks are compacted.
Two blocks are merged and form a newly created,
often larger, block.
Compaction can also modify data.
Dropping deleted data,
or restructuring the chunks
to increase the query performance.
Then let's talk about the index.
Having horizontal partitioning
already makes most queries quicker,
but not those that need to go through
all the data to get results.
Because then you would have to query all your partitions.
The index is an inverted index
to provide better query performance,
also in cases where all data needs to be queried.
Each series is assigned a unique ID.
For example, ID one, two, and three.
And the index will contain an inverted index for the labels.
For example, from label env=production,
it'll have one and three as IDs
if those series contain the label env=production.
So, let's say you have three data points,
and only two of those data points
are labeled env=production.
Well, you don't have to query all three,
and using this inverted index,
you know that you only have to look at ID one and three,
and not at ID two to get the data for env=production.
So, what about disc size?
On average, Prometheus needs one to two bytes per sample.
You can use the following formula
to calculate the space needed.
So, here's a formula that you then can use to know
how much disc space you need on your Prometheus server.
How to, then, reduce disc size?
Well, you can increase the scrape interval,
which gets you less data over time.
Or, you can decrease the targets or series you scrape.
Or you can also reduce the retention.
How long you keep the data.
So, for example, here's a storage.tdsb.retention flag
that you can use when Prometheus starts up.
And this determines when to remove all data,
which defaults to 15 days in the version that I'm using,
but you can change it.
To read the full story of Prometheus time series database,
read the blog post from Fabian Reinartz,
at this URL.
So, he explains, in this post, how he went from
one of the first versions of this time series database
to the version that I explained in this lecture.
It's definitely a good read
if you want to understand in more detail
how the storage engine works.