“THANOS” — Monitoring with Prometheus and Grafana

Baseline Monitoring Setup with Prometheus and Grafana

In order to setup a baseline monitoring setup in a containerized environment, you need to have a basic understanding of Docker and prometheus. Here I have wrote a blogpost previously where I have setup the stack in an easy and simple way.

Thanos

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.

Why Integrate Prometheus with Thanos?

When practicing monitoring with prometheus, there were certain capabilities and areas that needs upgrade in order to meet requirements such as:

  • Storing historical data in a reliable and cost-efficient way
  • Accessing all metrics, whether they are new or old using a single-query API
  • Merging replicated data collected via Prometheus high-availability (HA) setups

Thanos Components Architecture Overview

Here is the official and complete Architecture diagram for implementation of Thanos with prometheus.

Image from Thanos Github
Block Diagram of Prometheus with Thanos

Deployment Overview

Let’s write a docker-compose file to deploy the complete stack. we will do the following steps;

prometheus0:
image: prom/prometheus:v2.9.2
container_name: prometheus0
user: root
volumes:
- thanos0:/data
- ./data/prom0/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/data/prom0"
- "--web.enable-lifecycle"
- "--storage.tsdb.min-block-duration=2h"
- "--storage.tsdb.max-block-duration=2h"
- "--web.listen-address=0.0.0.0:9090"
networks:
- thanos
sidecar0:
image: thanosio/thanos:v0.10.0
container_name: thanos-sidecar0
command:
- "sidecar"
- "--debug.name=sidecar-0"
- "--grpc-address=0.0.0.0:10901"
- "--http-address=0.0.0.0:10902"
- "--prometheus.url=http://prometheus0:9090"
- "--tsdb.path=/data/prom0"
- "--objstore.config-file=/bucket.yml"
volumes:
- thanos0:/data
- ./data/bucket.yml:/bucket.yml
depends_on:
- prometheus0
networks:
- thanos
query0:
image: thanosio/thanos:v0.10.0
container_name: thanos-query0
command:
- "query"
- "--grpc-address=0.0.0.0:10903"
- "--http-address=0.0.0.0:10904"
- "--query.replica-label=prometheus"
- "--store=sidecar0:10901"
- "--store=store:10905"
ports:
- 10904:10904
depends_on:
- sidecar0
- store
networks:
- thanos
store:
image: thanosio/thanos:v0.10.0
container_name: thanos-store
restart: always
command:
- "store"
- "--grpc-address=0.0.0.0:10905"
- "--http-address=0.0.0.0:10906"
- "--data-dir=/data/store"
- "--objstore.config-file=/bucket.yml"
volumes:
- store:/data
- ./data/bucket.yml:/bucket.yml
networks:
- thanos
compactor:
image: thanosio/thanos:v0.10.0
container_name: compactor
command:
- "compact"
- "--wait"
- "--data-dir=/tmp/thanos-compact"
- "--objstore.config-file=/bucket.yml"
- "--http-address=0.0.0.0:10902"
volumes:
- compact:/tmp
- ./data/bucket.yml:/bucket.yml
depends_on:
- sidecar0
- store
networks:
- thanos
node-exporter:
image: prom/node-exporter:v0.18.1
container_name: node-exporter
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
ports:
- 9100:9100
networks:
- thanos
grafana:
image: grafana/grafana:6.5.2
container_name: grafana
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
ports:
- 3000:3000
networks:
- thanos
networks:
thanos: {}
volumes:
thanos0: {}
store: {}
compact: {}
- DevOps
|---docker-compose.yml
|---data
|---bucket.yml
|---prom0
|---prometheus.yml
type: FILESYSTEM
config:
directory: “/data/prom0”
global:
external_labels:
prometheus: prom-0
scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- "localhost:9090"
- job_name: 'nodeexporter'
scrape_interval: 5s
static_configs:
- targets:
- "node-exporter:9100"
- job_name: thanos-sidecar
scrape_interval: 5s
static_configs:
- targets:
- "sidecar0:10902"
- job_name: thanos-store
scrape_interval: 5s
static_configs:
- targets:
- "store:10906"
- job_name: thanos-query
scrape_interval: 5s
static_configs:
- targets:
- "query0:10904"

Let’s Execute it and See the Results

All the configuration has been now completed firstly we we will create the folder and files structure as shown above, we will run the following command and it will create our whole cluster as shown;

$ docker-compose up -d
Image from Docker Desktop application
Thanos Query UI
Grafaana Data Source Add Option
Adding Thanos Query URL to data source and Save it.
Grafana Query and Dashboard

In A Nutshell

We have seen and test how Thanos is being used along with prometheus and how it can be deployed with a basic docker setup. We have seen how it interacts with Prometheus and the type of advantages it can give us. It is now clear from the concept that how Thanos Sidecar and Storage are inherently advantageous when it comes to scaling, in relation to a typical Prometheus setup. Moreover if we talk about Thanos Query, we can analyze that how a single metric collection point is a good thing for bigger environments. Lastly, downsampling through the use of the Thanos Compactor seems like a performance no brainer. Large datasets can be easily handled using the downsampling method.
Hopefully in the upcoming blogpost, we can deploy it in Kubernetes environment and see how efficiently things work there.

Image Taken from Unsplash

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Syed Saad Ahmed

Syed Saad Ahmed

Python, DevOps, Cryptography, Infra-Structure Automation. https://syedsaadahmed.com/