Kicking the tires of Prometheus using Docker on Fedora

Straight from the Prometheus documentation: "Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud."

I haven't had a chance to even take a look at using Prometheus.  Here I'll go over the steps I had to follow to get a working local Prometheus install monitoring my local Docker daemon so I could see metrics through the Prometheus dashboard.

First things first, here are the versions of what I am using (eh, until we find out what the problem is (listed below)):

• Fedora 25
• Docker
• docker-1.12.6-6.gitae7d637.fc25.x86_64
• docker-common-1.12.6-6.gitae7d637.fc25.x86_64
• docker-latest-1.12.6-2.git51ef5a8.fc25.x86_64
• Prometheus
• prom/prometheus b0195cb1a666

So, there were a couple of places I went for documentation to get started:

Prometheus

https://prometheus.io/docs/introduction/getting_started/

Docker

https://docs.docker.com/engine/admin/prometheus/

So, following those docs, I tried to use the default Fedora Docker configuration.  That did not work.  The Docker documentation was off, at least for the version of Docker I am using.  By default, in Fedora, you get a Docker package that is a bit out of date.  Here are the steps I took and what I had to do as a workaround.

First, grab the latest version of Prometheus from Docker hub.

$ sudo docker pull prom/prometheus

Start the Prometheus container and test dashboard access.

Create a /tmp/prometheus.yml file with the following contents:

# my global config

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9323']

Run the Docker container

$ sudo docker run -dt -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml docker.io/prom/prometheus

The next thing to do is enable the Docker daemon to expose and endpoint that Prometheus can monitor.  This is where the Docker docs are off a bit.  They suggest that you create a /etc/docker/daemon.json with the following configuration.

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

The issue with that is, Docker doesn't respect that configuration.  So, instead, I chose to use the latest version of Docker in Fedora.  To do this, I went to the Docker website for instructions. Following those got me to these versions:

$ dnf list docker-ce.x86_64  --showduplicates | sort -r

Last metadata expiration check: 0:35:28 ago on Thu Jun  1 16:20:35 2017.

Installed Packages

docker-ce.x86_64               17.05.0.ce-1.fc25                docker-ce-edge 
docker-ce.x86_64               17.05.0.ce-1.fc25                @docker-ce-edge
docker-ce.x86_64               17.05.0.ce-1.fc25                @docker-ce-edge
docker-ce.x86_64               17.04.0.ce-1.fc25                docker-ce-edge 
docker-ce.x86_64               17.03.1.ce-1.fc25                docker-ce-stable
docker-ce.x86_64               17.03.0.ce-1.fc25                docker-ce-stable
Available Packages

Now that I had those, I could go back to the previous documentation from Docker to get started with Prometheus.  That link is here: https://docs.docker.com/engine/admin/prometheus/ and for the most part it worked.  The issue was, I could start the container, but when I tried to hit the Prometheus target of http://localhost:9090/targets, I received the following error:

Get http://localhost:9323/metrics: dial tcp 127.0.0.1:9323: getsockopt: connection refused

It has the "Docker" target in a "Down" state.  Through a bit more investigation, I was able to change 2 things to enable it:

1. Change the /etc/docker/daemon.json from 127.0.0.1:9323 to 0.0.0.0:9323

2. Restart the Prometheus image with --network=host

Now I have the target as up, and a graph with Docker attributes that I can start monitoring.  Check out the graph below.

At this point, I can now start evaluating Prometheus and start learning more about it.

Thanks for hanging in there.  If you have any questions or comments, please do engage below.