> ## Content Index
> Fetch the complete content index at: https://blog.sudowl.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# A Prometheus + Grafana Monitoring Stack in Docker
- URL: https://blog.sudowl.com/prometheus-grafana-monitoring-stack-docker/
- Published: 2026-06-29T10:00:00.000Z
- Updated: 2026-10-11T10:16:27.000Z
- Description: Stand up Prometheus, Grafana and node_exporter with Docker Compose — real scrape config, node metrics, a dashboard, and alert rules that actually fire.
- Author: Lukas
- Tags: Docker, Linux

Uptime Kuma tells me *if* a box is down. Netdata tells me what's happening *right now* on one host. Neither answers "what did CPU look like last Tuesday at 3am when the job OOM'd?" For that I want time-series with retention, and that's Prometheus scraping metrics into a database with Grafana drawing the graphs. It's more moving parts, so I only run it where I actually need history — but where I do, it's the thing I open first.

**TL;DR** — three containers, one compose file:

```yaml
# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports: ["9090:9090"]
    restart: unless-stopped
  node-exporter:
    image: prom/node-exporter:latest
    pid: host
    volumes: ["/:/host:ro,rslave"]
    command: ["--path.rootfs=/host"]
    restart: unless-stopped
  grafana:
    image: grafana/grafana:latest
    volumes: [grafana-data:/var/lib/grafana]
    ports: ["3000:3000"]
    restart: unless-stopped
volumes: { prom-data: {}, grafana-data: {} }

```

Details below. New to compose? Start with [/docker-install-ubuntu/](https://blog.sudowl.com/docker-install-ubuntu/).

## What each piece does

- **Prometheus** is the brain. It *pulls* metrics from targets over HTTP every N seconds and stores them as a time series. Pull, not push — you tell Prometheus where to scrape, not the other way round.
- **node\_exporter** is a tiny agent that exposes host metrics (CPU, RAM, disk, network) at `:9100/metrics`. One per machine you want to watch.
- **Grafana** is the dashboards. It queries Prometheus with PromQL and draws the graphs. It stores nothing about your metrics itself, just how to display them.

The mental model that took me too long to get: Grafana is not the database. Prometheus is. Grafana is a nice window onto it.

## The scrape config

Prometheus does nothing until you tell it what to scrape. This is the whole config for a single host plus itself:

```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          instance: web01

```

Because everything's on the same compose network, `node-exporter:9100` resolves by service name — no IPs. Adding a second server later is one more target line pointing at its `host:9100` (open 9100 only to the Prometheus box, never the world).

```bash
docker compose up -d
curl -s localhost:9090/-/ready     # Prometheus healthy

```

Open `http://server:9090/targets` — both jobs should be green `UP`. If `node` is red, it's almost always the exporter not reachable on the network, not Prometheus.

## Wire up Grafana

Log in at `:3000` (admin/admin, it forces a change). Add the data source **once**:

- Connections → Data sources → Prometheus
- URL: `http://prometheus:9090` (service name again, not localhost — Grafana is in a container)
- Save & test → green.

Then don't build a node dashboard by hand. Import the community one: Dashboards → Import → ID **1860** (Node Exporter Full). Pick your Prometheus source, and you've got CPU, memory, disk I/O, network and filesystem panels in about ten seconds. I've never found a reason to redraw that one from scratch.

## Alerts that actually fire

Graphs you have to look at are useless at 3am. Prometheus evaluates alert rules and flips them to `firing`; a separate Alertmanager routes them to email/Slack/Telegram. Start with the rules — routing can come later:

```yaml
# alerts.yml
groups:
  - name: host
    rules:
      - alert: DiskWillFill
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Root disk under 10% on {{ $labels.instance }}"

      - alert: HostDown
        expr: up{job="node"} == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "node_exporter unreachable on {{ $labels.instance }}"

```

The `for: 10m` matters — it means "only fire if this stays true for 10 minutes", which kills the flapping alerts that train you to ignore the channel. Reload with `curl -X POST localhost:9090/-/reload` (needs `--web.enable-lifecycle` in the command) or just restart the container. Watch them at `:9090/alerts`.

## Retention and disk

Prometheus keeps 15 days by default and it *will* grow. Set retention explicitly on the command so you know what you signed up for:

```yaml
  prometheus:
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

```

A single node at 15s scrape is a few hundred MB a month — trivial. Scraping twenty hosts with high-cardinality app metrics is a different conversation, and that's when you learn what "cardinality" means the hard way.

## Gotchas

- **localhost inside a container is the container.** The single most common mistake: pointing Grafana at `http://localhost:9090`. That's Grafana's own localhost. Use the service name.
- **node\_exporter needs the host, not the container.** Without `pid: host` and the rootfs mount it happily reports the *container's* metrics — near-empty and useless. The `command`/volume in the TL;DR is what makes it see the real host.
- **Exposing 9090 and 3000 to the internet.** Prometheus has no auth at all. Put both behind a reverse proxy with a login, or bind them to localhost and tunnel. See [/nginx-reverse-proxy-setup/](https://blog.sudowl.com/nginx-reverse-proxy-setup/).
- **Scrape interval vs. rule interval.** A `for: 2m` on a 15s scrape is fine. A `for: 30s` with a 60s scrape can never satisfy the window — the rule sees one sample and resets. Keep `for` a comfortable multiple of the interval.
- **`latest` tags.** Fine for a homelab, a footgun in prod. Pin versions once it matters so an unattended `pull` doesn't change your storage format under you.

## Where this fits

This is the heavyweight of my monitoring setup. For a plain "is it up, ping me if not" I still reach for [/uptime-kuma-self-hosted-monitoring/](https://blog.sudowl.com/uptime-kuma-self-hosted-monitoring/), and for a live per-host glance without any storage story [/netdata-monitoring-ubuntu/](https://blog.sudowl.com/netdata-monitoring-ubuntu/) is one command. Prometheus is what you add when you need to answer questions about the *past*. Run the containers under systemd if you want them treated as real services: [/systemd-service-unit-file/](https://blog.sudowl.com/systemd-service-unit-file/).

---

### Related posts

- [Self-Hosted Uptime Monitoring with Uptime Kuma](https://blog.diengdoh.com/uptime-kuma-self-hosted-monitoring/?ref=blog.sudowl.com)
- [Apache as a Reverse Proxy for Docker Apps (and the Panel Trap)](https://blog.diengdoh.com/apache-reverse-proxy-docker-apps/?ref=blog.sudowl.com)
- [Rebuilding My Ghost Blog After the Database Was Gone](https://blog.diengdoh.com/ghost-blog-rebuild-after-database-loss/?ref=blog.sudowl.com)