{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifqeuvyw4dizhhplkrluu3pslymwrrwiq24mwvmyrkgb2dhwurzji",
    "uri": "at://did:plc:46ti67tc37qcmwp2vaynk6fq/app.bsky.feed.post/3mgdp4ubemxb2"
  },
  "path": "/en/blog/2026-prometheus-metrics-discovery-docker-labels",
  "publishedAt": "2026-03-05T21:03:44.748Z",
  "site": "https://vincent.bernat.ch",
  "tags": [
    "Akvorado",
    "Traefik",
    "Docker Compose",
    "Docker labels",
    "Prometheus metrics",
    "Grafana Alloy",
    "Traefik & Docker",
    "Metrics discovery with Alloy",
    "Discovering Docker containers",
    "Relabeling targets",
    "Scraping and forwarding",
    "Built-in exporters",
    "listens for events on the Docker socket",
    "`discovery.docker`",
    "1",
    "2",
    "3",
    "Prometheus exporters",
    "Docker socket proxy",
    "↩︎",
    "\"host.docker.internal:host-gateway\"]` in its definition. [↩︎"
  ],
  "textContent": "Akvorado, a network flow collector, relies on Traefik, a reverse HTTP proxy, to expose HTTP endpoints for services implemented in a Docker Compose setup. Docker labels attached to each service define the routing rules. Traefik picks them up automatically when a container starts. Instead of maintaining a static configuration file to collect Prometheus metrics, we can apply the same approach with Grafana Alloy, making its configuration simpler.\n\n  * Traefik & Docker\n  * Metrics discovery with Alloy\n    * Discovering Docker containers\n    * Relabeling targets\n    * Scraping and forwarding\n  * Built-in exporters\n\n\n\n# Traefik & Docker\n\nTraefik listens for events on the Docker socket. Each service advertises its configuration through labels. For example, here is the Loki service in Akvorado:\n\n\n    services:\n      loki:\n        # …\n        expose:\n          - 3100/tcp\n        labels:\n          - traefik.enable=true\n          - traefik.http.routers.loki.rule=PathPrefix(`/loki`)\n\n\nOnce the container is healthy, Traefik creates a router forwarding requests matching `/loki` to its first exposed port. Colocating Traefik configuration with the service definition is attractive. How do we achieve the same for Prometheus metrics?\n\n# Metrics discovery with Alloy\n\nGrafana Alloy, a metrics collector that can scrape Prometheus endpoints, includes a `discovery.docker` component. Just like Traefik, it connects to the Docker socket.1 With a few relabeling rules, we can teach it to use Docker labels to locate and scrape metrics.\n\nWe define three labels on each service:\n\n  * `metrics.enable` set to `true` enables metrics collection,\n  * `metrics.port` specifies the port exposing the Prometheus endpoint, and\n  * `metrics.path` specifies the path to the metrics endpoint.\n\n\n\nIf there is more than one exposed port, `metrics.port` is mandatory, otherwise it defaults to the only exposed port. The default value for `metrics.path` is `/metrics`. The Loki service from earlier becomes:\n\n\n    services:\n      loki:\n        # …\n        expose:\n          - 3100/tcp\n        labels:\n          - traefik.enable=true\n          - traefik.http.routers.loki.rule=PathPrefix(`/loki`)\n          - metrics.enable=true\n          - metrics.path=/loki/metrics\n\n\nAlloy’s configuration is split into four parts:\n\n  1. **discover** containers through the Docker socket,\n  2. **filter and relabel** targets using Docker labels,\n  3. **scrape** the matching endpoints, and\n  4. **forward** the metrics to Prometheus.\n\n\n\n## Discovering Docker containers\n\nThe first building block discovers running containers:\n\n\n    discovery.docker \"docker\" {\n      host             = \"unix:///var/run/docker.sock\"\n      refresh_interval = \"30s\"\n      filter {\n        name   = \"label\"\n        values = [\"com.docker.compose.project=akvorado\"]\n      }\n    }\n\n\nThis connects to the Docker socket and lists containers every 30 seconds.2 The `filter` block restricts discovery to containers belonging to the `akvorado` project, avoiding interference with unrelated containers on the same host. For each discovered container, Alloy produces a target with labels such as `__meta_docker_container_label_metrics_port` for the `metrics.port` Docker label.\n\n## Relabeling targets\n\nThe relabeling step filters and transforms raw targets from Docker discovery into scrape targets. The first stage keeps only targets with `metrics.enable` set to `true`:\n\n\n    discovery.relabel \"prometheus\" {\n      targets = discovery.docker.docker.targets\n\n      // Keep only targets with metrics.enable=true\n      rule {\n        source_labels = [\"__meta_docker_container_label_metrics_enable\"]\n        regex         = `true`\n        action        = \"keep\"\n      }\n\n      // …\n    }\n\n\nThe second stage overrides the discovered port when we define `metrics.port`:\n\n\n    // When metrics.port is set, override __address__.\n    rule {\n      source_labels = [\"__address__\", \"__meta_docker_container_label_metrics_port\"]\n      regex         = `(.+):\\d+;(.+)`\n      target_label  = \"__address__\"\n      replacement   = \"$1:$2\"\n    }\n\n\nNext, we handle containers in `host` network mode. When `__meta_docker_network_name` equals `host`, the address is rewritten to `host.docker.internal` instead of `localhost`:3\n\n\n    // When host networking, override __address__ to host.docker.internal.\n    rule {\n      source_labels = [\"__meta_docker_container_label_metrics_port\", \"__meta_docker_network_name\"]\n      regex         = `(.+);host`\n      target_label  = \"__address__\"\n      replacement   = \"host.docker.internal:$1\"\n    }\n\n\nThe next stage derives the job name from the service name, stripping any numbered suffix. The instance label is the address without the port:\n\n\n    rule {\n      source_labels = [\"__meta_docker_container_label_com_docker_compose_service\"]\n      regex         = `(.+)(?:-\\d+)?`\n      target_label  = \"job\"\n    }\n    rule {\n      source_labels = [\"__address__\"]\n      regex         = `(.+):\\d+`\n      target_label  = \"instance\"\n    }\n\n\nIf a container defines `metrics.path`, Alloy uses it as a path. Otherwise, it defaults to `/metrics`:\n\n\n    rule {\n      source_labels = [\"__meta_docker_container_label_metrics_path\"]\n      regex         = `(.+)`\n      target_label  = \"__metrics_path__\"\n    }\n    rule {\n      source_labels = [\"__metrics_path__\"]\n      regex         = \"\"\n      target_label  = \"__metrics_path__\"\n      replacement   = \"/metrics\"\n    }\n\n\n## Scraping and forwarding\n\nWith the targets properly relabeled, scraping and forwarding are straightforward:\n\n\n    prometheus.scrape \"docker\" {\n      targets         = discovery.relabel.prometheus.output\n      forward_to      = [prometheus.remote_write.default.receiver]\n      scrape_interval = \"30s\"\n    }\n\n    prometheus.remote_write \"default\" {\n      endpoint {\n        url = \"http://prometheus:9090/api/v1/write\"\n      }\n    }\n\n\n`prometheus.scrape` periodically fetches metrics from the discovered targets. `prometheus.remote_write` sends them to Prometheus.\n\n# Built-in exporters\n\nSome services do not expose a Prometheus endpoint. Redis and Kafka are common examples. Alloy ships built-in Prometheus exporters that query these services and expose metrics on their behalf.\n\n\n    prometheus.exporter.redis \"docker\" {\n      redis_addr = \"redis:6379\"\n    }\n    discovery.relabel \"redis\" {\n      targets = prometheus.exporter.redis.docker.targets\n      rule {\n        target_label = \"job\"\n        replacement  = \"redis\"\n      }\n    }\n    prometheus.scrape \"redis\" {\n      targets         = discovery.relabel.redis.output\n      forward_to      = [prometheus.remote_write.default.receiver]\n      scrape_interval = \"30s\"\n    }\n\n\nThe same pattern applies to Kafka:\n\n\n    prometheus.exporter.kafka \"docker\" {\n      kafka_uris = [\"kafka:9092\"]\n    }\n    discovery.relabel \"kafka\" {\n      targets = prometheus.exporter.kafka.docker.targets\n      rule {\n        target_label = \"job\"\n        replacement  = \"kafka\"\n      }\n    }\n    prometheus.scrape \"kafka\" {\n      targets         = discovery.relabel.kafka.output\n      forward_to      = [prometheus.remote_write.default.receiver]\n      scrape_interval = \"30s\"\n    }\n\n\nEach exporter is a separate component with its own relabeling and scrape configuration. The `job` label is set explicitly since there is no Docker metadata to derive it from.\n\n* * *\n\nWith this setup, adding metrics to a new service with a Prometheus endpoint is a few-label change in `docker-compose.yml`, just like adding a Traefik route. Alloy picks it up automatically. 🩺\n\n* * *\n\n  1. Both Traefik and Alloy require access to the Docker socket, which grants root-level access to the host. A Docker socket proxy mitigates this by exposing only the read-only API endpoints needed for discovery. ↩︎\n\n  2. Unlike Traefik, which watches for events, Grafana Alloy polls the container list at regular intervals—a behavior inherited from Prometheus. ↩︎\n\n  3. The Alloy service needs `extra_hosts: \"host.docker.internal:host-gateway\"]` in its definition. [↩︎\n\n\n",
  "title": "Vincent Bernat: Automatic Prometheus metrics discovery with Docker labels",
  "updatedAt": "2026-03-05T16:40:24.000Z"
}