【技术分享】Prometheus & Grafana 部署

为什么是 Prometheus + Grafana

K8s 时代之前，监控选型百花齐放（Zabbix / Nagios / Cacti / InfluxDB+Telegraf...）。Prometheus 出现后把这件事重新定义了一遍 —— Pull 模型 + 多维度 label + PromQL，几乎成了 Cloud Native 监控的事实标准。Grafana 则是它最佳搭档，只负责一件事：把数据画好看。

两者的边界很清晰：Prometheus 负责采集 + 存储 + 计算，Grafana 负责展现 + 告警 UI。

整体架构

[Exporters] ── scrape ──▶ [Prometheus] ──▶ [Grafana] ──▶ 浏览器
                              │
                              ▼
                        [Alertmanager] ──▶ 告警通道（钉钉/邮件/PagerDuty）

最小生产级部署需要 4 类组件：

准备工作

下面以一台 Ubuntu 22.04 LTS 为例，所有组件跑在 Docker 里。

一、目录结构

mkdir -p /opt/monitoring/{prometheus/config,prometheus/rules,alertmanager/config,grafana/provisioning,grafana/dashboards,data/prometheus,data/grafana,data/alertmanager}
cd /opt/monitoring

# 数据目录权限 —— 容器里跑的 uid 各不相同
sudo chown -R 65534:65534 data/prometheus       # nobody:nogroup
sudo chown -R 472:472     data/grafana          # grafana
sudo chown -R 65534:65534 data/alertmanager

二、Prometheus 配置

prometheus/config/prometheus.yml：

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod
    replica: 0

rule_files:
  - /etc/prometheus/rules/*.rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  # Prometheus 自监控
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  # 主机指标
  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']

  # 黑盒探测（HTTP 健康检查）
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://yantao.wiki
          - https://example.com/healthz
    relabel_configs:
      - source_labels: [__address__]
        target_label:  __param_target
      - source_labels: [__param_target]
        target_label:  instance
      - target_label:  __address__
        replacement:   blackbox-exporter:9115

三、告警规则

prometheus/rules/node.rules.yml：

groups:
  - name: node
    rules:
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "节点 {{ $labels.instance }} 失联超过 2 分钟"
          description: "Prometheus 抓取 {{ $labels.job }}/{{ $labels.instance }} 失败"

      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} CPU 持续超过 85%"

      - alert: DiskFull
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"}
           / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} 根分区剩余 < 10%"

四、Alertmanager 配置

alertmanager/config/alertmanager.yml：

global:
  resolve_timeout: 5m

route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = critical
      receiver: critical-pager
      continue: true       # 让 critical 也走 default 一份(双通道)

receivers:
  - name: default
    webhook_configs:
      - url: https://oapi.dingtalk.com/robot/send?access_token=YOUR_DD_TOKEN

  - name: critical-pager
    webhook_configs:
      - url: https://oapi.dingtalk.com/robot/send?access_token=YOUR_DD_TOKEN
    email_configs:
      - to: oncall@example.com
        smarthost: smtp.example.com:587
        auth_username: alert@example.com
        auth_password: '<smtp-pass>'
        from: alert@example.com
        require_tls: true

钉钉 / 飞书 / Slack 都用 webhook 接入，但 Alertmanager 默认 payload 格式不是它们的。中间通常加一个转换器，常用的是 prometheus-webhook-dingtalk。

五、Grafana 数据源 & 仪表盘自动导入

Grafana 推荐用 provisioning 把数据源和仪表盘做成代码，避免点点点。

grafana/provisioning/datasources/prometheus.yml：

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s
      httpMethod: POST

grafana/provisioning/dashboards/default.yml：

apiVersion: 1
providers:
  - name: default
    folder: ''
    type: file
    options:
      path: /var/lib/grafana/dashboards

把社区精品仪表盘的 JSON 下载到 grafana/dashboards/：

Grafana 启动时会自动加载，重启容器即生效。

六、Docker Compose 编排

docker-compose.yml：

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.55.0
    container_name: prometheus
    restart: unless-stopped
    user: '65534:65534'
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
      - --web.enable-lifecycle              # POST /-/reload 热加载
      - --web.external-url=https://prom.yantao.wiki
    volumes:
      - ./prometheus/config:/etc/prometheus
      - ./prometheus/rules:/etc/prometheus/rules
      - ./data/prometheus:/prometheus
    networks: [monitoring]

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    user: '65534:65534'
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
      - --web.external-url=https://am.yantao.wiki
    volumes:
      - ./alertmanager/config:/etc/alertmanager
      - ./data/alertmanager:/alertmanager
    networks: [monitoring]

  grafana:
    image: grafana/grafana:11.3.0
    container_name: grafana
    restart: unless-stopped
    user: '472:472'
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=changeme-now
      - GF_SERVER_ROOT_URL=https://grafana.yantao.wiki
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
      - ./data/grafana:/var/lib/grafana
    networks: [monitoring]

  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: unless-stopped
    pid: host
    network_mode: host
    volumes:
      - /:/host:ro,rslave
    command:
      - --path.rootfs=/host
      - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($|/)

  blackbox-exporter:
    image: prom/blackbox-exporter:v0.25.0
    container_name: blackbox-exporter
    restart: unless-stopped
    networks: [monitoring]

networks:
  monitoring:
    name: monitoring

启动：

docker compose up -d
docker compose ps

七、首次访问验证

curl -XPOST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"critical"},"annotations":{"summary":"测试"}}]'

八、生产化清单

部署能跑只是第一步，下面是上生产前必补的 6 层：

九、常见踩坑

docker exec prometheus wget -qO- http://node-exporter:9100/metrics | head

url: http://prometheus:9090   # ✅
url: http://localhost:9090    # ❌ 在容器里指向自己
url: http://127.0.0.1:9090    # ❌ 同上

内存(GB) ≈ active_series × 3.5 KB / 1024 / 1024

十、写在最后

下一步可以扩展的方向：

• Loki + Promtail —— 日志聚合，和 metrics 在 Grafana 里跨源关联
• Tempo —— 分布式追踪，OpenTelemetry-native
• Pyroscope —— 持续性能剖析（Continuous Profiling）
• Mimir / Thanos —— 多租户 + 长期存储

完整 LGTM Stack（Loki + Grafana + Tempo + Mimir）把 Logs / Metrics / Traces / Profiles 全部串起来，下一篇细讲。