Blackbox 网络监控

Blackbox Exporter 是什么

参考链接：Github Blackbox Exporter

‍

顾名思义，像 node_exporter 或mysql_exporter等，可以直接从内部获取到资源运行状态指标，此类可称为白盒。

而黑盒则是模拟用户身份探测服务的外部可见性。

‍

常用的模块包含 TCP，HTTP，DNS，ICMP等。

‍

部署

如需使用 ICMP 模块作为探针，需预先修改内核参数：

# /etc/sysctl.conf
net.ipv4.ping_group_range=0 2147483647
net.ipv4.ip_forward = 1

# sysctl -p

‍

配置文件 yaml 示例：

 # cat /etc/blackbox_exporter.yml
modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"
      source_ip_address: "192.168.2.10"

  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ip4"
      follow_redirects: false
      tls_config:
        insecure_skip_verify: true

  http_post_2xx:
    prober: http
    http:
      method: POST

  tcp_connect: 
    prober: tcp
    tcp: 
      ip_protocol_fallback: false
      preferred_ip_protocol: ip4
      tls: true
      tls_config: 
        insecure_skip_verify: true
    timeout: 15s

  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false

  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"

  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"

‍

prometheus 配置示例：

- job_name: 'blackbox_http'
  scrape_interval: 60s
  metrics_path: /probe
  params:
    module: [http_2xx]
  file_sd_configs:
    - files:
      - '/etc/prometheus/file_sd/blackbox_http.yml'
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9119

- job_name: 'blackbox_ping'
  scrape_interval: 60s
  metrics_path: /probe
  params:
    module: [icmp]
  file_sd_configs:
    - files:
        - '/etc/prometheus/file_sd/blackbox_ping.yml'
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9119

- job_name: 'blackbox_tcp'
  scrape_interval: 30s
  metrics_path: /probe
  params:
    module: [tcp_connect]
  file_sd_configs:
    - files:
      - '/etc/prometheus/file_sd/blackbox_tcp.yml'
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9119

- job_name: 'blackbox_tls'
  scrape_interval: 30s
  metrics_path: /probe
  params:
    module: [http_2xx]
  file_sd_configs:
    - files:
      - '/etc/prometheus/file_sd/blackbox_tls.yml'
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9119

‍

systemd 单元文件示例：

# cat /etc/systemd/system/blackbox_exporter.service 
[Unit]
Description=Blackbox Exporter
After=network-online.target

[Service]
Type=simple
PIDFile=/var/run/blackbox_exporter.pid
PermissionsStartOnly=true
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter.yml \
  --web.listen-address=0.0.0.0:9119

SyslogIdentifier=blackbox_exporter
KillMode=process
Restart=always
RestartSec=5


NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true

ProtectSystem=full

[Install]
WantedBy=multi-user.target

‍

常用PromQL

• ICMP metrics：指定时间间隔内的 ping 成功率

  sum_over_time(probe_success{job="blackbox_ping"}[$time_range]) / count_over_time(probe_success{job="blackbox_ping"}[$time_range])
  

‍

• ICMP metrics：ping 延时

  probe_icmp_duration_seconds{phase="rtt",job="blackbox_ping"}
  

‍

• ICMP metrics：ping 实时丢包率

  1 - avg_over_time(probe_success{job="blackbox_ping"}[60s])
  

‍

• HTTP metrics：连通性

  probe_success{job="blackbox_http"} - 0
  

‍

• HTTP metrics：返回状态码

  probe_http_status_code{job="blackbox_http"}
  

‍

• HTTP metrics：响应时间

  probe_duration_seconds{job="blackbox_http"}
  

‍

• HTTP metrics：指定时间间隔内的成功率

  sum_over_time(probe_success{job="blackbox_http"}[$time_range]) / count_over_time(probe_success{job="blackbox_http"}[$time_range])
  

‍

Grafana 绘图

​​

‍

总结

如果聚焦更细致的 ping 测试结果，可以尝试使用 smokeping。参考链接：github.com/oetiker/SmokePing

对比 blackbox，它能够获取到ping的连续结果，就类似我们在命令行终端直接使用ping命令。

而 blackbox 受限于 scrape_interval，只能够抓取有间隔的时间点数据。

‍

引用链接

• Github Blackbox Exporter
• github.com/oetiker/SmokePing