Differences

This shows you the differences between two versions of the page.

--- сервис_prometheus [2022/03/10 16:04]
val [prometheus-alertmanager]
+++ сервис_prometheus [2026/05/14 15:48] (current)
val [prometheus-apache-exporter]
@@ Line 4: / Line 4: @@
   * [[https://habr.com/ru/company/southbridge/blog/455290/|Полное руководство по Prometheus в 2019 году]]
   * [[https://www.shellhacks.com/ru/prometheus-delete-time-series-metrics/|Prometheus: Удаление Метрик Временных Рядов]]
+  * [[https://habr.com/ru/companies/tochka/articles/683608/|Человеческим языком про метрики]]
+  * [[https://selectel.ru/blog/tutorials/monitoring-in-k8s-with-prometheus/|Мониторинг в K8s с помощью Prometheus]]
 ===== Установка в Debian/Ubuntu =====
@@ Line 28: / Line 30: @@
 ==== Подключение ====
-  * [[http://server.corpX.un:9090/targets]]
+!!! Ссылки содержат DNS имена
+  * [[http://192.168.X.10:9090/]]
+  * [[http://server.corpX.un:9090/]]
   * [[http://server:9090/classic/targets]]
-  * [[http://192.168.X.10:9090/metrics]]
-  * [[http://10.5.N.100+X:9090/graph]]
+<code>
+# /usr/share/prometheus/install-ui.sh
+</code>
 ==== Источники данных ====
@@ Line 38: / Line 45: @@
 ===== prometheus-alertmanager =====
-  * [[https://medium.com/devops-dudes/prometheus-alerting-with-alertmanager-e1bbba8e6a8e|Prometheus Alerting with AlertManager]]
+  * [[https://samber.github.io/awesome-prometheus-alerts/rules|Awesome Prometheus alerts]]
-  * [[https://alex.dzyoba.com/blog/prometheus-alerts/|Prometheus alerts examples]]
+  * [[https://github.com/samber/awesome-prometheus-alerts/tree/master/dist/rules]]
-  * [[https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04|How To Use Alertmanager And Blackbox Exporter To Monitor Your Web Server On Ubuntu 16.04]]
-  * [[https://awesome-prometheus-alerts.grep.to/|Awesome Prometheus alerts]]
   * [[Сервис MTA#Установка и настройка MTA на обработку почты домена hostname]]
+  * [[Сервис adnanh/webhook]]
+==== Настройка маршрутизации уведомлений ====
 <code>
 # apt install prometheus-alertmanager
@@ Line 50: / Line 56: @@
 # cat /etc/prometheus/alertmanager.yml
 </code><code>
-...
 global:
-  smtp_smarthost: 'localhost:25'
+  smtp_smarthost: 'server.corpX.un:25'
-  smtp_from: 'prometheus@server.corpX.un'
+  smtp_from: 'alertmanager@corpX.un'
   smtp_require_tls: false
-#  smtp_auth_username: 'alertmanager'
+templates:
-#  smtp_auth_password: 'password'
+- '/etc/prometheus/alertmanager_templates/*.tmpl'
-...
-  # A default receiver
+route:
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 3h
   receiver: team-X-mails
-...
+#  routes:
+#    - matchers:
+#        - job="roundcube_monitor"
+#      receiver: script-receiver
 receivers:
 - name: 'team-X-mails'
   email_configs:
-#  - to: 'student@corpX.un'
+  - to: 'student@corpX.un'
-#  - to: 'userX@isp.un'
     send_resolved: true
-...
+#  - name: 'script-receiver'
+#    webhook_configs:
+#    - url: 'http://localhost:9000/hooks/my-alert-script'
 </code><code>
+# amtool check-config /etc/prometheus/alertmanager.yml
 # service prometheus-alertmanager restart
+</code>
+==== Тестирование alertmanager ====
+<code>
+$ curl -H "Content-Type: application/json" -d '[
+  {
+    "labels": {
+      "alertname": "TestAlert",
+      "severity": "critical",
+      "instance": "test-host"
+    },
+    "annotations": {
+      "summary": "This is a test alert",
+      "description": "Triggered manually via curl to test Alertmanager routing."
+    }
+  }
+]' http://localhost:9093/api/v2/alerts
+</code>
+==== Подключение готовых правил alertmanager ====
+  * [[Управление сервисами в Linux#Управление юнитами Systemd]] reset-failed
+<code>
+# wget -qO - https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/host-and-hardware/node-exporter.yml | tee /etc/prometheus/node_exporter_rules.yml
+</code>
+==== Создание правил alertmanager ====
+<code>
 # cat /etc/prometheus/first_rules.yml
 </code><code>
@@ Line 94: / Line 139: @@
   - alert: CriticalTraffic
-    expr: rate(if_in_octets{instance="router"}[1m])>125000
+    expr: rate(ifInOctets{instance="router"}[1m])>125000
     for: 1m
     labels:
@@ Line 100: / Line 145: @@
     annotations:
       summary: "CriticalTraffic {{ $labels.instance }}"
-</code><code>
+#  - alert: RoundcubeDown
+#    expr: roundcube_login_logout_runtime == 0
+#    for: 1m
+#    labels:
+#      severity: "critical"
+#    annotations:
+#      summary: "Endpoint {{ $labels.instance }} down"
+#      dashboard_url: "http://server.corpX.un:3000/.../refresh=auto"
+</code>
+==== Подключение Alertmanager к Prometheus ====
+<code>
 # cat /etc/prometheus/prometheus.yml
 </code><code>
@@ Line 114: / Line 171: @@
   - "first_rules.yml"
   # - "second_rules.yml"
+  # - "node_exporter_rules.yml"
 ...
@@ Line 127: / Line 185: @@
 </code>
-  * [[http://192.168.X.10:9090/alerts]]
+  * [[http://server.corpX.un:9090/classic/alerts]]
+==== Отладка Alertmanager ====
+<code>
+# cat /etc/default/prometheus-alertmanager
+ARGS="--log.level=debug"
+</code>
 ===== Exporters =====
 ==== prometheus-node-exporter ====
-  * В Debian ставится как зависимость к пакету prometheus и добавлен в конфигурацию
+  * В Debian/Ubuntu ставится как зависимость к пакету prometheus и добавлен в конфигурацию
   * [[http://server.corpX.un:9100/metrics]]
@@ Line 141: / Line 205: @@
 $ df /
 ...
-/dev/mapper/debian--vg-root  15662008 1877488  12969212  13% /
+/dev/.....  15662008 1877488  12969212  13% /
 ...
 # TYPE node_filesystem_free_bytes gauge
-node_filesystem_free_bytes{device="/dev/mapper/debian--vg-root",fstype="ext4",mountpoint="/"} = (15662008 - 1877488) * 1024
+node_filesystem_free_bytes{fstype="ext4",mountpoint="/"} = (15662008 - 1877488) * 1024
+  или
+{__name__="node_filesystem_free_bytes",fstype="ext4",mountpoint="/"}
 </code>
 == node_network_receive_bytes_total ==
 <code>
-$ cat /sys/class/net/eth1/statistics/rx_bytes
+$ cat /sys/class/net/eth0/statistics/rx_bytes
   или
 $ cat /sys/class/net/bond0/statistics/rx_bytes
 # TYPE node_network_receive_bytes_total counter
-node_network_receive_bytes_total{device="eth1"}
+node_network_receive_bytes_total{device="bond0"}
 </code>
@@ Line 173: / Line 242: @@
 *rate(node_network_receive_bytes_total[1m])
-*rate(node_network_receive_bytes_total{device="eth1"}[1m])
+*rate(node_network_receive_bytes_total{device="bond0"}[1m])
-*rate(node_network_receive_bytes_total{device="eth1",instance="localhost:9100",job="node"}[1m])
+*rate(node_network_receive_bytes_total{device="eth0",instance="localhost:9100",job="node"}[1m])
 </code>
@@ Line 210: / Line 279: @@
     static_configs:
       - targets:
-        - https://google.com
+        - https://val.bmstu.ru
         - https://ya.ru
     relabel_configs:
@@ Line 250: / Line 319: @@
 === Пример использования file-based service discovery и сервиса ping ===
-  * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter|github blackbox_exporter]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]]
+  * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter?tab=readme-ov-file#permissions|github blackbox_exporter permissions]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]]
 <code>
@@ Line 293: / Line 362: @@
   * [[https://blogspot.sysadm.kz/2018/07/grafana-prometheus-cisco-snmp.html|Grafana + Prometheus мониторинг траффика Cisco SNMP]]
   * [[https://medium.com/@openmohan/snmp-monitoring-and-easing-it-with-prometheus-b157c0a42c0c|SNMP monitoring and easing it with Prometheus]]
+  * [[https://grafana.com/blog/2022/02/01/an-advanced-guide-to-network-monitoring-with-grafana-and-prometheus/|An advanced guide to network monitoring with Grafana and Prometheus]]
 <code>
 # apt install prometheus-snmp-exporter
+</code>
+=== Создание файла конфигурации "вручную" ===
+<code>
 # cat /etc/prometheus/snmp.yml
 </code><code>
-#if_mib:   # по умолчанию, позволяет не указывать module в http запросе
+auths:
-snmp_in_out_octets:
+  public_v2:
-  version: 2
-  auth:
     community: public
-  walk:
+    version: 2
-  - 1.3.6.1.2.1.2.2.1.10
+modules:
-  - 1.3.6.1.2.1.2.2.1.16
+  if_mib:
-  - 1.3.6.1.2.1.2.2.1.2
+    walk:
-  metrics:
+    - 1.3.6.1.2.1.2.2.1.10
-  - name: if_in_octets
+    - 1.3.6.1.2.1.2.2.1.16
-    oid: 1.3.6.1.2.1.2.2.1.10
+    - 1.3.6.1.2.1.2.2.1.2
-    type: counter
+    metrics:
-    indexes:
+    - name: ifInOctets
-    - labelname: ifIndex
+      oid: 1.3.6.1.2.1.2.2.1.10
-      type: Integer
+      type: counter
-    lookups:
+      indexes:
-    - labels:
+      - labelname: ifIndex
-      - ifIndex
+        type: Integer
-      labelname: ifDescr
+      lookups:
-      oid: 1.3.6.1.2.1.2.2.1.2
+      - labels:
-      type: DisplayString
+        - ifIndex
-  - name: if_out_octets
+        labelname: ifDescr
-    oid: 1.3.6.1.2.1.2.2.1.16
+        oid: 1.3.6.1.2.1.2.2.1.2
-    type: counter
+        type: DisplayString
-    indexes:
+    - name: ifOutOctets
-    - labelname: ifIndex
+      oid: 1.3.6.1.2.1.2.2.1.16
-      type: Integer
+      type: counter
-    lookups:
+      indexes:
-    - labels:
+      - labelname: ifIndex
-      - ifIndex
+        type: Integer
-      labelname: ifDescr
+      lookups:
-      oid: 1.3.6.1.2.1.2.2.1.2
+      - labels:
-      type: DisplayString
+        - ifIndex
-</code><code>
+        labelname: ifDescr
+        oid: 1.3.6.1.2.1.2.2.1.2
+        type: DisplayString
+</code>
+=== Создание файла конфигурации через "generator" ===
+<code>
+# cp /usr/share/doc/prometheus-snmp-exporter/examples/generator.yml .
+  может понадобиться удалить все modules, кроме if_mib
+# prometheus-snmp-generator generate
+# cp snmp.yml /etc/prometheus/snmp.yml
+</code>
+=== Проверка конфигурации и перезапуск prometheus-snmp-exporter ===
+<code>
+# prometheus-snmp-exporter --dry-run
 # service prometheus-snmp-exporter restart
 </code>
+=== Примеры использования ===
   * [[http://server.corpX.un:9116/]]
 <code>
-# curl --noproxy 127.0.0.1 'http://127.0.0.1:9116/snmp?target=router&module=snmp_in_out_octets'
+# curl 'http://127.0.0.1:9116/snmp?target=router'
 </code><code>
 # cat /etc/prometheus/prometheus.yml
@@ Line 350: / Line 444: @@
         - router
     metrics_path: /snmp
-    params:
-      module: [snmp_in_out_octets]
     relabel_configs:
       - source_labels: [__address__]
@@ Line 364: / Line 456: @@
 <code>
-rate(if_in_octets{ifDescr="FastEthernet1/1",ifIndex="3",instance="router",job="snmp"}[1m])
+rate(ifInOctets{ifDescr="FastEthernet0/0",ifIndex="1",instance="router",job="snmp"}[1m])
   или
-rate(if_out_octets{ifIndex="5",instance="router",job="snmp"}[1m])
+rate(ifOutOctets{ifIndex="5",instance="router",job="snmp"}[1m])
-*rate(if_in_octets{ifDescr="FastEthernet1/1",instance="router"}[1m])
+*rate(ifInOctets{ifDescr="FastEthernet0/0",instance="router"}[1m])
   или
-*rate(if_out_octets{ifDescr="Port-channel1",instance="router"}[1m])
+*rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m])
+</code>
+===== prometheus-apache-exporter =====
+<code>
+root@server:~# apt install prometheus-apache-exporter
+root@server:~# prometheus-apache-exporter --scrape_uri="http://localhost/server-status?auto"
+root@server:~# cat /etc/default/prometheus-apache-exporter
+...
+ARGS=""
+rate(apache_accesses_total[1m])
 </code>
@@ Line 389: / Line 495: @@
   * [[#Проверка конфигурации и перезапуск]]
-==== Пример prometheus pushgateway на bash ====
+==== Отправка метрик из приложения ====
+=== Пример 1 ===
   * [[https://vinayakpandey-7997.medium.com/pushing-bash-script-result-to-prometheus-using-pushgateway-a0760cd261e|Scrape data using Bash script and push it to Prometheus using PushGateway]]
@@ Line 414: / Line 522: @@
 </code><code>
 * * * * * /root/ip_dhcp_binding.sh
+</code>
+=== Пример 2 ===
+  * Web интерфейс к почте [[Web интерфейс к почте#Roundcube]]
+  * [[Средства программирования shell#Тестирование Web приложения на shell]]
+  * [[Технология cgroup]]
+<code>
+root@server:~# cat /usr/local/bin/roundcube_monitor_stat_log.sh
+</code><code>
+#!/bin/bash
+start=$(date +%s.%N)
+/usr/local/bin/roundcube_monitor.sh 2>&1 | logger -t roundcube_monitor
+test ${PIPESTATUS[0]} -eq 0 && end=$(date +%s.%N) || end=$start
+runtime=$(echo "$end - $start" | bc)
+echo "roundcube_login_logout_runtime ${runtime}" | \
+  curl --data-binary @- http://localhost:9091/metrics/job/roundcube_monitor/host/$(hostname)
+echo "Выполнено за $runtime сек."
+</code><code>
+root@server:~# crontab -l
+</code><code>
+* * * * * /usr/local/bin/roundcube_monitor_stat_log.sh >/dev/null
 </code>

Методические материалы Лохтурова Вячеслава

User Tools

Site Tools

Differences

Page Tools