User Tools

Site Tools


сервис_prometheus

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
сервис_prometheus [2024/06/27 16:29]
val [prometheus-blackbox-exporter]
сервис_prometheus [2026/05/14 15:48] (current)
val [prometheus-apache-exporter]
Line 33: Line 33:
  
   * [[http://​192.168.X.10:​9090/​]]   * [[http://​192.168.X.10:​9090/​]]
-  * [[http://​server.corpX.un:​9090/​targets]]+  * [[http://​server.corpX.un:​9090/​]]
   * [[http://​server:​9090/​classic/​targets]]   * [[http://​server:​9090/​classic/​targets]]
 +
 +<​code>​
 +# /​usr/​share/​prometheus/​install-ui.sh
 +</​code>​
  
 ==== Источники данных ==== ==== Источники данных ====
Line 41: Line 45:
 ===== prometheus-alertmanager ===== ===== prometheus-alertmanager =====
  
-  * [[https://medium.com/devops-dudes/​prometheus-alerting-with-alertmanager-e1bbba8e6a8e|Prometheus Alerting with AlertManager]] +  * [[https://samber.github.io/awesome-prometheus-alerts/​rules|Awesome ​Prometheus alerts]] 
-  * [[https://​alex.dzyoba.com/​blog/​prometheus-alerts/​|Prometheus alerts ​examples]] +  * [[https://github.com/samber/​awesome-prometheus-alerts/​tree/​master/​dist/​rules]]
-  * [[https://www.digitalocean.com/community/​tutorials/​how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04|How To Use Alertmanager And Blackbox Exporter To Monitor Your Web Server On Ubuntu 16.04]] +
-  * [[https://​awesome-prometheus-alerts.grep.to/|Awesome Prometheus alerts]]+
  
   * [[Сервис MTA#​Установка и настройка MTA на обработку почты домена hostname]]   * [[Сервис MTA#​Установка и настройка MTA на обработку почты домена hostname]]
 +  * [[Сервис adnanh/​webhook]] 
 +==== Настройка маршрутизации уведомлений ====
 <​code>​ <​code>​
 # apt install prometheus-alertmanager # apt install prometheus-alertmanager
Line 53: Line 56:
 # cat /​etc/​prometheus/​alertmanager.yml # cat /​etc/​prometheus/​alertmanager.yml
 </​code><​code>​ </​code><​code>​
-... 
 global: global:
-  smtp_smarthost:​ 'localhost:25' +  smtp_smarthost:​ 'server.corpX.un:25' 
-  smtp_from: 'prometheus@server.corpX.un'​+  smtp_from: 'alertmanager@corpX.un'​
   smtp_require_tls:​ false   smtp_require_tls:​ false
-#  smtp_auth_username: 'alertmanager+templates: 
- smtp_auth_password'​password'​ +'/​etc/​prometheus/​alertmanager_templates/​*.tmpl
-... + 
-  ​# A default receiver+route: 
 +  group_wait30s 
 +  ​group_interval:​ 5m 
 +  ​repeat_interval:​ 3h
   receiver: team-X-mails   receiver: team-X-mails
-...+ 
 +#  routes: 
 +#    - matchers: 
 +#        - job="​roundcube_monitor"​ 
 +#      receiver: script-receiver 
 receivers: receivers:
 - name: '​team-X-mails'​ - name: '​team-X-mails'​
   email_configs:​   email_configs:​
-#  ​- to: '​student@corpX.un' +  ​- to: '​student@corpX.un'​
-#  - to: '​userX@isp.un'+
     send_resolved:​ true     send_resolved:​ true
-...+ 
 +#  - name: '​script-receiver'​ 
 +#    webhook_configs:​ 
 +#    - url: '​http://​localhost:​9000/​hooks/​my-alert-script'​
 </​code><​code>​ </​code><​code>​
 +# amtool check-config /​etc/​prometheus/​alertmanager.yml
 +
 # service prometheus-alertmanager restart # service prometheus-alertmanager restart
 +</​code>​
  
 +==== Тестирование alertmanager ====
 +<​code>​
 +$ curl -H "​Content-Type:​ application/​json"​ -d '[
 +  {
 +    "​labels":​ {
 +      "​alertname":​ "​TestAlert",​
 +      "​severity":​ "​critical",​
 +      "​instance":​ "​test-host"​
 +    },
 +    "​annotations":​ {
 +      "​summary":​ "This is a test alert",​
 +      "​description":​ "​Triggered manually via curl to test Alertmanager routing."​
 +    }
 +  }
 +]' http://​localhost:​9093/​api/​v2/​alerts
 +</​code>​
 +==== Подключение готовых правил alertmanager ====
 +
 +  * [[Управление сервисами в Linux#​Управление юнитами Systemd]] reset-failed
 +
 +<​code>​
 +# wget -qO - https://​raw.githubusercontent.com/​samber/​awesome-prometheus-alerts/​master/​dist/​rules/​host-and-hardware/​node-exporter.yml | tee /​etc/​prometheus/​node_exporter_rules.yml
 +</​code>​
 +
 +==== Создание правил alertmanager ====
 +
 +<​code>​
 # cat /​etc/​prometheus/​first_rules.yml # cat /​etc/​prometheus/​first_rules.yml
 </​code><​code>​ </​code><​code>​
Line 103: Line 145:
     annotations:​     annotations:​
       summary: "​CriticalTraffic {{ $labels.instance }}"       summary: "​CriticalTraffic {{ $labels.instance }}"
-</​code><​code>​+ 
 +#  - alert: RoundcubeDown 
 +#    expr: roundcube_login_logout_runtime == 0 
 +#    for: 1m 
 +#    labels: 
 +#      severity: "​critical"​ 
 +#    annotations:​ 
 +#      summary: "​Endpoint {{ $labels.instance }} down"​ 
 +#      dashboard_url:​ "​http://​server.corpX.un:​3000/​.../​refresh=auto"​ 
 +</​code>​ 
 + 
 +==== Подключение Alertmanager к Prometheus ==== 
 +<​code>​
 # cat /​etc/​prometheus/​prometheus.yml # cat /​etc/​prometheus/​prometheus.yml
 </​code><​code>​ </​code><​code>​
Line 117: Line 171:
   - "​first_rules.yml"​   - "​first_rules.yml"​
   # - "​second_rules.yml"​   # - "​second_rules.yml"​
 +  # - "​node_exporter_rules.yml"​
  
 ... ...
Line 130: Line 185:
 </​code>​ </​code>​
  
-  * [[http://192.168.X.10:​9090/​alerts]]+  * [[http://server.corpX.un:9090/classic/alerts]] 
 + 
 +==== Отладка Alertmanager ==== 
 +<​code>​ 
 +# cat /​etc/​default/​prometheus-alertmanager 
 +ARGS="​--log.level=debug"​ 
 +</​code>​
 ===== Exporters ===== ===== Exporters =====
  
Line 144: Line 205:
 $ df / $ df /
 ... ...
-/dev/mapper/​debian--vg-root ​ ​15662008 1877488 ​ 12969212 ​ 13% /+/dev/.....  ​15662008 1877488 ​ 12969212 ​ 13% /
 ... ...
 +
 # TYPE node_filesystem_free_bytes gauge # TYPE node_filesystem_free_bytes gauge
-node_filesystem_free_bytes{device="/​dev/​mapper/​debian--vg-root",​fstype="​ext4",​mountpoint="/"​} = (15662008 - 1877488) * 1024 +node_filesystem_free_bytes{fstype="​ext4",​mountpoint="/"​} = (15662008 - 1877488) * 1024  
 + 
 +  или 
 + 
 +{__name__="​node_filesystem_free_bytes",​fstype="​ext4",​mountpoint="/"​}
 </​code>​ </​code>​
  
Line 253: Line 319:
 === Пример использования file-based service discovery и сервиса ping === === Пример использования file-based service discovery и сервиса ping ===
  
-  * [[https://​www.robustperception.io/​icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://​github.com/​prometheus/​blackbox_exporter|github blackbox_exporter]],​ [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#​Переменные ядра]]+  * [[https://​www.robustperception.io/​icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://​github.com/​prometheus/​blackbox_exporter?​tab=readme-ov-file#​permissions|github blackbox_exporter ​permissions]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#​Переменные ядра]]
  
 <​code>​ <​code>​
Line 355: Line 421:
 </​code>​ </​code>​
  
-=== Примеры использования === 
  
 +=== Проверка конфигурации и перезапуск prometheus-snmp-exporter ===
 <​code>​ <​code>​
 +# prometheus-snmp-exporter --dry-run
 +
 # service prometheus-snmp-exporter restart # service prometheus-snmp-exporter restart
 </​code>​ </​code>​
 +
 +=== Примеры использования ===
  
   * [[http://​server.corpX.un:​9116/​]]   * [[http://​server.corpX.un:​9116/​]]
Line 393: Line 463:
   или   или
 8*rate(ifOutOctets{ifDescr="​Port-channel1",​instance="​router"​}[1m]) 8*rate(ifOutOctets{ifDescr="​Port-channel1",​instance="​router"​}[1m])
 +</​code>​
 +
 +===== prometheus-apache-exporter =====
 +
 +<​code>​
 +root@server:​~#​ apt install prometheus-apache-exporter
 +
 +root@server:​~#​ prometheus-apache-exporter --scrape_uri="​http://​localhost/​server-status?​auto"​
 +
 +root@server:​~#​ cat /​etc/​default/​prometheus-apache-exporter
 +...
 +ARGS=""​
 +
 +rate(apache_accesses_total[1m])
 </​code>​ </​code>​
  
Line 411: Line 495:
  
   * [[#​Проверка конфигурации и перезапуск]]   * [[#​Проверка конфигурации и перезапуск]]
-==== Пример ​prometheus pushgateway ​на bash ====+==== Отправка ​метрик из приложения ==== 
 + 
 +=== Пример 1 ===
  
   * [[https://​vinayakpandey-7997.medium.com/​pushing-bash-script-result-to-prometheus-using-pushgateway-a0760cd261e|Scrape data using Bash script and push it to Prometheus using PushGateway]]   * [[https://​vinayakpandey-7997.medium.com/​pushing-bash-script-result-to-prometheus-using-pushgateway-a0760cd261e|Scrape data using Bash script and push it to Prometheus using PushGateway]]
Line 436: Line 522:
 </​code><​code>​ </​code><​code>​
 * * * * * /​root/​ip_dhcp_binding.sh * * * * * /​root/​ip_dhcp_binding.sh
 +</​code>​
 +
 +=== Пример 2 ===
 +
 +  * Web интерфейс к почте [[Web интерфейс к почте#​Roundcube]]
 +  * [[Средства программирования shell#​Тестирование Web приложения на shell]]
 +  * [[Технология cgroup]]
 +
 +<​code>​
 +root@server:​~#​ cat /​usr/​local/​bin/​roundcube_monitor_stat_log.sh
 +</​code><​code>​
 +#!/bin/bash
 +
 +start=$(date +%s.%N)
 +/​usr/​local/​bin/​roundcube_monitor.sh 2>&1 | logger -t roundcube_monitor
 +test ${PIPESTATUS[0]} -eq 0 && end=$(date +%s.%N) || end=$start
 +
 +runtime=$(echo "$end - $start"​ | bc)
 +echo "​roundcube_login_logout_runtime ${runtime}"​ | \
 +  curl --data-binary @- http://​localhost:​9091/​metrics/​job/​roundcube_monitor/​host/​$(hostname)
 +echo "​Выполнено за $runtime сек."​
 +</​code><​code>​
 +root@server:​~#​ crontab -l
 +</​code><​code>​
 +* * * * * /​usr/​local/​bin/​roundcube_monitor_stat_log.sh >/​dev/​null
 </​code>​ </​code>​
сервис_prometheus.1719494967.txt.gz · Last modified: 2024/06/27 16:29 by val