This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
сервис_prometheus [2024/06/27 16:29] val [prometheus-blackbox-exporter] |
сервис_prometheus [2026/05/14 15:48] (current) val [prometheus-apache-exporter] |
||
|---|---|---|---|
| Line 33: | Line 33: | ||
| * [[http://192.168.X.10:9090/]] | * [[http://192.168.X.10:9090/]] | ||
| - | * [[http://server.corpX.un:9090/targets]] | + | * [[http://server.corpX.un:9090/]] |
| * [[http://server:9090/classic/targets]] | * [[http://server:9090/classic/targets]] | ||
| + | |||
| + | <code> | ||
| + | # /usr/share/prometheus/install-ui.sh | ||
| + | </code> | ||
| ==== Источники данных ==== | ==== Источники данных ==== | ||
| Line 41: | Line 45: | ||
| ===== prometheus-alertmanager ===== | ===== prometheus-alertmanager ===== | ||
| - | * [[https://medium.com/devops-dudes/prometheus-alerting-with-alertmanager-e1bbba8e6a8e|Prometheus Alerting with AlertManager]] | + | * [[https://samber.github.io/awesome-prometheus-alerts/rules|Awesome Prometheus alerts]] |
| - | * [[https://alex.dzyoba.com/blog/prometheus-alerts/|Prometheus alerts examples]] | + | * [[https://github.com/samber/awesome-prometheus-alerts/tree/master/dist/rules]] |
| - | * [[https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04|How To Use Alertmanager And Blackbox Exporter To Monitor Your Web Server On Ubuntu 16.04]] | + | |
| - | * [[https://awesome-prometheus-alerts.grep.to/|Awesome Prometheus alerts]] | + | |
| * [[Сервис MTA#Установка и настройка MTA на обработку почты домена hostname]] | * [[Сервис MTA#Установка и настройка MTA на обработку почты домена hostname]] | ||
| + | * [[Сервис adnanh/webhook]] | ||
| + | ==== Настройка маршрутизации уведомлений ==== | ||
| <code> | <code> | ||
| # apt install prometheus-alertmanager | # apt install prometheus-alertmanager | ||
| Line 53: | Line 56: | ||
| # cat /etc/prometheus/alertmanager.yml | # cat /etc/prometheus/alertmanager.yml | ||
| </code><code> | </code><code> | ||
| - | ... | ||
| global: | global: | ||
| - | smtp_smarthost: 'localhost:25' | + | smtp_smarthost: 'server.corpX.un:25' |
| - | smtp_from: 'prometheus@server.corpX.un' | + | smtp_from: 'alertmanager@corpX.un' |
| smtp_require_tls: false | smtp_require_tls: false | ||
| - | # smtp_auth_username: 'alertmanager' | + | templates: |
| - | # smtp_auth_password: 'password' | + | - '/etc/prometheus/alertmanager_templates/*.tmpl' |
| - | ... | + | |
| - | # A default receiver | + | route: |
| + | group_wait: 30s | ||
| + | group_interval: 5m | ||
| + | repeat_interval: 3h | ||
| receiver: team-X-mails | receiver: team-X-mails | ||
| - | ... | + | |
| + | # routes: | ||
| + | # - matchers: | ||
| + | # - job="roundcube_monitor" | ||
| + | # receiver: script-receiver | ||
| receivers: | receivers: | ||
| - name: 'team-X-mails' | - name: 'team-X-mails' | ||
| email_configs: | email_configs: | ||
| - | # - to: 'student@corpX.un' | + | - to: 'student@corpX.un' |
| - | # - to: 'userX@isp.un' | + | |
| send_resolved: true | send_resolved: true | ||
| - | ... | + | |
| + | # - name: 'script-receiver' | ||
| + | # webhook_configs: | ||
| + | # - url: 'http://localhost:9000/hooks/my-alert-script' | ||
| </code><code> | </code><code> | ||
| + | # amtool check-config /etc/prometheus/alertmanager.yml | ||
| + | |||
| # service prometheus-alertmanager restart | # service prometheus-alertmanager restart | ||
| + | </code> | ||
| + | ==== Тестирование alertmanager ==== | ||
| + | <code> | ||
| + | $ curl -H "Content-Type: application/json" -d '[ | ||
| + | { | ||
| + | "labels": { | ||
| + | "alertname": "TestAlert", | ||
| + | "severity": "critical", | ||
| + | "instance": "test-host" | ||
| + | }, | ||
| + | "annotations": { | ||
| + | "summary": "This is a test alert", | ||
| + | "description": "Triggered manually via curl to test Alertmanager routing." | ||
| + | } | ||
| + | } | ||
| + | ]' http://localhost:9093/api/v2/alerts | ||
| + | </code> | ||
| + | ==== Подключение готовых правил alertmanager ==== | ||
| + | |||
| + | * [[Управление сервисами в Linux#Управление юнитами Systemd]] reset-failed | ||
| + | |||
| + | <code> | ||
| + | # wget -qO - https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/host-and-hardware/node-exporter.yml | tee /etc/prometheus/node_exporter_rules.yml | ||
| + | </code> | ||
| + | |||
| + | ==== Создание правил alertmanager ==== | ||
| + | |||
| + | <code> | ||
| # cat /etc/prometheus/first_rules.yml | # cat /etc/prometheus/first_rules.yml | ||
| </code><code> | </code><code> | ||
| Line 103: | Line 145: | ||
| annotations: | annotations: | ||
| summary: "CriticalTraffic {{ $labels.instance }}" | summary: "CriticalTraffic {{ $labels.instance }}" | ||
| - | </code><code> | + | |
| + | # - alert: RoundcubeDown | ||
| + | # expr: roundcube_login_logout_runtime == 0 | ||
| + | # for: 1m | ||
| + | # labels: | ||
| + | # severity: "critical" | ||
| + | # annotations: | ||
| + | # summary: "Endpoint {{ $labels.instance }} down" | ||
| + | # dashboard_url: "http://server.corpX.un:3000/.../refresh=auto" | ||
| + | </code> | ||
| + | |||
| + | ==== Подключение Alertmanager к Prometheus ==== | ||
| + | <code> | ||
| # cat /etc/prometheus/prometheus.yml | # cat /etc/prometheus/prometheus.yml | ||
| </code><code> | </code><code> | ||
| Line 117: | Line 171: | ||
| - "first_rules.yml" | - "first_rules.yml" | ||
| # - "second_rules.yml" | # - "second_rules.yml" | ||
| + | # - "node_exporter_rules.yml" | ||
| ... | ... | ||
| Line 130: | Line 185: | ||
| </code> | </code> | ||
| - | * [[http://192.168.X.10:9090/alerts]] | + | * [[http://server.corpX.un:9090/classic/alerts]] |
| + | |||
| + | ==== Отладка Alertmanager ==== | ||
| + | <code> | ||
| + | # cat /etc/default/prometheus-alertmanager | ||
| + | ARGS="--log.level=debug" | ||
| + | </code> | ||
| ===== Exporters ===== | ===== Exporters ===== | ||
| Line 144: | Line 205: | ||
| $ df / | $ df / | ||
| ... | ... | ||
| - | /dev/mapper/debian--vg-root 15662008 1877488 12969212 13% / | + | /dev/..... 15662008 1877488 12969212 13% / |
| ... | ... | ||
| + | |||
| # TYPE node_filesystem_free_bytes gauge | # TYPE node_filesystem_free_bytes gauge | ||
| - | node_filesystem_free_bytes{device="/dev/mapper/debian--vg-root",fstype="ext4",mountpoint="/"} = (15662008 - 1877488) * 1024 | + | node_filesystem_free_bytes{fstype="ext4",mountpoint="/"} = (15662008 - 1877488) * 1024 |
| + | |||
| + | или | ||
| + | |||
| + | {__name__="node_filesystem_free_bytes",fstype="ext4",mountpoint="/"} | ||
| </code> | </code> | ||
| Line 253: | Line 319: | ||
| === Пример использования file-based service discovery и сервиса ping === | === Пример использования file-based service discovery и сервиса ping === | ||
| - | * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter|github blackbox_exporter]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]] | + | * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter?tab=readme-ov-file#permissions|github blackbox_exporter permissions]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]] |
| <code> | <code> | ||
| Line 355: | Line 421: | ||
| </code> | </code> | ||
| - | === Примеры использования === | ||
| + | === Проверка конфигурации и перезапуск prometheus-snmp-exporter === | ||
| <code> | <code> | ||
| + | # prometheus-snmp-exporter --dry-run | ||
| + | |||
| # service prometheus-snmp-exporter restart | # service prometheus-snmp-exporter restart | ||
| </code> | </code> | ||
| + | |||
| + | === Примеры использования === | ||
| * [[http://server.corpX.un:9116/]] | * [[http://server.corpX.un:9116/]] | ||
| Line 393: | Line 463: | ||
| или | или | ||
| 8*rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m]) | 8*rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m]) | ||
| + | </code> | ||
| + | |||
| + | ===== prometheus-apache-exporter ===== | ||
| + | |||
| + | <code> | ||
| + | root@server:~# apt install prometheus-apache-exporter | ||
| + | |||
| + | root@server:~# prometheus-apache-exporter --scrape_uri="http://localhost/server-status?auto" | ||
| + | |||
| + | root@server:~# cat /etc/default/prometheus-apache-exporter | ||
| + | ... | ||
| + | ARGS="" | ||
| + | |||
| + | rate(apache_accesses_total[1m]) | ||
| </code> | </code> | ||
| Line 411: | Line 495: | ||
| * [[#Проверка конфигурации и перезапуск]] | * [[#Проверка конфигурации и перезапуск]] | ||
| - | ==== Пример prometheus pushgateway на bash ==== | + | ==== Отправка метрик из приложения ==== |
| + | |||
| + | === Пример 1 === | ||
| * [[https://vinayakpandey-7997.medium.com/pushing-bash-script-result-to-prometheus-using-pushgateway-a0760cd261e|Scrape data using Bash script and push it to Prometheus using PushGateway]] | * [[https://vinayakpandey-7997.medium.com/pushing-bash-script-result-to-prometheus-using-pushgateway-a0760cd261e|Scrape data using Bash script and push it to Prometheus using PushGateway]] | ||
| Line 436: | Line 522: | ||
| </code><code> | </code><code> | ||
| * * * * * /root/ip_dhcp_binding.sh | * * * * * /root/ip_dhcp_binding.sh | ||
| + | </code> | ||
| + | |||
| + | === Пример 2 === | ||
| + | |||
| + | * Web интерфейс к почте [[Web интерфейс к почте#Roundcube]] | ||
| + | * [[Средства программирования shell#Тестирование Web приложения на shell]] | ||
| + | * [[Технология cgroup]] | ||
| + | |||
| + | <code> | ||
| + | root@server:~# cat /usr/local/bin/roundcube_monitor_stat_log.sh | ||
| + | </code><code> | ||
| + | #!/bin/bash | ||
| + | |||
| + | start=$(date +%s.%N) | ||
| + | /usr/local/bin/roundcube_monitor.sh 2>&1 | logger -t roundcube_monitor | ||
| + | test ${PIPESTATUS[0]} -eq 0 && end=$(date +%s.%N) || end=$start | ||
| + | |||
| + | runtime=$(echo "$end - $start" | bc) | ||
| + | echo "roundcube_login_logout_runtime ${runtime}" | \ | ||
| + | curl --data-binary @- http://localhost:9091/metrics/job/roundcube_monitor/host/$(hostname) | ||
| + | echo "Выполнено за $runtime сек." | ||
| + | </code><code> | ||
| + | root@server:~# crontab -l | ||
| + | </code><code> | ||
| + | * * * * * /usr/local/bin/roundcube_monitor_stat_log.sh >/dev/null | ||
| </code> | </code> | ||