User Tools
сервис_prometheus
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
сервис_prometheus [2022/03/10 16:04] val [prometheus-alertmanager] |
сервис_prometheus [2025/04/18 09:21] (current) val [prometheus-alertmanager] |
||
|---|---|---|---|
| Line 4: | Line 4: | ||
| * [[https://habr.com/ru/company/southbridge/blog/455290/|Полное руководство по Prometheus в 2019 году]] | * [[https://habr.com/ru/company/southbridge/blog/455290/|Полное руководство по Prometheus в 2019 году]] | ||
| * [[https://www.shellhacks.com/ru/prometheus-delete-time-series-metrics/|Prometheus: Удаление Метрик Временных Рядов]] | * [[https://www.shellhacks.com/ru/prometheus-delete-time-series-metrics/|Prometheus: Удаление Метрик Временных Рядов]] | ||
| + | * [[https://habr.com/ru/companies/tochka/articles/683608/|Человеческим языком про метрики]] | ||
| + | * [[https://selectel.ru/blog/tutorials/monitoring-in-k8s-with-prometheus/|Мониторинг в K8s с помощью Prometheus]] | ||
| ===== Установка в Debian/Ubuntu ===== | ===== Установка в Debian/Ubuntu ===== | ||
| Line 28: | Line 30: | ||
| ==== Подключение ==== | ==== Подключение ==== | ||
| - | * [[http://server.corpX.un:9090/targets]] | + | !!! Ссылки содержат DNS имена |
| + | |||
| + | * [[http://192.168.X.10:9090/]] | ||
| + | * [[http://server.corpX.un:9090/]] | ||
| * [[http://server:9090/classic/targets]] | * [[http://server:9090/classic/targets]] | ||
| - | * [[http://192.168.X.10:9090/metrics]] | ||
| - | * [[http://10.5.N.100+X:9090/graph]] | ||
| ==== Источники данных ==== | ==== Источники данных ==== | ||
| Line 38: | Line 41: | ||
| ===== prometheus-alertmanager ===== | ===== prometheus-alertmanager ===== | ||
| - | * [[https://medium.com/devops-dudes/prometheus-alerting-with-alertmanager-e1bbba8e6a8e|Prometheus Alerting with AlertManager]] | + | * [[https://samber.github.io/awesome-prometheus-alerts/rules|Awesome Prometheus alerts]] |
| - | * [[https://alex.dzyoba.com/blog/prometheus-alerts/|Prometheus alerts examples]] | + | * [[https://github.com/samber/awesome-prometheus-alerts/tree/master/dist/rules]] |
| - | * [[https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04|How To Use Alertmanager And Blackbox Exporter To Monitor Your Web Server On Ubuntu 16.04]] | + | |
| - | * [[https://awesome-prometheus-alerts.grep.to/|Awesome Prometheus alerts]] | + | |
| * [[Сервис MTA#Установка и настройка MTA на обработку почты домена hostname]] | * [[Сервис MTA#Установка и настройка MTA на обработку почты домена hostname]] | ||
| Line 50: | Line 51: | ||
| # cat /etc/prometheus/alertmanager.yml | # cat /etc/prometheus/alertmanager.yml | ||
| </code><code> | </code><code> | ||
| - | ... | ||
| global: | global: | ||
| - | smtp_smarthost: 'localhost:25' | + | smtp_smarthost: 'server.corpX.un:25' |
| - | smtp_from: 'prometheus@server.corpX.un' | + | smtp_from: 'alertmanager@corpX.un' |
| smtp_require_tls: false | smtp_require_tls: false | ||
| - | # smtp_auth_username: 'alertmanager' | + | templates: |
| - | # smtp_auth_password: 'password' | + | - '/etc/prometheus/alertmanager_templates/*.tmpl' |
| - | ... | + | |
| - | # A default receiver | + | route: |
| + | group_wait: 30s | ||
| + | group_interval: 5m | ||
| + | repeat_interval: 3h | ||
| receiver: team-X-mails | receiver: team-X-mails | ||
| - | ... | + | |
| receivers: | receivers: | ||
| - name: 'team-X-mails' | - name: 'team-X-mails' | ||
| email_configs: | email_configs: | ||
| - | # - to: 'student@corpX.un' | + | - to: 'student@corpX.un' |
| - | # - to: 'userX@isp.un' | + | |
| send_resolved: true | send_resolved: true | ||
| - | ... | ||
| </code><code> | </code><code> | ||
| # service prometheus-alertmanager restart | # service prometheus-alertmanager restart | ||
| Line 94: | Line 95: | ||
| - alert: CriticalTraffic | - alert: CriticalTraffic | ||
| - | expr: rate(if_in_octets{instance="router"}[1m])>125000 | + | expr: rate(ifInOctets{instance="router"}[1m])>125000 |
| for: 1m | for: 1m | ||
| labels: | labels: | ||
| Line 127: | Line 128: | ||
| </code> | </code> | ||
| - | * [[http://192.168.X.10:9090/alerts]] | + | * [[http://server.corpX.un:9090/classic/alerts]] |
| ===== Exporters ===== | ===== Exporters ===== | ||
| ==== prometheus-node-exporter ==== | ==== prometheus-node-exporter ==== | ||
| - | * В Debian ставится как зависимость к пакету prometheus и добавлен в конфигурацию | + | * В Debian/Ubuntu ставится как зависимость к пакету prometheus и добавлен в конфигурацию |
| * [[http://server.corpX.un:9100/metrics]] | * [[http://server.corpX.un:9100/metrics]] | ||
| Line 149: | Line 150: | ||
| == node_network_receive_bytes_total == | == node_network_receive_bytes_total == | ||
| <code> | <code> | ||
| - | $ cat /sys/class/net/eth1/statistics/rx_bytes | + | $ cat /sys/class/net/eth0/statistics/rx_bytes |
| или | или | ||
| $ cat /sys/class/net/bond0/statistics/rx_bytes | $ cat /sys/class/net/bond0/statistics/rx_bytes | ||
| # TYPE node_network_receive_bytes_total counter | # TYPE node_network_receive_bytes_total counter | ||
| - | node_network_receive_bytes_total{device="eth1"} | + | node_network_receive_bytes_total{device="bond0"} |
| </code> | </code> | ||
| Line 173: | Line 174: | ||
| 8*rate(node_network_receive_bytes_total[1m]) | 8*rate(node_network_receive_bytes_total[1m]) | ||
| - | 8*rate(node_network_receive_bytes_total{device="eth1"}[1m]) | + | 8*rate(node_network_receive_bytes_total{device="bond0"}[1m]) |
| - | 8*rate(node_network_receive_bytes_total{device="eth1",instance="localhost:9100",job="node"}[1m]) | + | 8*rate(node_network_receive_bytes_total{device="eth0",instance="localhost:9100",job="node"}[1m]) |
| </code> | </code> | ||
| Line 210: | Line 211: | ||
| static_configs: | static_configs: | ||
| - targets: | - targets: | ||
| - | - https://google.com | + | - https://val.bmstu.ru |
| - https://ya.ru | - https://ya.ru | ||
| relabel_configs: | relabel_configs: | ||
| Line 250: | Line 251: | ||
| === Пример использования file-based service discovery и сервиса ping === | === Пример использования file-based service discovery и сервиса ping === | ||
| - | * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter|github blackbox_exporter]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]] | + | * [[https://www.robustperception.io/icmp-pings-with-the-blackbox-exporter|ICMP Pings with the Blackbox exporter]], [[https://github.com/prometheus/blackbox_exporter?tab=readme-ov-file#permissions|github blackbox_exporter permissions]], [[POSIX capabilities]] или [[Управление ядром и модулями в Linux#Переменные ядра]] |
| <code> | <code> | ||
| Line 293: | Line 294: | ||
| * [[https://blogspot.sysadm.kz/2018/07/grafana-prometheus-cisco-snmp.html|Grafana + Prometheus мониторинг траффика Cisco SNMP]] | * [[https://blogspot.sysadm.kz/2018/07/grafana-prometheus-cisco-snmp.html|Grafana + Prometheus мониторинг траффика Cisco SNMP]] | ||
| * [[https://medium.com/@openmohan/snmp-monitoring-and-easing-it-with-prometheus-b157c0a42c0c|SNMP monitoring and easing it with Prometheus]] | * [[https://medium.com/@openmohan/snmp-monitoring-and-easing-it-with-prometheus-b157c0a42c0c|SNMP monitoring and easing it with Prometheus]] | ||
| + | * [[https://grafana.com/blog/2022/02/01/an-advanced-guide-to-network-monitoring-with-grafana-and-prometheus/|An advanced guide to network monitoring with Grafana and Prometheus]] | ||
| <code> | <code> | ||
| # apt install prometheus-snmp-exporter | # apt install prometheus-snmp-exporter | ||
| + | </code> | ||
| + | === Создание файла конфигурации "вручную" === | ||
| + | |||
| + | <code> | ||
| # cat /etc/prometheus/snmp.yml | # cat /etc/prometheus/snmp.yml | ||
| </code><code> | </code><code> | ||
| - | #if_mib: # по умолчанию, позволяет не указывать module в http запросе | + | auths: |
| - | snmp_in_out_octets: | + | public_v2: |
| - | version: 2 | + | |
| - | auth: | + | |
| community: public | community: public | ||
| - | walk: | + | version: 2 |
| - | - 1.3.6.1.2.1.2.2.1.10 | + | modules: |
| - | - 1.3.6.1.2.1.2.2.1.16 | + | if_mib: |
| - | - 1.3.6.1.2.1.2.2.1.2 | + | walk: |
| - | metrics: | + | - 1.3.6.1.2.1.2.2.1.10 |
| - | - name: if_in_octets | + | - 1.3.6.1.2.1.2.2.1.16 |
| - | oid: 1.3.6.1.2.1.2.2.1.10 | + | - 1.3.6.1.2.1.2.2.1.2 |
| - | type: counter | + | metrics: |
| - | indexes: | + | - name: ifInOctets |
| - | - labelname: ifIndex | + | oid: 1.3.6.1.2.1.2.2.1.10 |
| - | type: Integer | + | type: counter |
| - | lookups: | + | indexes: |
| - | - labels: | + | - labelname: ifIndex |
| - | - ifIndex | + | type: Integer |
| - | labelname: ifDescr | + | lookups: |
| - | oid: 1.3.6.1.2.1.2.2.1.2 | + | - labels: |
| - | type: DisplayString | + | - ifIndex |
| - | - name: if_out_octets | + | labelname: ifDescr |
| - | oid: 1.3.6.1.2.1.2.2.1.16 | + | oid: 1.3.6.1.2.1.2.2.1.2 |
| - | type: counter | + | type: DisplayString |
| - | indexes: | + | - name: ifOutOctets |
| - | - labelname: ifIndex | + | oid: 1.3.6.1.2.1.2.2.1.16 |
| - | type: Integer | + | type: counter |
| - | lookups: | + | indexes: |
| - | - labels: | + | - labelname: ifIndex |
| - | - ifIndex | + | type: Integer |
| - | labelname: ifDescr | + | lookups: |
| - | oid: 1.3.6.1.2.1.2.2.1.2 | + | - labels: |
| - | type: DisplayString | + | - ifIndex |
| - | </code><code> | + | labelname: ifDescr |
| + | oid: 1.3.6.1.2.1.2.2.1.2 | ||
| + | type: DisplayString | ||
| + | </code> | ||
| + | |||
| + | === Создание файла конфигурации через "generator" === | ||
| + | |||
| + | <code> | ||
| + | # cp /usr/share/doc/prometheus-snmp-exporter/examples/generator.yml . | ||
| + | может понадобиться удалить все modules, кроме if_mib | ||
| + | |||
| + | # prometheus-snmp-generator generate | ||
| + | |||
| + | # cp snmp.yml /etc/prometheus/snmp.yml | ||
| + | </code> | ||
| + | |||
| + | |||
| + | === Проверка конфигурации и перезапуск prometheus-snmp-exporter === | ||
| + | <code> | ||
| + | # prometheus-snmp-exporter --dry-run | ||
| # service prometheus-snmp-exporter restart | # service prometheus-snmp-exporter restart | ||
| </code> | </code> | ||
| + | |||
| + | === Примеры использования === | ||
| * [[http://server.corpX.un:9116/]] | * [[http://server.corpX.un:9116/]] | ||
| <code> | <code> | ||
| - | # curl --noproxy 127.0.0.1 'http://127.0.0.1:9116/snmp?target=router&module=snmp_in_out_octets' | + | # curl 'http://127.0.0.1:9116/snmp?target=router' |
| </code><code> | </code><code> | ||
| # cat /etc/prometheus/prometheus.yml | # cat /etc/prometheus/prometheus.yml | ||
| Line 350: | Line 376: | ||
| - router | - router | ||
| metrics_path: /snmp | metrics_path: /snmp | ||
| - | params: | ||
| - | module: [snmp_in_out_octets] | ||
| relabel_configs: | relabel_configs: | ||
| - source_labels: [__address__] | - source_labels: [__address__] | ||
| Line 364: | Line 388: | ||
| <code> | <code> | ||
| - | rate(if_in_octets{ifDescr="FastEthernet1/1",ifIndex="3",instance="router",job="snmp"}[1m]) | + | rate(ifInOctets{ifDescr="FastEthernet0/0",ifIndex="1",instance="router",job="snmp"}[1m]) |
| или | или | ||
| - | rate(if_out_octets{ifIndex="5",instance="router",job="snmp"}[1m]) | + | rate(ifOutOctets{ifIndex="5",instance="router",job="snmp"}[1m]) |
| - | 8*rate(if_in_octets{ifDescr="FastEthernet1/1",instance="router"}[1m]) | + | 8*rate(ifInOctets{ifDescr="FastEthernet0/0",instance="router"}[1m]) |
| или | или | ||
| - | 8*rate(if_out_octets{ifDescr="Port-channel1",instance="router"}[1m]) | + | 8*rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m]) |
| </code> | </code> | ||
сервис_prometheus.1646917480.txt.gz · Last modified: 2022/03/10 16:04 by val
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
