====== Сервис Zabbix ======

===== Документация =====

  * [[https://www.zabbix.com/ru/manuals|Zabbix Документация]]

===== Установка и запуск сервера =====

  * [[https://www.zabbix.com/release_notes|Release Notes for Zabbix]]
==== Установка SQL сервера ====

  * [[Сервис MySQL]] (для штатного пакета поставится как зависимость)

  * [[https://ma.ttias.be/finding-biggest-data-storage-consumers-zabbix/|Finding the biggest data (storage) consumers in Zabbix]]



==== Установка из репозитория производителя ====

  * [[https://www.zabbix.com/download|Download and install Zabbix]]
  * [[Управление ПО в Linux#Загрузка пакетов и зависимостей для offline установки]]

==== Установка из репозитория Debian ====

<code>
# apt install zabbix-server-mysql   #2m

# less /usr/share/doc/zabbix-server-mysql/README.Debian

# cat zabbix.sql
</code><code>
#drop database zabbix;
create database zabbix character set utf8 collate utf8_bin;

#debian11
#grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix';

#ubuntu20
#create  USER zabbix@localhost identified by 'zabbix';
#grant all privileges on zabbix.* to zabbix@localhost;
</code><code>
# mysql < zabbix.sql
</code><code>
# zcat /usr/share/zabbix-server-mysql/{schema,images,data}.sql.gz | mysql -uzabbix -pzabbix zabbix  

# cat /etc/zabbix/zabbix_server.conf.d/corpX.conf
</code><code>
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
#end newline!!!
</code><code>
# systemctl enable zabbix-server

# service zabbix-server start
</code>

===== Установка и запуск web интерфейса =====

Все следующие компоненты поставятся, как зависимости:

  * [[Сервис HTTP#Установка и запуск сервера Apache]]
  * [[Сервис HTTP#Модуль php]]
  * [[Язык программирования PHP#Модуль php mysql]]

==== Debian ====

<code>
# apt install zabbix-frontend-php php-mysql

# less /usr/share/doc/zabbix-frontend-php/README.Debian

# cat /etc/apache2/conf-available/zabbix-frontend-php.conf
</code><code>
...
php_value date.timezone Europe/Moscow
...
php_value date.timezone Europe/Moscow
...
</code><code>
# a2enconf zabbix-frontend-php

# service apache2 reload

# touch /etc/zabbix/zabbix.conf.php

# chown www-data /etc/zabbix/zabbix.conf.php
</code><code>

http://server.corpX.un/zabbix/setup.php

!!! После работы визарда рекомендуется 

# mv /usr/share/zabbix/setup.php /usr/share/zabbix/setup.php_stop

что бы никто его не повторил)
</code><code>
# less /etc/zabbix/zabbix.conf.php
</code><code>
<?php
// Zabbix GUI configuration file.
global $DB;

$DB['TYPE']     = 'MYSQL';
$DB['SERVER']   = 'localhost';
$DB['PORT']     = '0';
$DB['DATABASE'] = 'zabbix';
$DB['USER']     = 'zabbix';
$DB['PASSWORD'] = 'zabbix';

// Schema name. Used for IBM DB2 and PostgreSQL.
$DB['SCHEMA'] = '';

$ZBX_SERVER      = 'localhost';
$ZBX_SERVER_PORT = '10051';
$ZBX_SERVER_NAME = '';

$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;
</code>



===== Установка и запуск proxy =====

==== sqlite3 ====

=== Из репозитория Zabbix ===

  * Подключить репозиторий из раздела [[https://www.zabbix.com/download|Download and install Zabbix]]
<code>
# apt install zabbix-proxy-sqlite3

# mkdir /var/lib/zabbix

# В 6-й версии не нужно, БД будет создана автоматически
# zcat /usr/share/doc/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db
</code>
=== Из репозитория Debian ===
<code>
# apt install zabbix-proxy-sqlite3

# less /usr/share/doc/zabbix-proxy-sqlite3/README.Debian

# zcat /usr/share/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db
</code>
=== Настройка и запуск ===
<code>
# chown -R zabbix:zabbix /var/lib/zabbix

# cat /etc/zabbix/zabbix_proxy.conf
</code><code>
...
Hostname=gate
ConfigFrequency=60
Server=server
DBName=/var/lib/zabbix/zabbix.db
</code>

==== MySQL ====

=== Из репозитория Debian ===

<code>
gate# apt install zabbix-proxy-mysql

gate# cat zabbix_proxy.sql
</code><code>
#drop database zabbix_proxy;
create database zabbix_proxy character set utf8 collate utf8_bin;
grant all privileges on zabbix_proxy.* to zabbix@localhost identified by 'zabbix';
</code><code>
gate# mysql < zabbix_proxy.sql

gate# zcat /usr/share/zabbix-proxy-mysql/schema.sql.gz | mysql -uzabbix -pzabbix zabbix_proxy

gate# cat /etc/zabbix/zabbix_proxy.conf
</code><code>
...
Hostname=gate
ConfigFrequency=60
Server=server
DBHost=localhost
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix
</code><code>
gate# systemctl enable zabbix-proxy

gate# service zabbix-proxy start
</code>


===== Добавление proxy в zabbix server =====

<code>
Administration->Proxies
  Proxy name: gate
  Proxy mode: Active
  Proxy address: gate
</code>
===== Обновление Zabbix =====

  * [[https://www.zabbix.com/documentation/current/en/manual/installation/upgrade|Zabbix Documentation Upgrade procedure]] см. log_bin_trust_function_creators
  * [[https://www.zabbix.com/documentation/current/en/manual/installation/upgrade/packages/debian_ubuntu|upgrade packages Debian/Ubuntu]] и !!![[https://www.zabbix.com/documentation/current/en/manual/installation/upgrade_notes_700|Upgrade notes for 7.0.0]]
  * [[https://bestmonitoringtools.com/upgrade-zabbix-to-the-latest-version/|Upgrade Zabbix (5.0, 5.2, 5.4) to 6.0 like a Pro (+6.0 to 6.4 guide)]]




===== Настройка мониторинга =====

  * !!! Однажды Setup не заработал в firefox !!!
  * [[https://www.zabbix.com/documentation/current/ru/manual/quickstart|Zabbix Documentation. Быстрый старт]]
  * Login: Admin; Pass: zabbix

<code>
http://192.168.X.10/zabbix/
</code>
==== Настройка уведомлений о событиях ====

  * [[https://www.zabbix.com/documentation/3.0/ru/manual/quickstart/notification|Zabbix Documentation Получение оповещения о проблеме]]
  * [[http://cavaliercoder.com/blog/testing-zabbix-actions.html|Testing Zabbix actions]]
  * [[https://www.zabbix.com/forum/zabbix-help/43643-how-to-switch-back-from-problem-to-ok|How to switch back from PROBLEM to OK ?]]

=== Настройка подсистемы Email ===
<code>
Administration->Media types->Email
  SMTP server: server.corpX.un
  SMTP helo: server.corpX.un
  SMTP email: zabbix@corpX.un

Administration->Users->student->Media
  Type: Email
  Send to: student@corpX.un
</code>
=== Настройка уведомлений о срабатывании триггеров ===
<code>
Configuration->Actions->Event source: Triggers
  Report problems to Zabbix administrators->Enable
  ...
</code>

=== Использование скриптов ===

  * [[https://www.zabbix.com/documentation/current/manual/config/notifications/media/script|Zabbix Documentation/CUSTOM ALERTSCRIPTS]]

<code>
# grep AlertScriptsPath /etc/zabbix/zabbix_server.conf
</code>

=== Пример XMPP ===

  * [[Сервис OpenFire]] Настройка DNS, Установка, Spark
  * [[Сервис OpenFire#bash xmpp]]

<code>
# getent passwd zabbix

# mkdir /var/lib/zabbix/

# cat /var/lib/zabbix/.sendxmpprc
admin@corpX.un Pa$$w0rd

# chmod 600 /var/lib/zabbix/.sendxmpprc

# chown -R zabbix:zabbix /var/lib/zabbix/

# cat /usr/lib/zabbix/alertscripts/notification_xmpp.sh
</code><code>
#!/bin/bash

logger -t zabbix_xmpp -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF | sendxmpp -f /var/lib/zabbix/.sendxmpprc -s "$subject" "$to"
$body
EOF
</code><code>
Administration->Media types->Create media type
  Name: XMPP
  Type: Script
  Script Name: notification_xmpp.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
    Problem recovery
    Problem update
</code>
<code>
Administration->Users->student->Media
  Type: XMPP
  Send to: student@corpX.un
</code>

=== Пример c Asterisk ===

  * уровень severity - Disaster, настроить этот уровень для проверок доступности провайдеров
  * [[Сервис Asterisk]] (Установка, в sip.conf можно только канал 403)
  * [[Пакет sudo]]
  * [[Сервис Festival]]
  * [[Сервис Asterisk#Синтез речи с использованием пакета Festival]]
  * [[Сервис Asterisk#Организация обратных вызовов]]

<code>
# cat /usr/lib/zabbix/alertscripts/notification_pstn.sh
</code><code>
#!/bin/bash

logger -t zabbix_pstn -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF > /tmp/zabbix.txt
$subject
$body
EOF

cat /tmp/zabbix.txt | text2wave -eval '(voice_msu_ru_nsh_clunits)' > /tmp/zabbix.wav

sox /tmp/zabbix.wav -r 8000 -c 1 /tmp/zabbix.raw

rm /tmp/zabbix.wav

#sudo /usr/sbin/asterisk -x "channel originate SIP/smg1016/$to extension 326@default"
sudo /usr/sbin/asterisk -x "channel originate SIP/$to extension 326@default"
</code>

  * Media Type аналогично XMPP но только шаблон Problem и упростить текст:

<code>
Administration->Media types->Create media type
  Name: PSTN
  Type: Script
  Script Name: notification_pstn.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
  Message templates->
    Subject: Звонок с работы.
    Message:
Обнаружена проблема: {EVENT.NAME}.
Имя узла: {HOST.NAME}.
</code>
<code>
Administration->Users->student->Media
  Type: PSTN
  Send to: 8495123456X или 403
  Use if severity
    Disaster
</code>

=== Отчет об отправленных сообщениях ===
<code>
Reports -> Action log
</code>

=== Добавление информации о значении элемента в сообщение ===

Zabbix 4

<code>
Configuration->Actions->Event source: Triggers
  Report problems to Zabbix administrators

    Operations:
      Default message
</code><code>
...
{ITEM.NAME1} ({HOST.NAME1}:{ITEM.KEY1}): 

==CURRENT==

{ITEM.VALUE1}

==PREVIOUS==

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}
</code>

Zabbix 5

<code>
Administration->Media type->Email->Message templates
  Problem

...
Current:

{ITEM.VALUE1}

Previous:

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}
</code>

Zabbix 6

  * [[https://www.zabbix.com/forum/zabbix-help/445808-display-an-items-previous-value|Display an items previous value...does not work...]]
==== Простые проверки ====

<code>
Host: ya.ru                         или val.bmstu.ru
  ЛЮБОЙ ИНТЕРФЕЙС: ya.ru            или val.bmstu.ru
  New group: External Hosts
  Items 
    Name: check perf http
    Type: Simple check
    Key: net.tcp.service.perf[https]
    Type of information: Numeric (float)
</code>

<code>
Host: server.corpX.un
...
  Items
    Name: check ping gate.isp.un
    Type: Simple check
    Key: icmpping[gate.isp.un]
    Applications: Network check

Host: server.corpX.un
...
  Items
    Name: check rdp admin/host windows
    Type: Simple check
    Key: net.tcp.service[tcp,192.168.X.5,3389]
#    Key: net.tcp.service[tcp,192.168.X.29,3389]
    Applications: Service check
</code>
==== Внешние проверки ====

  * [[https://www.zabbix.com/documentation/4.0/ru/manual/config/items/itemtypes/external|Zabbix Documentation Внешние проверки]]

<code>
server# zabbix_server --help | grep ExternalScripts

server# cat /etc/zabbix/zabbix_server.conf
</code><code>
...
Timeout=30
...
ExternalScripts=/etc/zabbix/externalscripts
...
</code><code>
server# mkdir /etc/zabbix/externalscripts
</code>

=== Пример простого скрипта ===

Пример 1

<code>
server# cat /etc/zabbix/externalscripts/ping_avg.sh
</code><code>
#!/bin/sh
ping -c"$1" "$2" | tail -n1 | cut -d'/' -f5
</code><code>
server# /etc/zabbix/externalscripts/ping_avg.sh 3 ya.ru
</code><code>
Cofiguration->Hosts->ya.ru или val.bmstu.ru
  Items
    Name: Ping AVG
    Type: External Check
    Key: ping_avg.sh[3,"{HOST.CONN}"]
    Type of information: Numeric (float)
    Units: ms
</code>

Пример 2

  * [[Сервис speedtest]]

<code>
server# cat /etc/zabbix/externalscripts/speedtest.sh
</code><code>
#!/bin/sh

if [ "x$1" = xupload ]
then
        A="--no-download"
        F=8
elif [ "x$1" = xdownload ]
then
        A="--no-upload"
        F=7
else
        exit 1
fi

speedtest-cli --csv $A | cut -d',' -f $F
</code><code>
# /etc/zabbix/externalscripts/speedtest.sh upload

# /etc/zabbix/externalscripts/speedtest.sh download
</code><code>
Cofiguration->Hosts->server.corpX.un
  Items
    Name: speedtest download
    Type: External Check
    Key: speedtest.sh[download]
    Type of information: Numeric (float)
    Units: Бит/сек
    Update interval: 30m

...    
    Name: speedtest upload
...
</code>

=== Пример скрипта, требующего повышения привилегий ===

  * [[Утилита nmap]]
  * [[Пакет sudo]]

!!! Для некоторых хостов (например, val.bmstu.ru) не хватит Timeout

<code>
server# cat /etc/zabbix/externalscripts/detect_host_nmap.sh
</code><code>
#!/bin/sh
sudo /usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done'
</code><code>
Cofiguration->Hosts->gate
  Items
    Name: Detect host operating system by nmap
    Type: External Check
    Key: detect_host_nmap.sh["{HOST.CONN}"]
    Type of information: Text
</code>

=== Пример запуска скрипта на удаленной системе ===

<code>
server# service zabbix-server stop
server# service zabbix-agent stop
</code>

  * [[Управление учетными записями в Linux#Назначение пользователю домашнего каталога]]

<code>
server# service zabbix-server start
server# service zabbix-agent start

server# sudo -u zabbix bash

zabbix@server:~$ ssh-keygen

zabbix@server:~$ ssh-copy-id root@gate

zabbix@server:~$ ssh root@gate hostname
</code>

  * [[Сервис DHCP#Статистика DHCP сервера]]

<code>
zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh CUR

zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh MAX

server# cat /etc/zabbix/externalscripts/dhcp_stat_ext.sh
</code><code>
#!/bin/sh
ssh root@$1 /usr/local/bin/dhcp_stat.sh $2
</code><code>
zabbix@server:~$ /etc/zabbix/externalscripts/dhcp_stat_ext.sh gate CUR
</code><code>
gate->Items
  Name: DHCP stat CUR
  Type: External check
  Key: dhcp_stat_ext.sh["{HOST.CONN}",CUR]
</code>
==== Элементы типа trapper ====

=== Пример численного элемента ===

Пример 1

<code>
HOSTNAME_IN_CONFIG->Items
  Name: my item
    Type: Zabbix trapper
    Key:  my.item
    Allowed hosts: 127.0.0.1, 192.168.X.0/24
</code><code>
# apt install zabbix-sender

$ zabbix_sender -z IP/DNSNAME -p 10051 -s HOSTNAME_IN_CONFIG -k my.item -o 1
</code>

Пример 2

<code>
server.corpX.un->Items
  Name: speedtest download trap
    Type: Zabbix trapper
    Key:  speedtest.download
    Type of information: Numeric (float) или Numeric (unsigned)
    Units: бит/с  или  bit/s
    Allowed hosts: 127.0.0.1
  Preprocessing    может понадобиться
    Custom multiplier: 8  
...
  Name: speedtest upload trap
...
</code><code>
# cat /root/speedtest.sh
</code><code>
#!/bin/sh

### speedtest-cli ### result bits/s
MY_RES=`speedtest-cli --csv`
MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f7`
MY_UPLOAD=`echo $MY_RES | cut -d',' -f8`

### speedtest ### result Bytes/s (use preprocess Custom multiplier)
#MY_RES=`speedtest -f csv`
#MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f6`
#Y_UPLOAD=`echo $MY_RES | cut -d',' -f7`

zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.download -o $MY_DOWNLOAD
zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.upload -o $MY_UPLOAD
</code><code>
# crontab -l
</code><code>
...
X * * * * /root/speedtest.sh >/dev/null 2>&1
</code>

=== Пример текстового элемента ===
<code>
val.bmstu.ru->Items
  Name: my nmap
    Type: Zabbix trapper
    Key:  my.nmap
    Type of information: Text
    Allowed hosts: 127.0.0.1
</code><code>
server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(nmap -O val.bmstu.ru)"
</code><code>
server# cat /root/detect_host_nmap.sh
</code><code>
#!/bin/sh
/usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done\|Network Distance'
</code><code>
server# chmod +x /root/detect_host_nmap.sh
</code><code>
server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(/root/detect_host_nmap.sh val.bmstu.ru)"
</code>
==== Вычисляемые элементы ====

<code>
ya.ru->Items
  Name: avg perf http
    Type: Calculated
    Key:  my.avg.perf.http
    Formula: avg(net.tcp.service.perf[https],5m)
    Type of information: Numeric (float)

#---------------------

gate.corpX.un->Items
  Name: DHCP stat CUR
    Type: Zabbix agent
    Key: dhcp.stat[CUR]

  Name: DHCP stat MAX
    Type: Zabbix agent
    Key: dhcp.stat[MAX]
    
  Name: DHCP stat CUR MAX percent
    Type: Calculated
    Key:  DHCP.stat.CUR.MAX.percent
    Formula: last(dhcp.stat[CUR])/last(dhcp.stat[MAX])*100
    
#---------------------

last(openvpn1:openvpn.server.clients)+last(openvpn2:openvpn.server.clients)+last(openvpn3:openvpn.server.clients)
</code>
==== Web сценарии ====

  * [[https://www.zabbix.com/documentation/4.0/ru/manual/web_monitoring/example|Сценарий из реальной жизни]]
  * [[https://stackoverflow.com/questions/58641673/zabbix-web-scenario-debug|Zabbix web scenario debug]]

=== Исследование приложения ===

  * !!! [[https://stackoverflow.com/questions/15603561/how-can-i-debug-a-http-post-in-chrome|How can I debug a HTTP POST in Chrome?]]

<code>
Шаг 1.

Браузер: http://192.168.X.10/mail/

view-source
...
...Roundcube Webmail...
...
<input type="hidden" name="_token" value="29JVrZhgW97xID7K2pkSRRHsngGDRGCY">
...

Шаг 2.
Браузер: вводим логин/пароль и нажимаем "Войти"

server# tcpdump -n -A port 80 | tee dump2.txt
...
POST /mail/?_task=login HTTP/1.1
...
_token=29JVrZhgW97xID7K2pkSRRHsngGDRGCY&_task=login&_action=login&_timezone=Europe%2FMoscow&_url=&_user=student&_pass=password
...
HTTP/1.1 302 Found
...
Location: ./?_task=mail&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP
...

view-source
...
...button-logout...
...
<input type="hidden" name="_token" value="pWUje42O61E2Rm0r8zgKzOPXWGby8ugP">
...

3.
Браузер: нажимаем "Выход"

server# tcpdump -n -A port 80 | tee dump3.txt
...
GET /mail/?_task=logout&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP HTTP/1.1
...
</code>

=== Web scenario ===
<code>
Name: mail corpX
Variables
  {login} student
  {password} password

Steps

Step 1
  Name: First page
  URL: http://server.corpX.un/mail/

  Variables
    {token1} regex:name="_token" value="([0-9A-Za-z]{32})"
Можно проще:
    {token1} regex:name="_token" value="(.{32})"

  Required string: rcmloginsubmit
  Required status codes: 200
  
Step 2
  Name: Log in
  URL: http://server.corpX.un/mail/
  
  Post fields
    _token: {token1}
    _task: login
    _action: login
    _user: {login}
    _pass: {password}

  Variables
    {token2}: regex:name="_token" value="(.{32})"
    
  Follow redirects: YES
  
  Required string: button-logout
  Required status codes: 200
  
Step 3
  Name: Log out
  URL: http://server.corpX.un/mail/
  
  Query fields
    _task: logout
    _token: {token2}
    
  Required string: rcmloginsubmit
  Required status codes: 200
</code>

==== Настройка триггеров ====

  * [[https://www.zabbix.com/documentation/current/en/manual/config/triggers/trigger|Configuring a trigger]]

=== Примеры простых целочисленных триггеров ===
<code>
Host: server.corpX.un
    Name: gate.isp.un is unreachable
    Expression: {server.corpX.un:icmpping[gate.isp.un].last()}=0    Zabbix < 5.4
    Expression: last(/server.corpX.un/icmpping[gate.isp.un])=0     Zabbix >= 5.4
    Severity: High

    Name: RDP service on host is not available
    Expression: {server.corpX.un:net.tcp.service[tcp,192.168.X.5,3389].last()}=0
    Expression: last(/server.corpX.un/net.tcp.service[tcp,192.168.X.29,3389])=0
    Severity: Warning

Host: ya.ru 
    Name: HTTPS service on {HOST.NAME} is not available
    Expression: {ya.ru:net.tcp.service.perf[https].max(#2)}=0
    Expression: max(/ya.ru/net.tcp.service.perf[https],#2)=0

    Dependencies: Zabbix server: gate.isp.un is unreachable   
        !!!Можно добавить после эксперимента с недоступностью gate.isp.un и двумя, сработавшими триггерами

    Severity: Average
</code>

=== Пример триггера основанного на значении элемента в течении периода ===
<code>
Host: ya.ru                                       
    Name: HTTP service on {HOST.NAME} is slow
    
    Значение подобрать на основе графика
    
    Expression: {ya.ru:my.avg.perf.http.last()}>0.4
    Expression: {ya.ru:net.tcp.service.perf[https].avg(5m)}>0.4  
    Expression: avg(/ya.ru/net.tcp.service.perf[https],5m)>0.4

    Severity: Warning
</code>

=== Пример с макросами и Recovery expression ===

!!! Добавить в поле "Operational data" информацию о текущем и максимальном количестве адресов

<code>
    Name: On {HOST.NAME} dhcp subnet is full
    Problem expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100>{$DHCP.POOLS.MAX.PERCENT}
    Recovery expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100<{$DHCP.POOLS.OK.PERCENT}
    Description:
      В заканчиваются адреса.
      Израсходовано более {$DHCP.POOLS.MAX.PERCENT} процентов.
</code>

=== Пример с текстовым элементом ===
Zabbix 5
<code>
Name: Host {HOST.NAME} nmap change
Severity: Warning

Expression: {gate.corpX.un:detect_host_nmap.sh["{HOST.CONN}"].diff()}=1
  или
Expression: {val.bmstu.ru:my.nmap.diff()}=1
</code>
Zabbix 6
<code>
Name: installed soft on host {HOST.NAME} change
Severity: Warning

Expression: (last(/Template OS Windows list installed soft Active/listinstalledsoft,#1)<>last(/Template OS Windows list installed soft Active/listinstalledsoft,#2))=1
</code>

=== Пример в веб сценарии ===

  * [[https://www.zabbix.com/documentation/current/ru/manual/web_monitoring/items|Элементы данных веб-мониторинга]]

<code>
Name: Web scenario mail corpX FAIL
Severity: Hight
Expression: {server.corpX.un:web.test.fail[mail corpX].last()}>0
Expression: last(/server.corpX.un/web.test.fail[mail corpX])>0
</code>

==== Пользовательские графики ====

<code>
Configuration->Hosts->ya.ru->Graphs->Create graph             или val.bmstu.ru
  Name: perf http
  Y axis MIN value: Fixed 0
  Items
    ya.ru: avg perf http                                      или val.bmstu.ru
    ya.ru: check perf http                                    или val.bmstu.ru
  Draw style: Bold Line
</code>

==== Мониторинг с использованием Zabbix agents ====

=== Установка агента из репозитория вендора ===

  * [[https://www.zabbix.com/download|Download and install Zabbix Packages]]

=== Установка агента из репозитория Debian/Ubuntu ===
<code>
# apt install zabbix-agent
</code>

=== Установка агента в Windows ===

  * [[https://www.zabbix.com/download_agents|Download and install pre-compiled Zabbix agents]]
  * [[http://val.bmstu.ru/unix/zabbix/zabbix_agent-4.0.17-windows-amd64-openssl.msi]]

=== Список элементов агента ===
<code>
# zabbix_agentd -p

# zabbix_agentd -p | grep agent.version
</code>
  * [[https://www.zabbix.com/documentation/5.0/ru/manual/appendix/items/vm.memory.size_params|ПАРАМЕТРЫ VM.MEMORY.SIZE]]
<code>
# zabbix_agentd -p | grep vm.memory.size

# zabbix_agentd -t vm.memory.size[available]

# cat /proc/meminfo | grep MemAvailable

# zabbix_agentd -t system.sw.packages

# dpkg -l
</code>

=== Пассивный режим ===

Проверка связи с агентом:
<code>
# apt install zabbix-get

$ zabbix_get -s IP/DNSNAME -p 10050 -k agent.version
</code>

Минимальная конфигурация агента, включая PSK
<code>
# cat /etc/zabbix/zabbix_agentd.conf

или

# cat `echo /etc/zabbix/zabbix_agentd.*d/`corpX.conf
</code><code>
Server=server

# ListenPort=10050

# Hostname=Zabbix server
# must match hostname as configured in Zabbix
# Иначе в журнале будут сообщения: cannot send list of active checks

#TLSConnect=psk
#TLSAccept=psk
##TLSAccept=unencrypted,psk
#TLSPSKFile=/etc/zabbix/zabbix_agentd.psk
#TLSPSKIdentity=gate
</code><code>
gate# openssl rand -hex 32 > /etc/zabbix/zabbix_agentd.psk

gate# scp /etc/zabbix/zabbix_agentd.psk server:gate.psk

gate# service zabbix-agent restart

server# zabbix_get -s gate -k system.sw.packages --tls-connect=psk --tls-psk-identity="gate" --tls-psk-file=gate.psk
</code>

=== Активный режим ===

  * [[https://docs.linuxconsulting.mn.it/notes/zabbix-active-agent|How to configure an Active Agent in Zabbix 2.2]]

== Настройка авторегистрации систем с агентами, работающими в активном режиме ==

<code>
Configuration - Actions - Auto registration 
  Name: Add Windows clients                                          # or Add Linux clients
  Conditions: Host name contains CLIENT                              # or client (lowercase) for linux
  Action operations: 
    Add to host groups: Windows clients                              # or Linux clients
    Link to templates: Windows by Zabbix agent active                # or Linux by Zabbix agent active
                     Template OS Windows list installed soft Active  # work in linux too))
  Set host inventory mode: Automatic
</code>

== Настройка агента на активный режим ==

<code>
LogFile=C:\Program Files\Zabbix Agent\zabbix_agentd.log
#Server=server
ListenIP=0.0.0.0
StartAgents=0
ServerActive=server
#Hostname=CLIENTN
</code>
<code>
/var/log/cisco_routers.log:
Jun 10 15:06:23 192.168.32.224 4278: Jun 10 15:06:22: %IP-4-DUPADDR: Duplicate address 192.168.0.254 on Port-channel6.664, sourced by 30de.4bf5.e9cc
Jun 10 16:25:07 192.168.32.224 4308: Jun 10 16:25:06: %SYS-5-CONFIG_I: Configured from console by val on vty0 (192.168.32.6)
last(/helper.bmstu.ru/log[/var/log/cisco_routers.log,Duplicate address])<>0
</code>

=== Использование UserParameter ===

== Примеры для Linux ==

  * [[Сервис DHCP#Статистика DHCP сервера]]

<code>
gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf
</code><code>
UserParameter=dhcp.stat[*],/usr/local/bin/dhcp_stat.sh $1
</code><code>
server# zabbix_get -s gate -k dhcp.stat[CUR]
server# zabbix_get -s gate -k dhcp.stat[MAX]
</code>

  * [[Управление ПО в Linux#Список desktop приложений]]

<code>
linclient2:~# cat /etc/zabbix/zabbix_agentd.conf.d/listinstalledsoft.conf
</code><code>
UserParameter=listinstalledsoft,ls /usr/share/applications | awk -F '.desktop' ' { print $1}' -
# UserParameter=listinstalledsoft,ls /usr/share/applications /usr/local/share/applications | awk -F '.desktop' ' { print $1}' -
</code>

== Примеры для Windows ==

  * [[PowerShell#Список установленного ПО]]

<code>
Admin C:\> C:\Program Files\Zabbix Agent\zabbix_agentd.conf
</code><code>
...
Timeout=30

#UserParameter=listinstalledsoft,powershell -Command Get-ChildItem HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall; powershell -Command Get-ChildItem HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall

UserParameter=listinstalledsoft,C:\bin\listinstalledsoft.bat | findstr /v "^$"

UnsafeUserParameters=1
UserParameter=dir[*],dir $1
UserParameter=runcommand[*],$1
UserParameter=lmstat[*],C:\Progra~1\PTC\flexnet\bin\lmutil lmstat -a -c $1
</code><code>
C:\>notepad C:\bin\listinstalledsoft.bat
</code><code>
@echo off

powershell -command "Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"
powershell -command "Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"

</code>
==== Мониторинг по протоколу SNMP ====

  * Была необходима [[Сервис SNMP#Установка snmp консоли]]

<code>
# service zabbix-server restart
</code>

=== Автоматическое добавление ===


  * [[https://www.zabbix.com/documentation/current/en/manual/discovery/network_discovery|Network discovery]] см. Discovery status


<code>
Configuration->Discovery
Discovery rules
  Name: Local network    или    Cisco Router Discovery
  IP range: 172.16.1.190-199, 192.168.X.51-60
  Checks: 
    Check type: SNMPv2 agent 
    SNMP community: public
    SNMP OID .1.3.6.1.2.1.1.5.0
      или
             SNMPv2-MIB::sysName.0       !!! Требуется установка MIB
    Add
  Update interval: 3m

  Host name: SNMPv2 agent ...            !!! Остальные варианты требуют PTR записи

  Enabled: yes

Configuration->Actions
  Event source: Discovery
    Name: Action add snmp device to zabbix
    Conditions: 
      Received value: like или contains router
      Discovery status: equals Up                 без этого появляются ghost hosts :)
      Add
    Operations: 
      Add to host groups: Cisco devices             !!! В Discovered hosts добавится автоматически !!!
      Link to templates: 
             3.XX Template SNMP Device
             4.XX Template Module Generic SNMPv2
             4.XX Template Module Interfaces Simple SNMPv2
             5.XX Template Net Cisco IOS SNMP       !!! Уменьшить период Network interfaces discovery !!!
             6.XX Cisco IOS by SNMP
      Set host inventory mode: Automatic
    Add
</code>

=== SNMPv3 ===

  * [[https://blog.zabbix.com/monitoring-network-hardware-with-snmpv3-in-zabbix/10093/|Monitoring network hardware with SNMPv3 in Zabbix]]
==== Проверки через SSH ====

  * [[https://www.zabbix.com/documentation/3.0/ru/manual/config/items/itemtypes/ssh_checks|Zabbix Documentation Проверки через SSH]]

==== Обработка SNMPTRAP ====

  * [[http://va0816.blogspot.ru/2013/06/zabbix-snmp-traps.html|Настройка Zabbix SNMP traps]]
  * [[https://www.zabbix.com/documentation/3.0/ru/manual/config/items/itemtypes/snmptrap|Zabbix Documentation SNMP трапы]]
  * [[https://gist.github.com/jpawlowski/152abb4951f39ce1cfa0b1c5220b8635|SNMPTT installation on CentOS 7 for Zabbix integration]]
  * [[https://programmersought.com/article/39768892920/|How to deploy SNMP Traps in Centos8]]
  * [[https://www.zabbix.com/forum/zabbix-help/28463-catch-all-snmp-traps-with-general-event|catch all snmp traps with general event]]
  * [[https://blog.zabbix.com/snmp-traps-in-zabbix/8210/|SNMP Traps in Zabbix]]
  * [[https://youtu.be/fVK2YWdTalQ|youtube SNMP Traps in Zabbix Tutorial]]

<code>
# apt install snmptt

# systemctl disable snmptt

# systemctl stop snmptt

# cat /etc/snmp/snmptt.conf
</code><code>
EVENT general .* "General event" Normal
FORMAT ZBXTRAP $aA $ar
</code><code>
# cat /etc/snmp/snmptt.ini
</code><code>
...
date_time_format = %H:%M:%S %Y/%m/%d
...
#log_file = /var/log/snmptt/snmptt.log
log_file = /tmp/my_zabbix_traps.tmp
...
</code><code>
# cat /etc/snmp/snmptrapd.conf
</code><code>
traphandle default snmptt

authCommunity execute writetrap
</code><code>
# cat /lib/systemd/system/snmptrapd.service
</code><code>
...
ExecStart=/usr/sbin/snmptrapd -Lsd -f -On
...
</code><code>
# systemctl daemon-reload
# service snmptrapd restart
</code>




==== Создание своих шаблонов ====

  * Пример с использованием макроса
  * [[https://www.zabbix.com/documentation/5.0/ru/manual/config/items/itemtypes/simple_checks|ПРОСТЫЕ ПРОВЕРКИ]]

<code>
Configuration
  Templates
    Template App SSH Service/SSH Service
  Full Clone
    Template App SSH Port Service/SSH Port Service
  Add

  Templates
    Template App SSH Port Service/SSH Port Service
      Items
        SSH service is running
          Key: net.tcp.service[ssh,,{$SSH_PORT}]
          Update interval: 30s
      Macros
        {$SSH_PORT}=22
</code>
====  Low-Level Discovery (LLD) ====

  * [[https://www.zabbix.com/documentation/3.0/ru/manual/discovery/low_level_discovery|Zabbix Documentation Низкоуровневое обнаружение]]
  * [[https://habrahabr.ru/company/zabbix/blog/193460/|Автоматизируем мониторинг: низкоуровневое обнаружение]]
  * [[https://www.zabbix.com/forum/in-russian/44171-perl-script-%D0%90%D0%B2%D1%82%D0%BE%D1%80%D0%B8%D1%81%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1%8F-%D1%82%D0%BE%D0%BF%D0%BE%D0%BB%D0%BE%D0%B3%D0%B8%D0%B8-%D1%81%D0%B5%D1%82%D0%B8-%D0%B2-zabbix|Perl script Авторисования топологии сети в Zabbix]]
  * [[https://youtu.be/MEdVvZU18ek|Видеоурок - Самый простой, но очень полезный пример использования низкоуровнего обнаружения (LLD) в Zabbix]]


=== Штатные примеры ===

<code>
server# zabbix_get -s gate -k vfs.fs.discovery
</code>
  * [[Утилита jq]]
<code>
server# zabbix_get -s gate -k vfs.fs.size[/,free]
server# zabbix_get -s gate -k vfs.fs.size[/,total]
server# zabbix_get -s gate -k vfs.fs.size[/,used]
...

server# zabbix_get -s gate -k net.if.discovery | jq
...

server# zabbix_get -s gate -k net.if.in[eth0]
...
</code>

=== Статистика дискового ввода/вывода ===

  * Пример использования встроенных средств для LLD

<code>
agent# zabbix_agentd -p | grep vfs.dev
...

server# zabbix_get -s agent -k "vfs.dev.read[sda,sps]"
...

agent# cat /etc/zabbix/zabbix_agentd.conf.d/my.linux.disk.discovery.conf
</code><code>
UserParameter=my.disks.discovery,/bin/lsblk -dJ | /bin/sed -e 's/blockdevices/data/' -e 's/name/{#NAME}/g' -e 's/type/{#TYPE}/g'
</code><code>
server# zabbix_get -s agent -k my.disks.discovery | jq
...
</code><code>
Configuration->Templates->Create template
  Template name: My Template Linux disks utilization
  Groups: Templates/Server hardware

  Discovery rules->
    Name: my disks discovery
    Key: my.disks.discovery
    Filters->
      {#TYPE} matches disk
    Item prototypes->
      Name: disk {#NAME} read bytes
      Key: vfs.dev.read[{#NAME},sps]
      Type of information: Numeric (float)
      Units: Байт
      Preprocessing->
        Preprocessing steps
        Custom multiplier 512

      Name: disk {#NAME} write bytes
      ...  
</code>

=== Статистика DHCP сервера ===

  * Пример разработки собственного скрипта для LLD
  * [[Обработка XML файлов]]
  * [[https://stackoverflow.com/questions/12524437/output-json-from-bash-script|Output JSON from Bash script]]

<code>
gate# cat /etc/zabbix/dhcp-pools-discovery.sh
</code><code>
#!/bin/bash

echo -n '{"data":['

str=`/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
/usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
-o '{"{#POOLNAME}":"' -v location -o '"},'`

echo -n ${str::-1}

echo -n ']}'
</code><code>
gate# /etc/zabbix/dhcp-pools-discovery.sh | jq

gate# cat /etc/zabbix/dhcp-pools-shared-network.sh
</code><code>
#!/bin/sh

res_field=2
test "x$2" = "xused" && res_field=3

/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
  /usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
  -v location -o ' ' -v defined -o ' ' -v used -n | \
  grep $1 | cut -d ' ' -f $res_field
</code><code>
gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN1 defined

gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN2 used

gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf
</code><code>
UserParameter=dhcp.pools.discovery,/etc/zabbix/dhcp-pools-discovery.sh

UserParameter=dhcp.pools.shared-network[*],/etc/zabbix/dhcp-pools-shared-network.sh $1 $2
</code>

<code>
server# zabbix_get -s gate -k dhcp.pools.discovery | jq

server# zabbix_get -s gate -k dhcp.pools.shared-network[LAN1,used]
</code>

<code>
Configuration->Templates->Create template
  Template name: Template App DHCP Pools
  Groups In groups: Templates/Applications

  Macros: {$DHCP.POOLS.MAX.PERCENT}=90
Add

  Applications: DHCP
  
  Discovery rules
    Name: Search DHCP Pools
    Type: Zabbix Agent
    Key: dhcp.pools.discovery
  Add
    Item prototypes
      Name: DHCP Pool $1 max addr      или DHCP Pool {#POOLNAME} max addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},defined]
      Applications: DHCP
    Add

      Name: DHCP Pool $1 cur addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},used]
      Applications: DHCP
    Add

    Graph prototypes
      Name: DHCP Pool {#POOLNAME} max cur
      Y axis MIN value: Fixed 0
      Items: 
        Template App DHCP Pools: DHCP Pool {#POOLNAME} cur addr
        Template App DHCP Pools: DHCP Pool {#POOLNAME} max addr

    Trigger prototypes

      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME}
или
      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME} used more {$DHCP.POOLS.MAX.PERCENT} percent
      
      Expression: {Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},used].last()}/{Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},defined].last()}*100 > {$DHCP.POOLS.MAX.PERCENT}
                  last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},used])/last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},defined])*100 > {$DHCP.POOLS.MAX.PERCENT}
      Severity: Warning
</code>
==== Экспорт/импорт в XML ====

<code>
Configuration
  Templates
    Export/Import
</code>


===== API =====

  * [[Формат JSON]]
  * [[https://www.zabbix.com/documentation/1.8/ru/api/getting_started|Zabbix Documentation Начало работы с Zabbix API]]

==== Аутентификация ====

  * [[https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/36900-api-key-lifetime|api key lifetime]]

<code>
server:~# apt install curl

server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
   "jsonrpc": "2.0",
   "method": "user.login",
   "params": {
      "user": "Admin",
      "password": "zabbix"
   },
   "id": 1
} ' http://127.0.0.1/zabbix/api_jsonrpc.php
</code><code>
{"jsonrpc":"2.0","result":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","id":1}
</code>

==== Работа с объектами host/template ====

  * Утилита для обработки JSON
<code>
server:~# apt install jq
</code>

  * Получение списка узлов и шаблонов из Zabbix

<code>
server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {},
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq
...
</code>
<code>
...
    "method": "template.get",
...
</code>

  * [[https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/25384-zabbix-json-api-output-parameter|Zabbix JSON API output parameter]]
  * Пример запроса определенных атрибутов и с фильтром
<code>
...
    "params": {
        "output": ["hostid", "host"],
        "templateids": ["10NNN"]
    },
...
</code>

==== Доступ к результатам мониторинга ====

  * [[https://www.zabbix.com/documentation/5.0/ru/manual/api/reference/history/get|Zabbix Documentation HISTORY.GET]]

  * history 0-число 4-текст

<code>
curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "history.get",
    "params": {
        "output": "extend",
        "history": 0,
        "itemids": "NNNNN",
        "sortfield": "clock",
        "sortorder": "DESC",
        "limit": 10
    },
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq
</code>
==== Оформление запросов в виде BASH скриптов ====

<code>
server:~# export AUTH=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

server:~# cat /root/zab_get_hosts.sh
</code><code>
#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"host.get\",
    \"params\": {},
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
</code><code>
server:~# /root/zab_get_hosts.sh | jq

Список имен узлов
server:~# /root/zab_get_hosts.sh | jq '.result | .[] | .name'
</code>
==== Получение списка карт и их элементов из Zabbix ====

<code>
server.corpX.un:~# cat /root/zab_get_maps.sh
</code><code>
#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.get\",
    \"params\": {
        \"selectLinks\": \"extend\",
        \"selectSelements\": \"extend\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
</code><code>
server.corpX.un:~# /root/zab_get_maps.sh | jq -c '.result | .[] | {name: .name, id: .sysmapid}'
</code>
==== Пример изменения конфигурации через Zabbix API ====
<code>
server.corp1.un:~# cat /root/zab_set_map_name.sh
</code><code>
#!/bin/sh

MAPID=$1
MAPNAME=$2

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.update\",
    \"params\": {
        \"sysmapid\": \"${MAPID}\",
        \"name\": \"${MAPNAME}\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
</code><code>
server.corp1.un:~# /root/zab_set_map_name.sh 2 "ISP1"
</code>