• Zabbix Документация

• Release Notes for Zabbix

• Сервис MySQL (для штатного пакета поставится как зависимость)

• Finding the biggest data (storage) consumers in Zabbix

• Download and install Zabbix
• Загрузка пакетов и зависимостей для offline установки

# apt install zabbix-server-mysql   #2m

# less /usr/share/doc/zabbix-server-mysql/README.Debian

# cat zabbix.sql

#drop database zabbix;
create database zabbix character set utf8 collate utf8_bin;

#debian11
#grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix';

#ubuntu20
#create  USER zabbix@localhost identified by 'zabbix';
#grant all privileges on zabbix.* to zabbix@localhost;

# mysql < zabbix.sql

# zcat /usr/share/zabbix-server-mysql/{schema,images,data}.sql.gz | mysql -uzabbix -pzabbix zabbix  

# cat /etc/zabbix/zabbix_server.conf.d/corpX.conf

DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
#end newline!!!

# systemctl enable zabbix-server

# service zabbix-server start

Все следующие компоненты поставятся, как зависимости:

• Установка и запуск сервера Apache
• Модуль php
• Модуль php mysql

# apt install zabbix-frontend-php php-mysql

# less /usr/share/doc/zabbix-frontend-php/README.Debian

# cat /etc/apache2/conf-available/zabbix-frontend-php.conf

...
php_value date.timezone Europe/Moscow
...
php_value date.timezone Europe/Moscow
...

# a2enconf zabbix-frontend-php

# service apache2 reload

# touch /etc/zabbix/zabbix.conf.php

# chown www-data /etc/zabbix/zabbix.conf.php

http://server.corpX.un/zabbix/setup.php

!!! После работы визарда рекомендуется 

# mv /usr/share/zabbix/setup.php /usr/share/zabbix/setup.php_stop

что бы никто его не повторил)

# less /etc/zabbix/zabbix.conf.php

<?php
// Zabbix GUI configuration file.
global $DB;

$DB['TYPE']     = 'MYSQL';
$DB['SERVER']   = 'localhost';
$DB['PORT']     = '0';
$DB['DATABASE'] = 'zabbix';
$DB['USER']     = 'zabbix';
$DB['PASSWORD'] = 'zabbix';

// Schema name. Used for IBM DB2 and PostgreSQL.
$DB['SCHEMA'] = '';

$ZBX_SERVER      = 'localhost';
$ZBX_SERVER_PORT = '10051';
$ZBX_SERVER_NAME = '';

$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;

• Подключить репозиторий из раздела Download and install Zabbix

# apt install zabbix-proxy-sqlite3

# mkdir /var/lib/zabbix

# В 6-й версии не нужно, БД будет создана автоматически
# zcat /usr/share/doc/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db

# apt install zabbix-proxy-sqlite3

# less /usr/share/doc/zabbix-proxy-sqlite3/README.Debian

# zcat /usr/share/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db

# chown -R zabbix:zabbix /var/lib/zabbix

# cat /etc/zabbix/zabbix_proxy.conf

#...
Hostname=gate

#v6
#ConfigFrequency=60
#v7
#ProxyConfigFrequency=60
  
Server=server
DBName=/var/lib/zabbix/zabbix.db

• https://github.com/kpod13/docker-compose-zabbix-proxy

lan:~/zabbix-proxy# cat docker-compose.yml

services:
  zabbix-proxy:
    environment:
      ZBX_CONFIGFREQUENCY: '60'
      ZBX_HOSTNAME: lan
      ZBX_PROXYCONFIGFREQUENCY: '60'
      ZBX_SERVER_HOST: server.corpX.un
      ZBX_TLSCONNECT: psk
      ZBX_TLSPSK: aa.............................................................7c
      ZBX_TLSPSKIDENTITY: lan
    image: zabbix/zabbix-proxy-sqlite3:ubuntu-7.0-latest
    ports:
    - published: 10051
      target: 10051
    restart: always
    stop_grace_period: 1m

gate# apt install zabbix-proxy-mysql

gate# cat zabbix_proxy.sql

#drop database zabbix_proxy;
create database zabbix_proxy character set utf8 collate utf8_bin;
grant all privileges on zabbix_proxy.* to zabbix@localhost identified by 'zabbix';

gate# mysql < zabbix_proxy.sql

gate# zcat /usr/share/zabbix-proxy-mysql/schema.sql.gz | mysql -uzabbix -pzabbix zabbix_proxy

gate# cat /etc/zabbix/zabbix_proxy.conf

...
Hostname=gate
ConfigFrequency=60
Server=server
DBHost=localhost
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix

gate# systemctl enable zabbix-proxy

gate# service zabbix-proxy start

Administration->Proxies
  Proxy name: gate
  Proxy mode: Active
  Proxy address: gate

• Zabbix Documentation Upgrade procedure см. log_bin_trust_function_creators
• upgrade packages Debian/Ubuntu и !!!Upgrade notes for 7.0.0
• Upgrade Zabbix (5.0, 5.2, 5.4) to 6.0 like a Pro (+6.0 to 6.4 guide)

• !!! Однажды Setup не заработал в firefox !!!
• Zabbix Documentation. Быстрый старт
• Login: Admin; Pass: zabbix

http://192.168.X.10/zabbix/

• Zabbix Documentation Получение оповещения о проблеме
• Testing Zabbix actions
• How to switch back from PROBLEM to OK ?

Administration/Alerts->Media types->Email
  SMTP server: server.corpX.un
  SMTP helo: server.corpX.un
  SMTP email: zabbix@corpX.un

Administration/Users->Users->student->Media
  Type: Email
  Send to: student@corpX.un

Configuration/Alerts->Actions->Event source: Triggers
  Report problems to Zabbix administrators->Enable
  ...

• Zabbix Documentation/CUSTOM ALERTSCRIPTS

# grep AlertScriptsPath /etc/zabbix/zabbix_server.conf

• Сервис OpenFire Настройка DNS, Установка, Spark
• bash xmpp

# getent passwd zabbix

# mkdir /var/lib/zabbix/

# cat /var/lib/zabbix/.sendxmpprc
admin@corpX.un Pa$$w0rd

# chmod 600 /var/lib/zabbix/.sendxmpprc

# chown -R zabbix:zabbix /var/lib/zabbix/

# cat /usr/lib/zabbix/alertscripts/notification_xmpp.sh

#!/bin/bash

logger -t zabbix_xmpp -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF | sendxmpp -f /var/lib/zabbix/.sendxmpprc -s "$subject" "$to"
$body
EOF

Administration->Media types->Create media type
  Name: XMPP
  Type: Script
  Script Name: notification_xmpp.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
    Problem recovery
    Problem update

Administration->Users->student->Media
  Type: XMPP
  Send to: student@corpX.un

• уровень severity - Disaster, настроить этот уровень для проверок доступности провайдеров
• Сервис Asterisk (Установка, в sip.conf можно только канал 403)
• Пакет sudo
• Сервис Festival
• Синтез речи с использованием пакета Festival
• Организация обратных вызовов

# cat /usr/lib/zabbix/alertscripts/notification_pstn.sh

#!/bin/bash

logger -t zabbix_pstn -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF > /tmp/zabbix.txt
$subject
$body
EOF

cat /tmp/zabbix.txt | text2wave -eval '(voice_msu_ru_nsh_clunits)' > /tmp/zabbix.wav

sox /tmp/zabbix.wav -r 8000 -c 1 /tmp/zabbix.raw

rm /tmp/zabbix.wav

#sudo /usr/sbin/asterisk -x "channel originate SIP/smg1016/$to extension 326@default"
sudo /usr/sbin/asterisk -x "channel originate SIP/$to extension 326@default"

• Media Type аналогично XMPP но только шаблон Problem и упростить текст:

Administration->Media types->Create media type
  Name: PSTN
  Type: Script
  Script Name: notification_pstn.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
  Message templates->
    Subject: Звонок с работы.
    Message:
Обнаружена проблема: {EVENT.NAME}.
Имя узла: {HOST.NAME}.

Administration->Users->student->Media
  Type: PSTN
  Send to: 8495123456X или 403
  Use if severity
    Disaster

Reports -> Action log

Zabbix 4

Configuration->Actions->Event source: Triggers
  Report problems to Zabbix administrators

    Operations:
      Default message

...
{ITEM.NAME1} ({HOST.NAME1}:{ITEM.KEY1}): 

==CURRENT==

{ITEM.VALUE1}

==PREVIOUS==

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}

Zabbix 5

Administration->Media type->Email->Message templates
  Problem

...
Current:

{ITEM.VALUE1}

Previous:

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}

Zabbix 6

• Display an items previous value...does not work...

Host: ya.ru                         или val.bmstu.ru
  ЛЮБОЙ ИНТЕРФЕЙС: ya.ru            или val.bmstu.ru
  New group: External Hosts
  Items 
    Name: check perf http
    Type: Simple check
    Key: net.tcp.service.perf[https]
    Type of information: Numeric (float)

Host: server.corpX.un
...
  Items
    Name: check ping gate.isp.un
    Type: Simple check
    Key: icmpping[gate.isp.un]
    Applications: Network check

Host: server.corpX.un
...
  Items
    Name: check rdp admin/host windows
    Type: Simple check
    Key: net.tcp.service[tcp,192.168.X.5,3389]
#    Key: net.tcp.service[tcp,192.168.X.29,3389]
    Applications: Service check

• Zabbix Documentation Внешние проверки

server# zabbix_server --help | grep ExternalScripts

server# cat /etc/zabbix/zabbix_server.conf

...
Timeout=30
...
ExternalScripts=/etc/zabbix/externalscripts
...

server# mkdir /etc/zabbix/externalscripts

Пример 1

server# cat /etc/zabbix/externalscripts/ping_avg.sh

#!/bin/sh
ping -c"$1" "$2" | tail -n1 | cut -d'/' -f5

server# /etc/zabbix/externalscripts/ping_avg.sh 3 ya.ru

Cofiguration->Hosts->ya.ru или val.bmstu.ru
  Items
    Name: Ping AVG
    Type: External Check
    Key: ping_avg.sh[3,"{HOST.CONN}"]
    Type of information: Numeric (float)
    Units: ms

Пример 2

• Сервис speedtest

server# cat /etc/zabbix/externalscripts/speedtest.sh

#!/bin/sh

if [ "x$1" = xupload ]
then
        A="--no-download"
        F=8
elif [ "x$1" = xdownload ]
then
        A="--no-upload"
        F=7
else
        exit 1
fi

speedtest-cli --csv $A | cut -d',' -f $F

# /etc/zabbix/externalscripts/speedtest.sh upload

# /etc/zabbix/externalscripts/speedtest.sh download

Cofiguration->Hosts->server.corpX.un
  Items
    Name: speedtest download
    Type: External Check
    Key: speedtest.sh[download]
    Type of information: Numeric (float)
    Units: Бит/сек
    Update interval: 30m

...    
    Name: speedtest upload
...

• Утилита nmap
• Пакет sudo

!!! Для некоторых хостов (например, val.bmstu.ru) не хватит Timeout

server# cat /etc/zabbix/externalscripts/detect_host_nmap.sh

#!/bin/sh
sudo /usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done'

Cofiguration->Hosts->gate
  Items
    Name: Detect host operating system by nmap
    Type: External Check
    Key: detect_host_nmap.sh["{HOST.CONN}"]
    Type of information: Text

server# service zabbix-server stop
server# service zabbix-agent stop

• Назначение пользователю домашнего каталога

server# service zabbix-server start
server# service zabbix-agent start

server# sudo -u zabbix bash

zabbix@server:~$ ssh-keygen

zabbix@server:~$ ssh-copy-id root@gate

zabbix@server:~$ ssh root@gate hostname

• Статистика DHCP сервера

zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh CUR

zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh MAX

server# cat /etc/zabbix/externalscripts/dhcp_stat_ext.sh

#!/bin/sh
ssh root@$1 /usr/local/bin/dhcp_stat.sh $2

zabbix@server:~$ /etc/zabbix/externalscripts/dhcp_stat_ext.sh gate CUR

gate->Items
  Name: DHCP stat CUR
  Type: External check
  Key: dhcp_stat_ext.sh["{HOST.CONN}",CUR]

Пример 1

HOSTNAME_IN_CONFIG->Items
  Name: my item
    Type: Zabbix trapper
    Key:  my.item
    Allowed hosts: 127.0.0.1, 192.168.X.0/24

# apt install zabbix-sender

$ zabbix_sender -z IP/DNSNAME -p 10051 -s HOSTNAME_IN_CONFIG -k my.item -o 1

Пример 2

server.corpX.un->Items
  Name: speedtest download trap
    Type: Zabbix trapper
    Key:  speedtest.download
    Type of information: Numeric (float) или Numeric (unsigned)
    Units: бит/с  или  bit/s
    Allowed hosts: 127.0.0.1
  Preprocessing    может понадобиться
    Custom multiplier: 8  
...
  Name: speedtest upload trap
...

# cat /root/speedtest.sh

#!/bin/sh

### iperf ###
MY_RES=$(iperf -c gate.isp.un -t 5 -R -f b -y c)
MY_DOWNLOAD=$(echo $MY_RES | cut -d',' -f9)
MY_RES=$(iperf -c gate.isp.un -t 5 -f b -y c)
MY_UPLOAD=$(echo $MY_RES | cut -d',' -f9)

### speedtest-cli ### result bits/s
#MY_RES=`speedtest-cli --csv`
#MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f7`
#MY_UPLOAD=`echo $MY_RES | cut -d',' -f8`

### speedtest ### result Bytes/s (use preprocess Custom multiplier)
#MY_RES=`speedtest -f csv`
#MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f6`
#MY_UPLOAD=`echo $MY_RES | cut -d',' -f7`

zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.download -o $MY_DOWNLOAD
zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.upload -o $MY_UPLOAD

# crontab -l

...
X * * * * /root/speedtest.sh >/dev/null 2>&1

val.bmstu.ru->Items
  Name: my nmap
    Type: Zabbix trapper
    Key:  my.nmap
    Type of information: Text
    Allowed hosts: 127.0.0.1

server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(nmap -O val.bmstu.ru)"

server# cat /root/detect_host_nmap.sh

#!/bin/sh
/usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done\|Network Distance'

server# chmod +x /root/detect_host_nmap.sh

server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(/root/detect_host_nmap.sh val.bmstu.ru)"

ya.ru->Items
  Name: avg perf http
    Type: Calculated
    Key:  my.avg.perf.http
    Formula: avg(net.tcp.service.perf[https],5m)
    Type of information: Numeric (float)

#---------------------

gate.corpX.un->Items
  Name: DHCP stat CUR
    Type: Zabbix agent
    Key: dhcp.stat[CUR]

  Name: DHCP stat MAX
    Type: Zabbix agent
    Key: dhcp.stat[MAX]
    
  Name: DHCP stat CUR MAX percent
    Type: Calculated
    Key:  DHCP.stat.CUR.MAX.percent
    Formula: last(dhcp.stat[CUR])/last(dhcp.stat[MAX])*100
    
#---------------------

last(openvpn1:openvpn.server.clients)+last(openvpn2:openvpn.server.clients)+last(openvpn3:openvpn.server.clients)

• Сценарий из реальной жизни
• Zabbix web scenario debug

• !!! How can I debug a HTTP POST in Chrome?

Шаг 1.

Браузер: http://192.168.X.10/mail/

view-source
...
...Roundcube Webmail...
...
<input type="hidden" name="_token" value="29JVrZhgW97xID7K2pkSRRHsngGDRGCY">
...

Шаг 2.
Браузер: вводим логин/пароль и нажимаем "Войти"

server# tcpdump -n -A port 80 | tee dump2.txt
...
POST /mail/?_task=login HTTP/1.1
...
_token=29JVrZhgW97xID7K2pkSRRHsngGDRGCY&_task=login&_action=login&_timezone=Europe%2FMoscow&_url=&_user=student&_pass=password
...
HTTP/1.1 302 Found
...
Location: ./?_task=mail&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP
...

view-source
...
...button-logout...
...
<input type="hidden" name="_token" value="pWUje42O61E2Rm0r8zgKzOPXWGby8ugP">
...

3.
Браузер: нажимаем "Выход"

server# tcpdump -n -A port 80 | tee dump3.txt
...
GET /mail/?_task=logout&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP HTTP/1.1
...

Name: mail corpX
Variables
  {login} student
  {password} password

Steps

Step 1
  Name: First page
  URL: http://server.corpX.un/mail/

  Variables
    {token1} regex:name="_token" value="([0-9A-Za-z]{32})"
Можно проще:
    {token1} regex:name="_token" value="(.{32})"

  Required string: rcmloginsubmit
  Required status codes: 200
  
Step 2
  Name: Log in
  URL: http://server.corpX.un/mail/
  
  Post fields
    _token: {token1}
    _task: login
    _action: login
    _user: {login}
    _pass: {password}

  Variables
    {token2}: regex:name="_token" value="(.{32})"
    
  Follow redirects: YES
  
  Required string: button-logout
  Required status codes: 200
  
Step 3
  Name: Log out
  URL: http://server.corpX.un/mail/
  
  Query fields
    _task: logout
    _token: {token2}
    
  Required string: rcmloginsubmit
  Required status codes: 200

• Configuring a trigger

• https://www.zabbix.com/documentation/7.0/ru/manual/config/triggers/expression

Host: server.corpX.un
    Name: gate.isp.un is unreachable
    Expression: {server.corpX.un:icmpping[gate.isp.un].last()}=0    Zabbix < 5.4
    Expression: last(/server.corpX.un/icmpping[gate.isp.un])=0     Zabbix >= 5.4
    Severity: High

    Name: RDP service on host is not available
    Expression: {server.corpX.un:net.tcp.service[tcp,192.168.X.5,3389].last()}=0
    Expression: last(/server.corpX.un/net.tcp.service[tcp,192.168.X.29,3389])=0
    Severity: Warning

Host: ya.ru 
    Name: HTTPS service on {HOST.NAME} is not available
    Expression: {ya.ru:net.tcp.service.perf[https].max(#2)}=0
    Expression: max(/ya.ru/net.tcp.service.perf[https],#2)=0

    Dependencies: Zabbix server: gate.isp.un is unreachable   
        !!!Можно добавить после эксперимента с недоступностью gate.isp.un и двумя, сработавшими триггерами

    Severity: Average

Host: ya.ru                                       
    Name: HTTP service on {HOST.NAME} is slow
    
    Значение подобрать на основе графика
    
    Expression: {ya.ru:my.avg.perf.http.last()}>0.4
    Expression: {ya.ru:net.tcp.service.perf[https].avg(5m)}>0.4  
    Expression: avg(/ya.ru/net.tcp.service.perf[https],5m)>0.4

    Severity: Warning

!!! Добавить в поле “Operational data” информацию о текущем и максимальном количестве адресов

    Name: On {HOST.NAME} dhcp subnet is full
    Problem expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100>{$DHCP.POOLS.MAX.PERCENT}
    Recovery expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100<{$DHCP.POOLS.OK.PERCENT}
    Description:
      В заканчиваются адреса.
      Израсходовано более {$DHCP.POOLS.MAX.PERCENT} процентов.

Zabbix 5

Name: Host {HOST.NAME} nmap change
Severity: Warning

Expression: {gate.corpX.un:detect_host_nmap.sh["{HOST.CONN}"].diff()}=1
  или
Expression: {val.bmstu.ru:my.nmap.diff()}=1

Zabbix 6

Name: installed soft on host {HOST.NAME} change
Severity: Warning

Expression: (last(/Template OS Windows list installed soft Active/listinstalledsoft,#1)<>last(/Template OS Windows list installed soft Active/listinstalledsoft,#2))=1

• Элементы данных веб-мониторинга

Name: Web scenario mail corpX FAIL
Severity: Hight
Expression: {server.corpX.un:web.test.fail[mail corpX].last()}>0
Expression: last(/server.corpX.un/web.test.fail[mail corpX])>0

Configuration->Hosts->ya.ru->Graphs->Create graph             или val.bmstu.ru
  Name: perf http
  Y axis MIN value: Fixed 0
  Items
    ya.ru: avg perf http                                      или val.bmstu.ru
    ya.ru: check perf http                                    или val.bmstu.ru
  Draw style: Bold Line

• Download and install Zabbix Packages

# apt install zabbix-agent

• Download and install pre-compiled Zabbix agents
• http://val.bmstu.ru/unix/zabbix/zabbix_agent-4.0.17-windows-amd64-openssl.msi

# zabbix_agentd -p

# zabbix_agentd -p | grep agent.version

• ПАРАМЕТРЫ VM.MEMORY.SIZE

# zabbix_agentd -p | grep vm.memory.size

# zabbix_agentd -t vm.memory.size[available]

# cat /proc/meminfo | grep MemAvailable

# zabbix_agentd -t system.sw.packages

# dpkg -l

Проверка связи с агентом:

# apt install zabbix-get

$ zabbix_get -s IP/DNSNAME -k agent.version #-p 10050

Минимальная конфигурация агента, включая PSK

# cat /etc/zabbix/zabbix_agentd.conf

или

# cat `echo /etc/zabbix/zabbix_agentd.*d/`corpX.conf

Server=server

# ListenPort=10050

# Hostname=Zabbix server
# must match hostname as configured in Zabbix
# Иначе в журнале будут сообщения: cannot send list of active checks

#TLSConnect=psk
#TLSAccept=psk
##TLSAccept=unencrypted,psk
#TLSPSKFile=/etc/zabbix/zabbix_agentd.psk
#TLSPSKIdentity=gate

gate# openssl rand -hex 32 > /etc/zabbix/zabbix_agentd.psk

gate# scp /etc/zabbix/zabbix_agentd.psk server:gate.psk

gate# service zabbix-agent restart

server# zabbix_get -s gate -k system.sw.packages --tls-connect=psk --tls-psk-identity="gate" --tls-psk-file=gate.psk

• How to configure an Active Agent in Zabbix 2.2

Alerts|Configuration - Actions - Auto registration 
  Name: Add Windows clients                                          # or Add Linux clients
  Conditions: Host name contains CLIENT                              # or client (lowercase) for linux
  Action operations: 
    Add to host groups: Windows clients                              # or Linux clients
    Link to templates: Windows by Zabbix agent active                # or Linux by Zabbix agent active
                     Template OS Windows list installed soft Active  # work in linux too))
  Set host inventory mode: Automatic

LogFile=C:\Program Files\Zabbix Agent\zabbix_agentd.log
#Server=server
ListenIP=0.0.0.0
StartAgents=0
ServerActive=server
#Hostname=CLIENTN

/var/log/cisco_routers.log:
Jun 10 15:06:23 192.168.32.224 4278: Jun 10 15:06:22: %IP-4-DUPADDR: Duplicate address 192.168.0.254 on Port-channel6.664, sourced by 30de.4bf5.e9cc
Jun 10 16:25:07 192.168.32.224 4308: Jun 10 16:25:06: %SYS-5-CONFIG_I: Configured from console by val on vty0 (192.168.32.6)
last(/helper.bmstu.ru/log[/var/log/cisco_routers.log,Duplicate address])<>0

• Статистика DHCP сервера

gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf

UserParameter=dhcp.stat[*],/usr/local/bin/dhcp_stat.sh $1

server# zabbix_get -s gate -k dhcp.stat[CUR]
server# zabbix_get -s gate -k dhcp.stat[MAX]

• Список desktop приложений

linclient2:~# cat /etc/zabbix/zabbix_agentd.conf.d/listinstalledsoft.conf

UserParameter=listinstalledsoft,ls /usr/share/applications | awk -F '.desktop' ' { print $1}' -
# UserParameter=listinstalledsoft,ls /usr/share/applications /usr/local/share/applications | awk -F '.desktop' ' { print $1}' -

• Список установленного ПО

Admin C:\> C:\Program Files\Zabbix Agent\zabbix_agentd.conf

...
Timeout=30

#UserParameter=listinstalledsoft,powershell -Command Get-ChildItem HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall; powershell -Command Get-ChildItem HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall

UserParameter=listinstalledsoft,C:\bin\listinstalledsoft.bat | findstr /v "^$"

UnsafeUserParameters=1
UserParameter=dir[*],dir $1
UserParameter=runcommand[*],$1
UserParameter=lmstat[*],C:\Progra~1\PTC\flexnet\bin\lmutil lmstat -a -c $1

C:\>notepad C:\bin\listinstalledsoft.bat

@echo off

powershell -command "Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"
powershell -command "Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"

• Была необходима Установка snmp консоли

# service zabbix-server restart

• Network discovery см. Discovery status

Data collection/Configuration->Discovery
Discovery rules
  Name: Local network    или    Cisco Router Discovery
  IP range: 172.16.1.190-199, 192.168.X.51-60
  Checks: 
    Check type: SNMPv2 agent 
    SNMP community: public
    SNMP OID .1.3.6.1.2.1.1.5.0
      или
             SNMPv2-MIB::sysName.0       !!! Требуется установка MIB
    Add
  Update interval: 3m

  Host name: SNMPv2 agent ...            !!! Остальные варианты требуют PTR записи

  Enabled: yes

Alert/Configuration->Actions
  Event source: Discovery
    Name: Action add Cisco device to zabbix
    Conditions: 
      Received value: like или contains router
      Discovery status: equals Up                 без этого появляются ghost hosts :)
      Add
    Operations: 
      Add to host groups: Cisco devices             !!! В Discovered hosts добавится автоматически !!!
      Link to templates: 
             3.XX Template SNMP Device
             4.XX Template Module Generic SNMPv2
             4.XX Template Module Interfaces Simple SNMPv2
             5.XX Template Net Cisco IOS SNMP       !!! Уменьшить период Network interfaces discovery !!!
        6.XX 7.XX Cisco IOS by SNMP
      Set host inventory mode: Automatic
    Add

• Monitoring network hardware with SNMPv3 in Zabbix

• Zabbix Documentation Проверки через SSH

• Настройка Zabbix SNMP traps
• Zabbix Documentation SNMP трапы
• SNMPTT installation on CentOS 7 for Zabbix integration
• How to deploy SNMP Traps in Centos8
• catch all snmp traps with general event
• SNMP Traps in Zabbix
• youtube SNMP Traps in Zabbix Tutorial

# apt install snmptt

# systemctl disable snmptt

# systemctl stop snmptt

# cat /etc/snmp/snmptt.conf

EVENT general .* "General event" Normal
FORMAT ZBXTRAP $aA $ar

# cat /etc/snmp/snmptt.ini

...
date_time_format = %H:%M:%S %Y/%m/%d
...
#log_file = /var/log/snmptt/snmptt.log
log_file = /tmp/my_zabbix_traps.tmp
...

# cat /etc/snmp/snmptrapd.conf

traphandle default snmptt

authCommunity execute writetrap

# cat /lib/systemd/system/snmptrapd.service

...
ExecStart=/usr/sbin/snmptrapd -Lsd -f -On
...

# systemctl daemon-reload
# service snmptrapd restart

• Пример с использованием макроса
• ПРОСТЫЕ ПРОВЕРКИ

Configuration
  Templates
    SSH Service
  Full Clone
    SSH Port Service
  Add

  Templates
    Name: SSH Port Service
      Template groups: Templates/Applications
      Items
        Name: SSH service is running
        Key: net.tcp.service[ssh,,{$SSH_PORT}]
        Update interval: 30s
      Triggers
        Name: SSH service is down on {HOST.NAME}
        Severity: Average
        Expression: max(/SSH Port Service/net.tcp.service[ssh,,{$SSH_PORT}],#3)=0
      Macros
        {$SSH_PORT}=22

• Zabbix Documentation Низкоуровневое обнаружение
• Автоматизируем мониторинг: низкоуровневое обнаружение
• Perl script Авторисования топологии сети в Zabbix
• Видеоурок - Самый простой, но очень полезный пример использования низкоуровнего обнаружения (LLD) в Zabbix

server# zabbix_get -s gate -k vfs.fs.discovery

• Утилита jq

server# zabbix_get -s gate -k vfs.fs.size[/,free]
server# zabbix_get -s gate -k vfs.fs.size[/,total]
server# zabbix_get -s gate -k vfs.fs.size[/,used]
...

server# zabbix_get -s gate -k net.if.discovery | jq
...

server# zabbix_get -s gate -k net.if.in[eth0]
...

• Пример использования встроенных средств для LLD

agent# zabbix_agentd -p | grep vfs.dev
...

server# zabbix_get -s agent -k "vfs.dev.read[sda,sps]"
...

agent# cat /etc/zabbix/zabbix_agentd.conf.d/my.linux.disk.discovery.conf

UserParameter=my.disks.discovery,/bin/lsblk -dJ | /bin/sed -e 's/blockdevices/data/' -e 's/name/{#NAME}/g' -e 's/type/{#TYPE}/g'

server# zabbix_get -s agent -k my.disks.discovery | jq
...

Configuration->Templates->Create template
  Template name: My Template Linux disks utilization
  Groups: Templates/Server hardware

  Discovery rules->
    Name: my disks discovery
    Key: my.disks.discovery
    Filters->
      {#TYPE} matches disk
    Item prototypes->
      Name: disk {#NAME} read bytes
      Key: vfs.dev.read[{#NAME},sps]
      Type of information: Numeric (float)
      Units: Байт
      Preprocessing->
        Preprocessing steps
        Custom multiplier 512

      Name: disk {#NAME} write bytes
      ...  

• Пример разработки собственного скрипта для LLD

• Сервис DHCP Статистика DHCP сервера
• Обработка XML файлов
• Output JSON from Bash script

• Язык программирования Python Zabbix LLD приложение

gate# cat /etc/zabbix/dhcp-pools-discovery.sh

#!/bin/bash

echo -n '['

str=`/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
/usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
-o '{"{#POOLNAME}":"' -v location -o '"},'`

echo -n ${str::-1}

echo -n ']'

gate# /etc/zabbix/dhcp-pools-discovery.sh | jq

gate# cat /etc/zabbix/dhcp-pools-shared-network.sh

#!/bin/sh

res_field=2
test "x$2" = "xused" && res_field=3

/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
  /usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
  -v location -o ' ' -v defined -o ' ' -v used -n | \
  grep $1 | cut -d ' ' -f $res_field

gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN1 defined

gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN2 used

gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf

UserParameter=dhcp.pools.discovery,/etc/zabbix/dhcp-pools-discovery.sh

UserParameter=dhcp.pools.shared-network[*],/etc/zabbix/dhcp-pools-shared-network.sh $1 $2

server# zabbix_get -s gate -k dhcp.pools.discovery | jq

server# zabbix_get -s gate -k dhcp.pools.shared-network[LAN1,used]

Configuration->Templates->Create template
  Template name: Template App DHCP Pools
  Groups In groups: Templates/Applications

  Macros: {$DHCP.POOLS.MAX.PERCENT}=90
Add

  Applications: DHCP
  
  Discovery rules
    Name: Search DHCP Pools
    Type: Zabbix Agent
    Key: dhcp.pools.discovery
  Add
    Item prototypes
      Name: DHCP Pool $1 max addr      или DHCP Pool {#POOLNAME} max addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},defined]
      Applications: DHCP
    Add

      Name: DHCP Pool $1 cur addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},used]
      Applications: DHCP
    Add

    Graph prototypes
      Name: DHCP Pool {#POOLNAME} max cur
      Y axis MIN value: Fixed 0
      Items: 
        Template App DHCP Pools: DHCP Pool {#POOLNAME} cur addr
        Template App DHCP Pools: DHCP Pool {#POOLNAME} max addr

    Trigger prototypes

      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME}
или
      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME} used more {$DHCP.POOLS.MAX.PERCENT} percent
      
      Expression: {Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},used].last()}/{Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},defined].last()}*100 > {$DHCP.POOLS.MAX.PERCENT}
                  last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},used])/last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},defined])*100 > {$DHCP.POOLS.MAX.PERCENT}
      Severity: Warning

Configuration
  Templates
    Export/Import

• https://www.zabbix.com/documentation/current/en/manual/api
• Формат JSON
• Zabbix Documentation Начало работы с Zabbix API

• api key lifetime
• Для и до 6-й версии “user”: “Admin”,

server:~# apt install curl

server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
   "jsonrpc": "2.0",
   "method": "user.login",
   "params": {
      "username": "Admin",
      "password": "zabbix"
   },
   "id": 1
} ' http://127.0.0.1/zabbix/api_jsonrpc.php

{"jsonrpc":"2.0","result":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","id":1}

• Утилита для обработки JSON

server:~# apt install jq

• Получение списка узлов и шаблонов из Zabbix

До 7-й версии, включительно

server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {},
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq
...

От 7-й и старше

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -H 'Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' -d '{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {},
    "id": 2
}' http://127.0.0.1/zabbix/api_jsonrpc.php | jq

• Zabbix JSON API output parameter
• Пример запроса определенных атрибутов и с фильтром

...
    "params": {
        "output": ["hostid", "host"],
        "templateids": ["10NNN"]
    },
...

• Zabbix Documentation HISTORY.GET

• history 0-число 4-текст

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "history.get",
    "params": {
        "output": "extend",
        "history": 0,
        "itemids": "NNNNN",
        "sortfield": "clock",
        "sortorder": "DESC",
        "limit": 10
    },
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq

server:~# export AUTH=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

server:~# cat /root/zab_get_hosts.sh

#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"host.get\",
    \"params\": {},
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php

server:~# /root/zab_get_hosts.sh | jq

Список имен узлов
server:~# /root/zab_get_hosts.sh | jq '.result | .[] | .name'

server.corpX.un:~# cat /root/zab_get_maps.sh

#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.get\",
    \"params\": {
        \"selectLinks\": \"extend\",
        \"selectSelements\": \"extend\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php

server.corpX.un:~# /root/zab_get_maps.sh | jq -c '.result | .[] | {name: .name, id: .sysmapid}'

• Язык программирования Python Zabbix API приложение

server.corp1.un:~# cat /root/zab_set_map_name.sh

#!/bin/sh

MAPID=$1
MAPNAME=$2

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.update\",
    \"params\": {
        \"sysmapid\": \"${MAPID}\",
        \"name\": \"${MAPNAME}\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php

server.corp1.un:~# /root/zab_set_map_name.sh 2 "ISP 1"