Table of Contents

Сервис Zabbix

Документация

Установка и запуск сервера

Установка SQL сервера

Установка из репозитория производителя

Установка из репозитория Debian

# apt install zabbix-server-mysql   #2m

# less /usr/share/doc/zabbix-server-mysql/README.Debian

# cat zabbix.sql
#drop database zabbix;
create database zabbix character set utf8 collate utf8_bin;

#debian11
#grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix';

#ubuntu20
#create  USER zabbix@localhost identified by 'zabbix';
#grant all privileges on zabbix.* to zabbix@localhost;
# mysql < zabbix.sql
# zcat /usr/share/zabbix-server-mysql/{schema,images,data}.sql.gz | mysql -uzabbix -pzabbix zabbix  

# cat /etc/zabbix/zabbix_server.conf.d/corpX.conf
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
#end newline!!!
# systemctl enable zabbix-server

# service zabbix-server start

Установка и запуск web интерфейса

Все следующие компоненты поставятся, как зависимости:

Debian

# apt install zabbix-frontend-php php-mysql

# less /usr/share/doc/zabbix-frontend-php/README.Debian

# cat /etc/apache2/conf-available/zabbix-frontend-php.conf
...
php_value date.timezone Europe/Moscow
...
php_value date.timezone Europe/Moscow
...
# a2enconf zabbix-frontend-php

# service apache2 reload

# touch /etc/zabbix/zabbix.conf.php

# chown www-data /etc/zabbix/zabbix.conf.php
http://server.corpX.un/zabbix/setup.php

!!! После работы визарда рекомендуется 

# mv /usr/share/zabbix/setup.php /usr/share/zabbix/setup.php_stop

что бы никто его не повторил)
# less /etc/zabbix/zabbix.conf.php
<?php
// Zabbix GUI configuration file.
global $DB;

$DB['TYPE']     = 'MYSQL';
$DB['SERVER']   = 'localhost';
$DB['PORT']     = '0';
$DB['DATABASE'] = 'zabbix';
$DB['USER']     = 'zabbix';
$DB['PASSWORD'] = 'zabbix';

// Schema name. Used for IBM DB2 and PostgreSQL.
$DB['SCHEMA'] = '';

$ZBX_SERVER      = 'localhost';
$ZBX_SERVER_PORT = '10051';
$ZBX_SERVER_NAME = '';

$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;

Установка и запуск proxy

sqlite3

Из репозитория Zabbix

# apt install zabbix-proxy-sqlite3

# mkdir /var/lib/zabbix

# В 6-й версии не нужно, БД будет создана автоматически
# zcat /usr/share/doc/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db

Из репозитория Debian

# apt install zabbix-proxy-sqlite3

# less /usr/share/doc/zabbix-proxy-sqlite3/README.Debian

# zcat /usr/share/zabbix-proxy-sqlite3/schema.sql.gz | sqlite3 /var/lib/zabbix/zabbix.db

Настройка и запуск

# chown -R zabbix:zabbix /var/lib/zabbix

# cat /etc/zabbix/zabbix_proxy.conf
...
Hostname=gate
ConfigFrequency=60
Server=server
DBName=/var/lib/zabbix/zabbix.db

MySQL

Из репозитория Debian

gate# apt install zabbix-proxy-mysql

gate# cat zabbix_proxy.sql
#drop database zabbix_proxy;
create database zabbix_proxy character set utf8 collate utf8_bin;
grant all privileges on zabbix_proxy.* to zabbix@localhost identified by 'zabbix';
gate# mysql < zabbix_proxy.sql

gate# zcat /usr/share/zabbix-proxy-mysql/schema.sql.gz | mysql -uzabbix -pzabbix zabbix_proxy

gate# cat /etc/zabbix/zabbix_proxy.conf
...
Hostname=gate
ConfigFrequency=60
Server=server
DBHost=localhost
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix
gate# systemctl enable zabbix-proxy

gate# service zabbix-proxy start

Добавление proxy в zabbix server

Administration->Proxies
  Proxy name: gate
  Proxy mode: Active
  Proxy address: gate

Обновление Zabbix

Настройка мониторинга

http://192.168.X.10/zabbix/

Настройка уведомлений о событиях

Настройка подсистемы Email

Administration->Media types->Email
  SMTP server: server.corpX.un
  SMTP helo: server.corpX.un
  SMTP email: zabbix@corpX.un

Administration->Users->student->Media
  Type: Email
  Send to: student@corpX.un

Настройка уведомлений о срабатывании триггеров

Configuration->Actions->Event source: Triggers
  Report problems to Zabbix administrators->Enable
  ...

Использование скриптов

# grep AlertScriptsPath /etc/zabbix/zabbix_server.conf

Пример XMPP

# getent passwd zabbix

# mkdir /var/lib/zabbix/

# cat /var/lib/zabbix/.sendxmpprc
admin@corpX.un Pa$$w0rd

# chmod 600 /var/lib/zabbix/.sendxmpprc

# chown -R zabbix:zabbix /var/lib/zabbix/

# cat /usr/lib/zabbix/alertscripts/notification_xmpp.sh
#!/bin/bash

logger -t zabbix_xmpp -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF | sendxmpp -f /var/lib/zabbix/.sendxmpprc -s "$subject" "$to"
$body
EOF
Administration->Media types->Create media type
  Name: XMPP
  Type: Script
  Script Name: notification_xmpp.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
    Problem recovery
    Problem update
Administration->Users->student->Media
  Type: XMPP
  Send to: student@corpX.un

Пример c Asterisk

# cat /usr/lib/zabbix/alertscripts/notification_pstn.sh
#!/bin/bash

logger -t zabbix_pstn -p local0.info "$1, $2, $3"

to=$1
subject=$2
body=$3

cat <<EOF > /tmp/zabbix.txt
$subject
$body
EOF

cat /tmp/zabbix.txt | text2wave -eval '(voice_msu_ru_nsh_clunits)' > /tmp/zabbix.wav

sox /tmp/zabbix.wav -r 8000 -c 1 /tmp/zabbix.raw

rm /tmp/zabbix.wav

#sudo /usr/sbin/asterisk -x "channel originate SIP/smg1016/$to extension 326@default"
sudo /usr/sbin/asterisk -x "channel originate SIP/$to extension 326@default"
Administration->Media types->Create media type
  Name: PSTN
  Type: Script
  Script Name: notification_pstn.sh
  Script parameters
    {ALERT.SENDTO}
    {ALERT.SUBJECT}
    {ALERT.MESSAGE}
  Message teplates
    Problem
  Message templates->
    Subject: Звонок с работы.
    Message:
Обнаружена проблема: {EVENT.NAME}.
Имя узла: {HOST.NAME}.
Administration->Users->student->Media
  Type: PSTN
  Send to: 8495123456X или 403
  Use if severity
    Disaster

Отчет об отправленных сообщениях

Reports -> Action log

Добавление информации о значении элемента в сообщение

Zabbix 4

Configuration->Actions->Event source: Triggers
  Report problems to Zabbix administrators

    Operations:
      Default message
...
{ITEM.NAME1} ({HOST.NAME1}:{ITEM.KEY1}): 

==CURRENT==

{ITEM.VALUE1}

==PREVIOUS==

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}

Zabbix 5

Administration->Media type->Email->Message templates
  Problem

...
Current:

{ITEM.VALUE1}

Previous:

{{HOSTNAME}:{TRIGGER.KEY}.last(#2)}

Zabbix 6

Простые проверки

Host: ya.ru                         или val.bmstu.ru
  ЛЮБОЙ ИНТЕРФЕЙС: ya.ru            или val.bmstu.ru
  New group: External Hosts
  Items 
    Name: check perf http
    Type: Simple check
    Key: net.tcp.service.perf[https]
    Type of information: Numeric (float)
Host: server.corpX.un
...
  Items
    Name: check ping gate.isp.un
    Type: Simple check
    Key: icmpping[gate.isp.un]
    Applications: Network check

Host: server.corpX.un
...
  Items
    Name: check rdp admin/host windows
    Type: Simple check
    Key: net.tcp.service[tcp,192.168.X.5,3389]
#    Key: net.tcp.service[tcp,192.168.X.29,3389]
    Applications: Service check

Внешние проверки

server# zabbix_server --help | grep ExternalScripts

server# cat /etc/zabbix/zabbix_server.conf
...
Timeout=30
...
ExternalScripts=/etc/zabbix/externalscripts
...
server# mkdir /etc/zabbix/externalscripts

Пример простого скрипта

Пример 1

server# cat /etc/zabbix/externalscripts/ping_avg.sh
#!/bin/sh
ping -c"$1" "$2" | tail -n1 | cut -d'/' -f5
server# /etc/zabbix/externalscripts/ping_avg.sh 3 ya.ru
Cofiguration->Hosts->ya.ru или val.bmstu.ru
  Items
    Name: Ping AVG
    Type: External Check
    Key: ping_avg.sh[3,"{HOST.CONN}"]
    Type of information: Numeric (float)
    Units: ms

Пример 2

server# cat /etc/zabbix/externalscripts/speedtest.sh
#!/bin/sh

if [ "x$1" = xupload ]
then
        A="--no-download"
        F=8
elif [ "x$1" = xdownload ]
then
        A="--no-upload"
        F=7
else
        exit 1
fi

speedtest-cli --csv $A | cut -d',' -f $F
# /etc/zabbix/externalscripts/speedtest.sh upload

# /etc/zabbix/externalscripts/speedtest.sh download
Cofiguration->Hosts->server.corpX.un
  Items
    Name: speedtest download
    Type: External Check
    Key: speedtest.sh[download]
    Type of information: Numeric (float)
    Units: Бит/сек
    Update interval: 30m

...    
    Name: speedtest upload
...

Пример скрипта, требующего повышения привилегий

!!! Для некоторых хостов (например, val.bmstu.ru) не хватит Timeout

server# cat /etc/zabbix/externalscripts/detect_host_nmap.sh
#!/bin/sh
sudo /usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done'
Cofiguration->Hosts->gate
  Items
    Name: Detect host operating system by nmap
    Type: External Check
    Key: detect_host_nmap.sh["{HOST.CONN}"]
    Type of information: Text

Пример запуска скрипта на удаленной системе

server# service zabbix-server stop
server# service zabbix-agent stop
server# service zabbix-server start
server# service zabbix-agent start

server# sudo -u zabbix bash

zabbix@server:~$ ssh-keygen

zabbix@server:~$ ssh-copy-id root@gate

zabbix@server:~$ ssh root@gate hostname
zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh CUR

zabbix@server:~$ ssh root@gate /usr/local/bin/dhcp_stat.sh MAX

server# cat /etc/zabbix/externalscripts/dhcp_stat_ext.sh
#!/bin/sh
ssh root@$1 /usr/local/bin/dhcp_stat.sh $2
zabbix@server:~$ /etc/zabbix/externalscripts/dhcp_stat_ext.sh gate CUR
gate->Items
  Name: DHCP stat CUR
  Type: External check
  Key: dhcp_stat_ext.sh["{HOST.CONN}",CUR]

Элементы типа trapper

Пример численного элемента

Пример 1

HOSTNAME_IN_CONFIG->Items
  Name: my item
    Type: Zabbix trapper
    Key:  my.item
    Allowed hosts: 127.0.0.1, 192.168.X.0/24
# apt install zabbix-sender

$ zabbix_sender -z IP/DNSNAME -p 10051 -s HOSTNAME_IN_CONFIG -k my.item -o 1

Пример 2

server.corpX.un->Items
  Name: speedtest download trap
    Type: Zabbix trapper
    Key:  speedtest.download
    Type of information: Numeric (float) или Numeric (unsigned)
    Units: бит/с  или  bit/s
    Allowed hosts: 127.0.0.1
  Preprocessing    может понадобиться
    Custom multiplier: 8  
...
  Name: speedtest upload trap
...
# cat /root/speedtest.sh
#!/bin/sh

### speedtest-cli ### result bits/s
MY_RES=`speedtest-cli --csv`
MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f7`
MY_UPLOAD=`echo $MY_RES | cut -d',' -f8`

### speedtest ### result Bytes/s (use preprocess Custom multiplier)
#MY_RES=`speedtest -f csv`
#MY_DOWNLOAD=`echo $MY_RES | cut -d',' -f6`
#Y_UPLOAD=`echo $MY_RES | cut -d',' -f7`

zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.download -o $MY_DOWNLOAD
zabbix_sender -z 127.0.0.1 -p 10051 -s server.corpX.un -k speedtest.upload -o $MY_UPLOAD
# crontab -l
...
X * * * * /root/speedtest.sh >/dev/null 2>&1

Пример текстового элемента

val.bmstu.ru->Items
  Name: my nmap
    Type: Zabbix trapper
    Key:  my.nmap
    Type of information: Text
    Allowed hosts: 127.0.0.1
server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(nmap -O val.bmstu.ru)"
server# cat /root/detect_host_nmap.sh
#!/bin/sh
/usr/bin/nmap -O $1 | grep -v 'Starting Nmap\|Host is up\|Nmap done\|Network Distance'
server# chmod +x /root/detect_host_nmap.sh
server# zabbix_sender -z 127.0.0.1 -p 10051 -s val.bmstu.ru -k my.nmap -o "$(/root/detect_host_nmap.sh val.bmstu.ru)"

Вычисляемые элементы

ya.ru->Items
  Name: avg perf http
    Type: Calculated
    Key:  my.avg.perf.http
    Formula: avg(net.tcp.service.perf[https],5m)
    Type of information: Numeric (float)

#---------------------

gate.corpX.un->Items
  Name: DHCP stat CUR
    Type: Zabbix agent
    Key: dhcp.stat[CUR]

  Name: DHCP stat MAX
    Type: Zabbix agent
    Key: dhcp.stat[MAX]
    
  Name: DHCP stat CUR MAX percent
    Type: Calculated
    Key:  DHCP.stat.CUR.MAX.percent
    Formula: last(dhcp.stat[CUR])/last(dhcp.stat[MAX])*100
    
#---------------------

last(openvpn1:openvpn.server.clients)+last(openvpn2:openvpn.server.clients)+last(openvpn3:openvpn.server.clients)

Web сценарии

Исследование приложения

Шаг 1.

Браузер: http://192.168.X.10/mail/

view-source
...
...Roundcube Webmail...
...
<input type="hidden" name="_token" value="29JVrZhgW97xID7K2pkSRRHsngGDRGCY">
...

Шаг 2.
Браузер: вводим логин/пароль и нажимаем "Войти"

server# tcpdump -n -A port 80 | tee dump2.txt
...
POST /mail/?_task=login HTTP/1.1
...
_token=29JVrZhgW97xID7K2pkSRRHsngGDRGCY&_task=login&_action=login&_timezone=Europe%2FMoscow&_url=&_user=student&_pass=password
...
HTTP/1.1 302 Found
...
Location: ./?_task=mail&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP
...

view-source
...
...button-logout...
...
<input type="hidden" name="_token" value="pWUje42O61E2Rm0r8zgKzOPXWGby8ugP">
...

3.
Браузер: нажимаем "Выход"

server# tcpdump -n -A port 80 | tee dump3.txt
...
GET /mail/?_task=logout&_token=pWUje42O61E2Rm0r8zgKzOPXWGby8ugP HTTP/1.1
...

Web scenario

Name: mail corpX
Variables
  {login} student
  {password} password

Steps

Step 1
  Name: First page
  URL: http://server.corpX.un/mail/

  Variables
    {token1} regex:name="_token" value="([0-9A-Za-z]{32})"
Можно проще:
    {token1} regex:name="_token" value="(.{32})"

  Required string: rcmloginsubmit
  Required status codes: 200
  
Step 2
  Name: Log in
  URL: http://server.corpX.un/mail/
  
  Post fields
    _token: {token1}
    _task: login
    _action: login
    _user: {login}
    _pass: {password}

  Variables
    {token2}: regex:name="_token" value="(.{32})"
    
  Follow redirects: YES
  
  Required string: button-logout
  Required status codes: 200
  
Step 3
  Name: Log out
  URL: http://server.corpX.un/mail/
  
  Query fields
    _task: logout
    _token: {token2}
    
  Required string: rcmloginsubmit
  Required status codes: 200

Настройка триггеров

Примеры простых целочисленных триггеров

Host: server.corpX.un
    Name: gate.isp.un is unreachable
    Expression: {server.corpX.un:icmpping[gate.isp.un].last()}=0    Zabbix < 5.4
    Expression: last(/server.corpX.un/icmpping[gate.isp.un])=0     Zabbix >= 5.4
    Severity: High

    Name: RDP service on host is not available
    Expression: {server.corpX.un:net.tcp.service[tcp,192.168.X.5,3389].last()}=0
    Expression: last(/server.corpX.un/net.tcp.service[tcp,192.168.X.29,3389])=0
    Severity: Warning

Host: ya.ru 
    Name: HTTPS service on {HOST.NAME} is not available
    Expression: {ya.ru:net.tcp.service.perf[https].max(#2)}=0
    Expression: max(/ya.ru/net.tcp.service.perf[https],#2)=0

    Dependencies: Zabbix server: gate.isp.un is unreachable   
        !!!Можно добавить после эксперимента с недоступностью gate.isp.un и двумя, сработавшими триггерами

    Severity: Average

Пример триггера основанного на значении элемента в течении периода

Host: ya.ru                                       
    Name: HTTP service on {HOST.NAME} is slow
    
    Значение подобрать на основе графика
    
    Expression: {ya.ru:my.avg.perf.http.last()}>0.4
    Expression: {ya.ru:net.tcp.service.perf[https].avg(5m)}>0.4  
    Expression: avg(/ya.ru/net.tcp.service.perf[https],5m)>0.4

    Severity: Warning

Пример с макросами и Recovery expression

!!! Добавить в поле “Operational data” информацию о текущем и максимальном количестве адресов

    Name: On {HOST.NAME} dhcp subnet is full
    Problem expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100>{$DHCP.POOLS.MAX.PERCENT}
    Recovery expression: {Template App DHCP Service:dhcp.stat[CUR].last()}/{Template App DHCP Service:dhcp.stat[MAX].last()}*100<{$DHCP.POOLS.OK.PERCENT}
    Description:
      В заканчиваются адреса.
      Израсходовано более {$DHCP.POOLS.MAX.PERCENT} процентов.

Пример с текстовым элементом

Zabbix 5

Name: Host {HOST.NAME} nmap change
Severity: Warning

Expression: {gate.corpX.un:detect_host_nmap.sh["{HOST.CONN}"].diff()}=1
  или
Expression: {val.bmstu.ru:my.nmap.diff()}=1

Zabbix 6

Name: installed soft on host {HOST.NAME} change
Severity: Warning

Expression: (last(/Template OS Windows list installed soft Active/listinstalledsoft,#1)<>last(/Template OS Windows list installed soft Active/listinstalledsoft,#2))=1

Пример в веб сценарии

Name: Web scenario mail corpX FAIL
Severity: Hight
Expression: {server.corpX.un:web.test.fail[mail corpX].last()}>0
Expression: last(/server.corpX.un/web.test.fail[mail corpX])>0

Пользовательские графики

Configuration->Hosts->ya.ru->Graphs->Create graph             или val.bmstu.ru
  Name: perf http
  Y axis MIN value: Fixed 0
  Items
    ya.ru: avg perf http                                      или val.bmstu.ru
    ya.ru: check perf http                                    или val.bmstu.ru
  Draw style: Bold Line

Мониторинг с использованием Zabbix agents

Установка агента из репозитория вендора

Установка агента из репозитория Debian/Ubuntu

# apt install zabbix-agent

Установка агента в Windows

Список элементов агента

# zabbix_agentd -p

# zabbix_agentd -p | grep agent.version
# zabbix_agentd -p | grep vm.memory.size

# zabbix_agentd -t vm.memory.size[available]

# cat /proc/meminfo | grep MemAvailable

# zabbix_agentd -t system.sw.packages

# dpkg -l

Пассивный режим

Проверка связи с агентом:

# apt install zabbix-get

$ zabbix_get -s IP/DNSNAME -p 10050 -k agent.version

Минимальная конфигурация агента, включая PSK

# cat /etc/zabbix/zabbix_agentd.conf

или

# cat `echo /etc/zabbix/zabbix_agentd.*d/`corpX.conf
Server=server

# ListenPort=10050

# Hostname=Zabbix server
# must match hostname as configured in Zabbix
# Иначе в журнале будут сообщения: cannot send list of active checks

#TLSConnect=psk
#TLSAccept=psk
##TLSAccept=unencrypted,psk
#TLSPSKFile=/etc/zabbix/zabbix_agentd.psk
#TLSPSKIdentity=gate
gate# openssl rand -hex 32 > /etc/zabbix/zabbix_agentd.psk

gate# scp /etc/zabbix/zabbix_agentd.psk server:gate.psk

gate# service zabbix-agent restart

server# zabbix_get -s gate -k system.sw.packages --tls-connect=psk --tls-psk-identity="gate" --tls-psk-file=gate.psk

Активный режим

Настройка авторегистрации систем с агентами, работающими в активном режиме
Configuration - Actions - Auto registration 
  Name: Add Windows clients                                          # or Add Linux clients
  Conditions: Host name contains CLIENT                              # or client (lowercase) for linux
  Action operations: 
    Add to host groups: Windows clients                              # or Linux clients
    Link to templates: Windows by Zabbix agent active                # or Linux by Zabbix agent active
                     Template OS Windows list installed soft Active  # work in linux too))
  Set host inventory mode: Automatic
Настройка агента на активный режим
LogFile=C:\Program Files\Zabbix Agent\zabbix_agentd.log
#Server=server
ListenIP=0.0.0.0
StartAgents=0
ServerActive=server
#Hostname=CLIENTN
/var/log/cisco_routers.log:
Jun 10 15:06:23 192.168.32.224 4278: Jun 10 15:06:22: %IP-4-DUPADDR: Duplicate address 192.168.0.254 on Port-channel6.664, sourced by 30de.4bf5.e9cc
Jun 10 16:25:07 192.168.32.224 4308: Jun 10 16:25:06: %SYS-5-CONFIG_I: Configured from console by val on vty0 (192.168.32.6)
last(/helper.bmstu.ru/log[/var/log/cisco_routers.log,Duplicate address])<>0

Использование UserParameter

Примеры для Linux
gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf
UserParameter=dhcp.stat[*],/usr/local/bin/dhcp_stat.sh $1
server# zabbix_get -s gate -k dhcp.stat[CUR]
server# zabbix_get -s gate -k dhcp.stat[MAX]
linclient2:~# cat /etc/zabbix/zabbix_agentd.conf.d/listinstalledsoft.conf
UserParameter=listinstalledsoft,ls /usr/share/applications | awk -F '.desktop' ' { print $1}' -
# UserParameter=listinstalledsoft,ls /usr/share/applications /usr/local/share/applications | awk -F '.desktop' ' { print $1}' -
Примеры для Windows
Admin C:\> C:\Program Files\Zabbix Agent\zabbix_agentd.conf
...
Timeout=30

#UserParameter=listinstalledsoft,powershell -Command Get-ChildItem HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall; powershell -Command Get-ChildItem HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall

UserParameter=listinstalledsoft,C:\bin\listinstalledsoft.bat | findstr /v "^$"

UnsafeUserParameters=1
UserParameter=dir[*],dir $1
UserParameter=runcommand[*],$1
UserParameter=lmstat[*],C:\Progra~1\PTC\flexnet\bin\lmutil lmstat -a -c $1
C:\>notepad C:\bin\listinstalledsoft.bat
@echo off

powershell -command "Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"
powershell -command "Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Get-ItemProperty | Where-Object 'DisplayName' | Sort-Object -Property DisplayName | Select-Object -Property DisplayName | Format-Table -AutoSize -HideTableHeaders"

Мониторинг по протоколу SNMP

# service zabbix-server restart

Автоматическое добавление

Configuration->Discovery
Discovery rules
  Name: Local network    или    Cisco Router Discovery
  IP range: 172.16.1.190-199, 192.168.X.51-60
  Checks: 
    Check type: SNMPv2 agent 
    SNMP community: public
    SNMP OID .1.3.6.1.2.1.1.5.0
      или
             SNMPv2-MIB::sysName.0       !!! Требуется установка MIB
    Add
  Update interval: 3m

  Host name: SNMPv2 agent ...            !!! Остальные варианты требуют PTR записи

  Enabled: yes

Configuration->Actions
  Event source: Discovery
    Name: Action add snmp device to zabbix
    Conditions: 
      Received value: like или contains router
      Discovery status: equals Up                 без этого появляются ghost hosts :)
      Add
    Operations: 
      Add to host groups: Cisco devices             !!! В Discovered hosts добавится автоматически !!!
      Link to templates: 
             3.XX Template SNMP Device
             4.XX Template Module Generic SNMPv2
             4.XX Template Module Interfaces Simple SNMPv2
             5.XX Template Net Cisco IOS SNMP       !!! Уменьшить период Network interfaces discovery !!!
             6.XX Cisco IOS by SNMP
      Set host inventory mode: Automatic
    Add

SNMPv3

Проверки через SSH

Обработка SNMPTRAP

# apt install snmptt

# systemctl disable snmptt

# systemctl stop snmptt

# cat /etc/snmp/snmptt.conf
EVENT general .* "General event" Normal
FORMAT ZBXTRAP $aA $ar
# cat /etc/snmp/snmptt.ini
...
date_time_format = %H:%M:%S %Y/%m/%d
...
#log_file = /var/log/snmptt/snmptt.log
log_file = /tmp/my_zabbix_traps.tmp
...
# cat /etc/snmp/snmptrapd.conf
traphandle default snmptt

authCommunity execute writetrap
# cat /lib/systemd/system/snmptrapd.service
...
ExecStart=/usr/sbin/snmptrapd -Lsd -f -On
...
# systemctl daemon-reload
# service snmptrapd restart

Создание своих шаблонов

Configuration
  Templates
    Template App SSH Service/SSH Service
  Full Clone
    Template App SSH Port Service/SSH Port Service
  Add

  Templates
    Template App SSH Port Service/SSH Port Service
      Items
        SSH service is running
          Key: net.tcp.service[ssh,,{$SSH_PORT}]
          Update interval: 30s
      Macros
        {$SSH_PORT}=22

Low-Level Discovery (LLD)

Штатные примеры

server# zabbix_get -s gate -k vfs.fs.discovery
server# zabbix_get -s gate -k vfs.fs.size[/,free]
server# zabbix_get -s gate -k vfs.fs.size[/,total]
server# zabbix_get -s gate -k vfs.fs.size[/,used]
...

server# zabbix_get -s gate -k net.if.discovery | jq
...

server# zabbix_get -s gate -k net.if.in[eth0]
...

Статистика дискового ввода/вывода

agent# zabbix_agentd -p | grep vfs.dev
...

server# zabbix_get -s agent -k "vfs.dev.read[sda,sps]"
...

agent# cat /etc/zabbix/zabbix_agentd.conf.d/my.linux.disk.discovery.conf
UserParameter=my.disks.discovery,/bin/lsblk -dJ | /bin/sed -e 's/blockdevices/data/' -e 's/name/{#NAME}/g' -e 's/type/{#TYPE}/g'
server# zabbix_get -s agent -k my.disks.discovery | jq
...
Configuration->Templates->Create template
  Template name: My Template Linux disks utilization
  Groups: Templates/Server hardware

  Discovery rules->
    Name: my disks discovery
    Key: my.disks.discovery
    Filters->
      {#TYPE} matches disk
    Item prototypes->
      Name: disk {#NAME} read bytes
      Key: vfs.dev.read[{#NAME},sps]
      Type of information: Numeric (float)
      Units: Байт
      Preprocessing->
        Preprocessing steps
        Custom multiplier 512

      Name: disk {#NAME} write bytes
      ...  

Статистика DHCP сервера

gate# cat /etc/zabbix/dhcp-pools-discovery.sh
#!/bin/bash

echo -n '{"data":['

str=`/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
/usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
-o '{"{#POOLNAME}":"' -v location -o '"},'`

echo -n ${str::-1}

echo -n ']}'
gate# /etc/zabbix/dhcp-pools-discovery.sh | jq

gate# cat /etc/zabbix/dhcp-pools-shared-network.sh
#!/bin/sh

res_field=2
test "x$2" = "xused" && res_field=3

/usr/bin/dhcpd-pools -c /etc/dhcp/dhcpd.conf -f x | \
  /usr/bin/xmlstarlet sel -T -t -m '//shared-network' \
  -v location -o ' ' -v defined -o ' ' -v used -n | \
  grep $1 | cut -d ' ' -f $res_field
gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN1 defined

gate# /etc/zabbix/dhcp-pools-shared-network.sh LAN2 used

gate# cat `echo /etc/zabbix/zabbix_agentd.*d/`dhcp_stat.conf
UserParameter=dhcp.pools.discovery,/etc/zabbix/dhcp-pools-discovery.sh

UserParameter=dhcp.pools.shared-network[*],/etc/zabbix/dhcp-pools-shared-network.sh $1 $2
server# zabbix_get -s gate -k dhcp.pools.discovery | jq

server# zabbix_get -s gate -k dhcp.pools.shared-network[LAN1,used]
Configuration->Templates->Create template
  Template name: Template App DHCP Pools
  Groups In groups: Templates/Applications

  Macros: {$DHCP.POOLS.MAX.PERCENT}=90
Add

  Applications: DHCP
  
  Discovery rules
    Name: Search DHCP Pools
    Type: Zabbix Agent
    Key: dhcp.pools.discovery
  Add
    Item prototypes
      Name: DHCP Pool $1 max addr      или DHCP Pool {#POOLNAME} max addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},defined]
      Applications: DHCP
    Add

      Name: DHCP Pool $1 cur addr
      Type: Zabbix Agent
      Key: dhcp.pools.shared-network[{#POOLNAME},used]
      Applications: DHCP
    Add

    Graph prototypes
      Name: DHCP Pool {#POOLNAME} max cur
      Y axis MIN value: Fixed 0
      Items: 
        Template App DHCP Pools: DHCP Pool {#POOLNAME} cur addr
        Template App DHCP Pools: DHCP Pool {#POOLNAME} max addr

    Trigger prototypes

      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME}
или
      Name: On {HOST.NAME} in the DHCP pool {#POOLNAME} used more {$DHCP.POOLS.MAX.PERCENT} percent
      
      Expression: {Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},used].last()}/{Template App DHCP Pools:dhcp.pools.shared-network[{#POOLNAME},defined].last()}*100 > {$DHCP.POOLS.MAX.PERCENT}
                  last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},used])/last(/Template App DHCP Pools/dhcp.pools.shared-network[{#POOLNAME},defined])*100 > {$DHCP.POOLS.MAX.PERCENT}
      Severity: Warning

Экспорт/импорт в XML

Configuration
  Templates
    Export/Import

API

Аутентификация

server:~# apt install curl

server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
   "jsonrpc": "2.0",
   "method": "user.login",
   "params": {
      "user": "Admin",
      "password": "zabbix"
   },
   "id": 1
} ' http://127.0.0.1/zabbix/api_jsonrpc.php
{"jsonrpc":"2.0","result":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","id":1}

Работа с объектами host/template

server:~# apt install jq
server:~# curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {},
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq
...
...
    "method": "template.get",
...
...
    "params": {
        "output": ["hostid", "host"],
        "templateids": ["10NNN"]
    },
...

Доступ к результатам мониторинга

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d '
{
    "jsonrpc": "2.0",
    "method": "history.get",
    "params": {
        "output": "extend",
        "history": 0,
        "itemids": "NNNNN",
        "sortfield": "clock",
        "sortorder": "DESC",
        "limit": 10
    },
    "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "id": 2
} ' http://127.0.0.1/zabbix/api_jsonrpc.php | jq

Оформление запросов в виде BASH скриптов

server:~# export AUTH=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

server:~# cat /root/zab_get_hosts.sh
#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"host.get\",
    \"params\": {},
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
server:~# /root/zab_get_hosts.sh | jq

Список имен узлов
server:~# /root/zab_get_hosts.sh | jq '.result | .[] | .name'

Получение списка карт и их элементов из Zabbix

server.corpX.un:~# cat /root/zab_get_maps.sh
#!/bin/sh

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.get\",
    \"params\": {
        \"selectLinks\": \"extend\",
        \"selectSelements\": \"extend\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
server.corpX.un:~# /root/zab_get_maps.sh | jq -c '.result | .[] | {name: .name, id: .sysmapid}'

Пример изменения конфигурации через Zabbix API

server.corp1.un:~# cat /root/zab_set_map_name.sh
#!/bin/sh

MAPID=$1
MAPNAME=$2

curl -s -k -X POST -H 'Content-Type: application/json-rpc' -d "
{
    \"jsonrpc\": \"2.0\",
    \"method\": \"map.update\",
    \"params\": {
        \"sysmapid\": \"${MAPID}\",
        \"name\": \"${MAPNAME}\"
    },
    \"auth\": \"${AUTH}\",
    \"id\": 2
} " http://127.0.0.1/zabbix/api_jsonrpc.php
server.corp1.un:~# /root/zab_set_map_name.sh 2 "ISP1"