====== Сервисы ELK ======


  * [[https://habr.com/ru/news/t/657169/|Elastic NV — американо-голландская компания, основанная в 2012 году в Амстердаме, Нидерланды, ранее известная как Elasticsearch, прекратила продажи ПО в РФ]] [update 24.03.22]

  * [[https://aws.amazon.com/ru/what-is/elk-stack/|Что такое стек ELK?]]
  * [[https://serveradmin.ru/ustanovka-i-nastroyka-elasticsearch-logstash-kibana-elk-stack/|Как установить и настроить Elasticsearch, Logstash, Kibana (ELK Stack) на Ubuntu, Debian, Centos]]
  * [[https://codedzen.ru/category/uroki/elasticsearch/|Уроки по Elasticsearch]]
  * [[https://habr.com/ru/sandbox/115014/|Глоссарий ElasticSearch]]

  * [[https://vk.com/wall-95686747_13620|Для установки Elasticsearch и остальных продуктов Elastic, можно использовать репозиторий https://mirror.yandex.ru/mirrors.]]

  * !!! От 8-x GB (8192 MB) и 2-х CPU на VM !!!

===== Elasticsearch ======

==== Установка пакета из репозитория ====

  * [[Переменные окружения]] (для установки через proxy)

  * [[https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html|Install Elasticsearch with Debian Packageedit]] any Debian-based system such as Debian and Ubuntu.

  * [[Управление ПО в Linux#Подключение сторонних репозиториев]]

<code>
# apt install elasticsearch
</code>
==== Запуск и проверка работоспособности ====

  * [[https://sleeplessbeastie.eu/2020/02/29/how-to-prevent-systemd-service-start-operation-from-timing-out/|How to prevent systemd service start operation from timing out]]

<code>
# systemctl edit elasticsearch
</code>

  * [[Управление сервисами в Linux#Настройка параметров запуска в Systemd]]

<code>
# systemctl enable elasticsearch
</code>

  * [[https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-password.html|elasticsearch-reset-password]]
<code>
# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
</code>
  или
<code>
# cat /etc/elasticsearch/elasticsearch.yml
</code><code>
...
xpack.security.enabled: false
...
http.host: [_local_]
...
</code><code>
# systemctl start elasticsearch

# systemctl status elasticsearch


# curl --noproxy localhost -X GET 'http://localhost:9200'
  или
# curl -X GET 'http://localhost:9200'
</code>

==== Примеры запросов ====

=== Список индексов ===
<code>
# curl --noproxy localhost -X GET 'http://localhost:9200/_cat/indices?pretty'
</code>

=== Добавление/замена документа ===
<code>
curl --noproxy localhost -H "Content-Type: application/json" -X PUT 'http://localhost:9200/myindex1/mytype1/myid1?pretty' -d '
curl --noproxy localhost -H "Content-Type: application/json" -X PUT 'http://localhost:9200/myindex1/_doc/myid1?pretty' -d '
{
"field1": 111,
"field2": "abcd",
"filed3": {
    "field4": 5.8,
	"field5": "dcba"
  },
"field6": [1,2,3]
}
'
</code>

=== Добавление документа с другим составом полей и автоматическим созданием идентификатора ===

<code>
curl --noproxy localhost -H "Content-Type: application/json" -X POST 'http://localhost:9200/myindex1/mytype1/?pretty' -d '
curl --noproxy localhost -H "Content-Type: application/json" -X POST 'http://localhost:9200/myindex1/_doc/?pretty' -d '
{
"field1": 222,
"field2": "def",
"filed3": {
	"field5": "fde",
	"field7": 222
  },
"field6": [4,5]
}
'
</code>

=== Выборка данных ===
<code>
# curl --noproxy localhost -X GET 'http://localhost:9200/myindex1/mytype1/myid1?pretty'
# curl --noproxy localhost -X GET 'http://localhost:9200/myindex1/_doc/myid1?pretty'

# curl --noproxy localhost -X GET 'http://localhost:9200/myindex1/_search?pretty'

# curl --noproxy localhost -X GET 'http://localhost:9200/_search?pretty'
</code>

=== Удаление ВСЕХ данных ===
<code>
# curl --noproxy localhost -X DELETE 'http://localhost:9200/_all'
</code>

===== Kibana ======

==== Установка пакета из репозитория ====

  * [[https://www.elastic.co/guide/en/kibana/current/deb.html|Install Kibana with Debian package]]

  * [[Управление ПО в Linux#Подключение сторонних репозиториев]]

<code>
# apt install kibana
</code>

==== Настройка и запуск ====
<code>
# cat /etc/kibana/kibana.yml
</code><code>
...
#server.port: 5601
...
server.host: "192.168.X.10"
...
#elasticsearch.hosts: ["http://localhost:9200"]
...
</code><code>
systemctl start kibana
systemctl enable kibana
systemctl status kibana
</code>

==== Подключение ====

Задержка 2-3 минуты

  * http://192.168.X.10:5601/status
  * http://192.168.X.10:5601/

==== Примеры запросов ====

  * Management - Dev Tools

<code>
GET _search
{
  "size": 10000,
  "query": {
    "match_all": {}
  }
}

GET /_cat/indices

GET /myindex1/mytype1/myid1
GET /myindex1/_doc/myid1

PUT /myindex1/mytype1/myid2
PUT /myindex1/_doc/myid2
{
"field1": 222,
"field2": "fghj",
"filed3": {
    "field4": 8.5,
	"field5": "hgfd"
  },
"field6": [4,5,6]
}

Обновление отдельных полей документа

POST /myindex1/mytype1/myid1/_update
POST /myindex1/_update/myid1
{
  "doc": {
    "filed3": {
      "field4": 6.8 
    }
  }
}

Типы данных полей индекса

GET /myindex1/

Все записи индекса

GET /myindex1/_search

DELETE /myindex1/_doc/myid2
</code>

==== Kibana Dashboard ====

=== Filebeat Netflow Top-N ===

Network Direction -> Inbound

или

Add filter -> Edit as Query DSL -> Elasticsearch Query DSL
<code>
{
  "match": {
    "netflow.destination_ipv4_address": "192.168.X.0/24"
  }
}
</code>

===== Logstash =====

  * [[https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories|Installing from Package Repositories]]
  * [[https://gist.github.com/justinjahn/85305bc7b7df9a6412baedce5f1a0ece|justinjahn/10-cisco-elasticsearch.conf]]
  * [[https://www.elastic.co/guide/en/logstash/current/plugins-outputs-exec.html|Exec output plugin]]
  * [[https://www.elastic.co/guide/en/logstash/current/config-examples.html|Logstash Configuration Examples]]

<code>
# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout {} elasticsearch {} }'
...
The stdin plugin is now waiting for input:
...
qwerty
{
      "@version" => "1",
    "@timestamp" => 2021-06-24T06:39:22.147Z,
          "host" => "server.corpX.un",
       "message" => "qwerty"
}
...
GET /logstash-*/_search
  или
GET /.ds-logs-generic-default-*/_search

GET /logstash-*/_search
GET /.ds-logs-generic-default-*/_search
{
  "query": {
    "term": {
      "message": {
        "value": "qwerty"
      }
    }
  }
}
</code>

Длится 3-4 минуты

<code>
# /usr/share/logstash/bin/logstash-plugin install logstash-output-exec
</code>

  * [[Пакет sudo]]
  * Для приведенных выражений grok требуется [[Общие настройки сетевого оборудования Cisco#Настройка времени]] с использованием NTP для сетевого оборудования Cisco

<code>
# cat /etc/logstash/conf.d/cisco-backup-config.conf
</code><code>
input {
  udp {
    port => "8514"
  }
}

filter {
  grok {
    match => [
        "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",
        "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}"
    ]
    overwrite => [ "message" ]
    remove_field => [ "syslog5424_pri", "@version" ]
  }
}

output {
  if [message] =~ /Configured/ {
    exec {
      command => "sudo /root/cisco-backup-config-logstash.sh %{host}"
    }
  }
#  stdout { codec => rubydebug }
  elasticsearch {
    index => "network-%{+YYYY.MM.dd}"
  }
}
</code><code>
# cat /root/cisco-backup-config-logstash.sh
</code><code>
#!/bin/sh

#ip=$1
ip=`echo $1 | cut -d: -f2 | cut -d'}' -f1`

#cisco_name=`host $ip | awk '{ print $NF }' | cut -d. -f1`
cisco_name=`getent hosts $ip | awk '{ print $NF }'`

#echo $1 $ip $cisco_name >> /tmp/cisco-backup-config-logstash.log

/usr/bin/sshpass -p cisco /usr/bin/scp ${cisco_name}:running-config /srv/tftp/${cisco_name}-running-config
cd /srv/tftp/
/usr/bin/git add *
/usr/bin/git --no-optional-locks status | grep 'modified\|deleted\|new file' | /usr/bin/git commit -a -F -
</code><code>
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/cisco-backup-config.conf

systemctl start logstash
systemctl enable logstash
systemctl status logstash

# tail -f /var/log/logstash/logstash-plain.log
</code><code>
GET /network-*/_search
</code>
===== Filebeat =====

  * [[https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html|Filebeat command reference]]

<code>
# apt install filebeat

# cat /etc/filebeat/filebeat.yml
</code><code>
...
- type: log
  или, начиная с версии 8.0
- type: filestream
...
  enabled: true
...
  exclude_lines: ['filebeat']
...
setup.kibana:
...
  host: "192.168.X.10:5601"
...
output.elasticsearch:
...
  hosts: ["localhost:9200"]
...
</code><code>
# filebeat test config -e -c /etc/filebeat/filebeat.yml
...
Config OK

systemctl enable filebeat
systemctl start filebeat
systemctl status filebeat
</code><code>
GET /.ds-filebeat-*/_count
 но, работает и так:
GET /filebeat-*/_count

GET /filebeat-*/_search
{
  "sort": [{"@timestamp": {"order": "desc"}}],
  "query": {
    "match": {
      "log.file.path": "/var/log/auth.log"
    }
  }
}
</code>

==== filebeat netflow module ====

  * [[https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-netflow.html|NetFlow module]]

<code>
# filebeat modules enable netflow

# filebeat setup -e
  2 минуты

# cat /etc/filebeat/modules.d/netflow.yml
</code><code>
- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 9555
</code><code>
# systemctl stop filebeat

# filebeat run --modules netflow -d "*" -e
...
Exiting: module netflow is configured but has no enabled filesets

# systemctl start filebeat
</code>

==== Приметры запросов ====

=== Все NetFlow записи для IP 192.168.X.10 ===
<code>
GET /filebeat-*/_search?size=10000
{
  "query": {
    "match": {
      "netflow.destination_ipv4_address": "192.168.X.10"
    }
  }
}
</code>

=== Все NetFlow записи для сети 192.168.X.0/24 за последние сутки ===
<code>
GET /filebeat-*/_search?size=10000
{
  "sort": [{"netflow.exporter.timestamp": {"order": "desc"}}],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "netflow.destination_ipv4_address": "192.168.X.0/24"
          }
        },
        {
          "range": {
            "netflow.exporter.timestamp": {
               "gte": "now-1d",
               "lte": "now"
            }   
          }
        }
      ]
    }
  }
}
</code>

=== Список уникальных IP сети предприятия за весь период наблюдения ===

  * "size": 0 - вывести только результат агрегации
  * "track_total_hits": true - обработать все подходящие записи

<code>
GET /filebeat-*/_search
{
  "size": 0,
  "aggs": {
    "unique_dst_ip": {
      "terms": {
        "field": "netflow.destination_ipv4_address"
      }
    }
  },
  "query": {
    "match": {
      "netflow.destination_ipv4_address": "192.168.X.0/24"
    }
  },
  "track_total_hits": true
}
</code>

=== Суммарный объем входящего трафика для данного IP адреса с некоторой даты до текущего момента ===
<code>
GET /filebeat-*/_search
{
  "size": 0,
  "aggregations": {
    "download_sum": {
      "sum": {
        "field": "network.bytes"
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {    
          "match": {
            "netflow.destination_ipv4_address": "192.168.X.128"
          }
        },
        {
          "range": {
            "netflow.exporter.timestamp": {
               "gte": "2021-07-07T00:00:00",
               "lt": "now"
            }   
          }
        }
      ]  
    }
  },
  "track_total_hits": true
}
</code>