• A Guide to Monitor Linux Server using Prometheus and Grafana
• Настройка уведомлений по электронной почте Grafana
• Смена "темы"

• Install on Debian or Ubuntu
• https://github.com/grafana/grafana/releases

• Технология Docker

# docker run -d -p 3000:3000 --name grafana grafana/grafana-oss:13.0.1

  или

# id=$(docker create grafana/grafana-oss:13.0.1)

# docker cp $id:/etc/grafana/ /etc/grafana/

# docker rm -v $id

# docker run -d -p 3000:3000 -v /etc/grafana:/etc/grafana --name grafana grafana/grafana-oss:13.0.1

# docker update --restart=always grafana

• l:admin p:admin
• http://server.corpX.un:3000
• http://192.168.X.10:3000

• Сервис VictoriaMetrics

kube1# helm repo add grafana-community https://grafana-community.github.io/helm-charts

kube1# helm repo update

kube1:~/vm# cat grafana-values.yaml

#env:
#  HTTP_PROXY: "http://gate.isp.un:3128"
#  HTTPS_PROXY: "http://gate.isp.un:3128"
#  NO_PROXY: "localhost,127.0.0.1,.svc,.cluster.local,vmsingle-victoria-metrics-single-server,vls-victoria-logs-single-server"

#persistence:
#  enabled: true
#  storageClassName: "longhorn"
#  accessModes:
#    - ReadWriteOnce
#  size: 2Gi

#downloadDashboards:
#  env:
#    HTTP_PROXY: "http://gate.isp.un:3128"
#    HTTPS_PROXY: "http://gate.isp.un:3128"
#    NO_PROXY: "localhost,127.0.0.1,.svc,.cluster.local,vmsingle-victoria-metrics-single-server,vls-victoria-logs-single-server"

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: victoriametrics
        type: prometheus
        orgId: 1
        url: http://vmsingle-victoria-metrics-single-server:8428
        access: proxy
        isDefault: true
        updateIntervalSeconds: 10
        editable: true

dashboardProviders:
 dashboardproviders.yaml:
   apiVersion: 1
   providers:
   - name: 'default'
     orgId: 1
     folder: ''
     type: file
     disableDeletion: true
     editable: true
     options:
       path: /var/lib/grafana/dashboards

dashboards:
  default:
    victoriametrics:
      gnetId: 10229
      revision: 22
      datasource: victoriametrics
    kubernetes:
      gnetId: 14205
      revision: 1
      datasource: victoriametrics
    node-exporter:
      gnetId: 1860
      revision: 37
      datasource: victoriametrics

kube1:~/vm# helm upgrade -i grafana grafana-community/grafana -f grafana-values.yaml -n vm --create-namespace --version 12.1.1

kube1# kubectl get secret -n vm grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

cmder> kubectl -n vm port-forward svc/grafana 3000:80

• Может понадобится удалить метки: systemd_service_name!=“”, name=~“^k8s_.*”,

Configuratin/Connections->Data Source->Prometheus 
  URL: http://localhost:9090
       http://192.168.X.10:9090  Если grafana в docker

• Official & community built dashboards
• Node Exporter Full
• CloudNativePG

+ Import->Import via grafana.com
    prometheus-node-exporter: 1860
    prometheus-blackbox-exporter: 7587
    prometheus-apache-exporter: 17898
  Data Source: Prometheus

• !!! Не забыть сохранить !!!
• Лучше в folder1 (понадобится для alert)

+ New dasboard->Add visualization
    Name: router
      Add New Panel
      
        Panel Title: FastEthernet0/0 connection to WAN
          или
        Panel Title: Port-channel1 connection to LAN
        
        Query: A
          Code        
            Metrics: 8*rate(ifInOctets{ifDescr="FastEthernet0/0",instance="router"}[1m])/1000000
              или
            Metrics: rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m])/125000
          Options->Legend->Custom: in Mbps

        Query: B
          Code        
            Metrics: 8*irate(ifOutOctets{ifDescr="FastEthernet0/0",instance="router"}[1m])/1000000
              или
            Metrics: irate(ifInOctets{ifDescr="Port-channel1",instance="router"}[1m])/125000          
          Options->Legend->Custom: out Mbps

# cat /etc/grafana/grafana.ini

...
[smtp]
enabled = true
host = server.corpX.un:25   #if docker, and may be, better use ip
from_address = grafana@corpX.un
ehlo_identity = server.corpX.un
startTLS_policy = NoStartTLS
...

# service grafana-server restart
  или
# docker restart grafana

Alerting->Notification configuration/Contact points ... 
  student/Email/student@corpX.un

A:
Define query and alert condition -> 
  Options -> Time Range: Last 15 minutes
  
B: Delete 
 
C (можно дать другое имя, оно будет в сообщении): Set as Alert Conditions
  Input: A
  Function: Last

D: Threshold
  Input: C
  IS Above: 1

  * A (Query): Сам запрос к базе данных (Prometheus/Loki).
  * B (Reduce): Превращение временного ряда в одно число (например, среднее за 5 минут).
  * C (Threshold): Сравнение этого числа с порогом.