• A Guide to Monitor Linux Server using Prometheus and Grafana
• Настройка уведомлений по электронной почте Grafana
• Смена "темы"

• Install on Debian or Ubuntu

• Технология Docker

# id=$(docker create grafana/grafana-oss)

# docker cp $id:/etc/grafana/ - > etc_grafana.tar
# tar -xvf etc_grafana.tar -C /etc
  или
# docker cp $id:/etc/grafana/ /etc/grafana/

# docker rm -v $id

# docker run -d -p 3000:3000 -v /etc/grafana:/etc/grafana --name grafana grafana/grafana-oss:7.5.4-ubuntu
  или
# docker run -d -p 3000:3000 -v /etc/grafana:/etc/grafana --name grafana grafana/grafana-oss
  или
# docker run -d -p 3000:3000 --name grafana grafana/grafana-oss

# docker update --restart=always grafana

• l:admin p:admin
• http://server.corpX.un:3000
• http://192.168.X.10:3000

• Сервис VictoriaMetrics

kube1# helm repo add grafana https://grafana.github.io/helm-charts
kube1# helm repo update

kube1:~/vm# cat my-grafana-values.yaml

  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: victoriametrics
          type: prometheus
          orgId: 1
          url: http://vmsingle-victoria-metrics-single-server:8428
          access: proxy
          isDefault: true
          updateIntervalSeconds: 10
          editable: true

  dashboardProviders:
   dashboardproviders.yaml:
     apiVersion: 1
     providers:
     - name: 'default'
       orgId: 1
       folder: ''
       type: file
       disableDeletion: true
       editable: true
       options:
         path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      victoriametrics:
        gnetId: 10229
        revision: 22
        datasource: victoriametrics
      kubernetes:
        gnetId: 14205
        revision: 1
        datasource: victoriametrics
      node-exporter:
        gnetId: 1860
        revision: 37
        datasource: victoriametrics

kube1:~/vm# helm upgrade -i my-grafana grafana/grafana -f my-grafana-values.yaml -n vm --create-namespace

kube1# kubectl get secret --namespace vm my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

cmder> kubectl -n vm port-forward svc/my-grafana 3000:80

Configuratin/Connections->Data Source->Prometheus 
  URL: http://localhost:9090
       http://192.168.X.10:9090  Если grafana в docker

• Official & community built dashboards
• Node Exporter Full

+ Import->Import via grafana.com
    prometheus-node-exporter: 1860
    prometheus-blackbox-exporter: 7587
  Data Source: Prometheus

• !!! Не забыть сохранить !!!
• Лучше в folder1 (понадобится для alert)

+ New dasboard->Add visualization
    Name: router
      Add New Panel
      
        Panel Title: FastEthernet0/0 connection to WAN
          или
        Panel Title: Port-channel1 connection to LAN
        
        Query: A
          Code        
            Metrics: 8*rate(ifInOctets{ifDescr="FastEthernet0/0",instance="router"}[1m])/1000000
              или
            Metrics: rate(ifOutOctets{ifDescr="Port-channel1",instance="router"}[1m])/125000
          Options->Legend->Custom: in Mbps

        Query: B
          Code        
            Metrics: 8*irate(ifOutOctets{ifDescr="FastEthernet0/0",instance="router"}[1m])/1000000
              или
            Metrics: irate(ifInOctets{ifDescr="Port-channel1",instance="router"}[1m])/125000          
          Options->Legend->Custom: out Mbps

# cat /etc/grafana/grafana.ini

...
[smtp]
enabled = true
host = server.corpX.un:25   #if docker, and may be, better use ip
from_address = grafana@corpX.un
ehlo_identity = server.corpX.un
startTLS_policy = NoStartTLS
...

# service grafana-server restart
  или
# docker restart grafana

Alerting->Notification Channels/Contact points ... 
  userX/Email/userX@isp.un
  student/Email/student@corpX.un

A:
Define query and alert condition -> 
  Options -> Time Range: Last 15 minutes
  
B: Delete 
 
C (можно дать другое имя, оно будет в сообщении): Set as Alert Conditions
  Input: A
  Function: Last

D: Threshold
  Input: C
  IS Above: 1

Dashboard->Panel (name)->Edit->Alert->...->
  Evaluate every: 15s For 1m
  WHEN avg() OF query(B,1m,now) IS ABOVE 8
  Send To: userX/student

Apply->Save Dashboard!!!!