This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
сервис_victoriametrics [2026/04/20 14:40] val [Метрики] |
сервис_victoriametrics [2026/06/10 15:38] (current) val [Подключение Grafana] |
||
|---|---|---|---|
| Line 4: | Line 4: | ||
| * [[https://docs.victoriametrics.com/sd_configs/|vmagent and single-node VictoriaMetrics supports the following Prometheus-compatible service discovery]] | * [[https://docs.victoriametrics.com/sd_configs/|vmagent and single-node VictoriaMetrics supports the following Prometheus-compatible service discovery]] | ||
| - | |||
| * [[https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/|Kubernetes monitoring with VictoriaMetrics Cluster]] | * [[https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster/|Kubernetes monitoring with VictoriaMetrics Cluster]] | ||
| - | |||
| * [[https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/|Kubernetes monitoring via VictoriaMetrics Single]] | * [[https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single/|Kubernetes monitoring via VictoriaMetrics Single]] | ||
| - | |||
| * [[https://docs.victoriametrics.com/scrape_config_examples/]] | * [[https://docs.victoriametrics.com/scrape_config_examples/]] | ||
| - | * [[Система Kubernetes#kube-state-metrics]] | + | * [[https://github.com/VictoriaMetrics/helm-charts/tags]] |
| + | |||
| + | kube-state-metrics | ||
| + | |||
| + | * [[Система Kubernetes#kube-state-metrics]] собирает информацию о состоянии объектов внутри кластера Kubernetes (поды, deployments, namespaces) | ||
| + | |||
| + | node-exporter | ||
| + | |||
| + | * prometheus-node-exporter собирает информацию о состоянии узлов кластера | ||
| <code> | <code> | ||
| (venv1) server# ansible all -f 4 -m apt -a 'pkg=prometheus-node-exporter state=present update_cache=true' -i /root/kubespray/inventory/mycluster/hosts.yaml | (venv1) server# ansible all -f 4 -m apt -a 'pkg=prometheus-node-exporter state=present update_cache=true' -i /root/kubespray/inventory/mycluster/hosts.yaml | ||
| root@9f823e41e1c4:/kubespray# ansible all -f 4 -m apt -a 'pkg=prometheus-node-exporter state=present update_cache=true' -i /inventory/inventory.ini | root@9f823e41e1c4:/kubespray# ansible all -f 4 -m apt -a 'pkg=prometheus-node-exporter state=present update_cache=true' -i /inventory/inventory.ini | ||
| + | </code> | ||
| + | victoriametrics | ||
| + | |||
| + | <code> | ||
| kube1# helm repo add vm https://victoriametrics.github.io/helm-charts/ | kube1# helm repo add vm https://victoriametrics.github.io/helm-charts/ | ||
| kube1# helm repo update | kube1# helm repo update | ||
| + | kube1# helm search repo vm | ||
| - | kube1:~/vm# cat guide-vmsingle-values.yaml | + | kube1:~/vm# helm show values vm/victoria-metrics-single --version 0.35.0 > vmsingle-values.yaml |
| + | |||
| + | kube1:~/vm# cat vmsingle-values.yaml | ||
| </code><code> | </code><code> | ||
| ... | ... | ||
| + | size: 16Gi # replace to 6 | ||
| + | ... | ||
| + | scrape: | ||
| + | enabled: true | ||
| + | ... | ||
| + | # End of COPY | ||
| - job_name: kube-state-metrics | - job_name: kube-state-metrics | ||
| kubernetes_sd_configs: | kubernetes_sd_configs: | ||
| Line 40: | Line 58: | ||
| - kube2.corpX.un:9100 | - kube2.corpX.un:9100 | ||
| - kube3.corpX.un:9100 | - kube3.corpX.un:9100 | ||
| + | ... | ||
| </code><code> | </code><code> | ||
| - | kube1:~/vm# helm upgrade -i vmsingle vm/victoria-metrics-single -f guide-vmsingle-values.yaml -n vm --create-namespace --version 0.35.0 | + | kube1:~/vm# helm upgrade -i vmsingle vm/victoria-metrics-single -f vmsingle-values.yaml -n vm --create-namespace --version 0.35.0 |
| cmder> kubectl port-forward svc/vmsingle-victoria-metrics-single-server 8428 -n vm | cmder> kubectl port-forward svc/vmsingle-victoria-metrics-single-server 8428 -n vm | ||
| </code> | </code> | ||
| + | |||
| + | ===== Визуализация ===== | ||
| * Сервис Grafana в [[Сервис Grafana#Kubernetes]] | * Сервис Grafana в [[Сервис Grafana#Kubernetes]] | ||
| Line 51: | Line 72: | ||
| * [[Сервис Prometheus#prometheus-alertmanager]] (ссылки на правила) | * [[Сервис Prometheus#prometheus-alertmanager]] (ссылки на правила) | ||
| + | * [[Web интерфейс к почте#Roundcube]] | ||
| <code> | <code> | ||
| (venv1) server# ansible-playbook /root/conf/ansible/roles/mail.yml | (venv1) server# ansible-playbook /root/conf/ansible/roles/mail.yml | ||
| - | + | </code> | |
| - | kube1:~/vm# helm show values vm/victoria-metrics-alert > vm-alert-values.yaml | + | ==== Настройка ==== |
| + | <code> | ||
| + | kube1:~/vm# helm show values vm/victoria-metrics-alert --version 0.37.0 > vm-alert-values.yaml | ||
| $ wget -qO - https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/host-and-hardware/node-exporter.yml | sed 's/^/ /' | $ wget -qO - https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/host-and-hardware/node-exporter.yml | sed 's/^/ /' | ||
| Line 81: | Line 104: | ||
| ... | ... | ||
| - name: KubestateExporter | - name: KubestateExporter | ||
| + | ... | ||
| + | # -- Vertical Pod Autoscaler | ||
| ... | ... | ||
| alertmanager: | alertmanager: | ||
| Line 103: | Line 128: | ||
| - to: 'student@corpX.un' | - to: 'student@corpX.un' | ||
| send_resolved: true | send_resolved: true | ||
| + | |||
| + | # -- Alertmanager extra templates | ||
| ... | ... | ||
| </code><code> | </code><code> | ||
| - | kube1:~/vm# helm upgrade -i vma vm/victoria-metrics-alert -f vm-alert-values.yaml -n vm | + | kube1:~/vm# helm template vma vm/victoria-metrics-alert -f vm-alert-values.yaml -n vm --version 0.37.0 |
| + | kube1:~/vm# helm upgrade -i vma vm/victoria-metrics-alert -f vm-alert-values.yaml -n vm --version 0.37.0 | ||
| + | </code> | ||
| + | ==== Отдадка ==== | ||
| + | <code> | ||
| kube1:~/vm# kubectl -n vm exec -ti pods/vma-victoria-metrics-alert-server-<TAB> -- sh | kube1:~/vm# kubectl -n vm exec -ti pods/vma-victoria-metrics-alert-server-<TAB> -- sh | ||
| </code><code> | </code><code> | ||
| Line 113: | Line 144: | ||
| 1 root 0:00 /vmalert-prod --datasource.url=http://vmsingle-victoria-metrics-single-server:8428 --envflag.enable --envflag.prefix=VM_ --httpListenAddr=:8880 --loggerFormat=json --notifier.url=http://vma-victoria-metrics-alert-alertmanager.vm.svc.cluster.local.:9093 --rule=/config/alert-rules.yaml | 1 root 0:00 /vmalert-prod --datasource.url=http://vmsingle-victoria-metrics-single-server:8428 --envflag.enable --envflag.prefix=VM_ --httpListenAddr=:8880 --loggerFormat=json --notifier.url=http://vma-victoria-metrics-alert-alertmanager.vm.svc.cluster.local.:9093 --rule=/config/alert-rules.yaml | ||
| ... | ... | ||
| + | |||
| / # cat /config/alert-rules.yaml | / # cat /config/alert-rules.yaml | ||
| ... | ... | ||
| - | </code><code> | ||
| - | cmder> kubectl -n vm port-forward svc/vma-victoria-metrics-alert-server 8880 | ||
| kube1:~/vm# kubectl -n vm exec -ti pods/vma-victoria-metrics-alert-alertmanager-<TAB> -- sh | kube1:~/vm# kubectl -n vm exec -ti pods/vma-victoria-metrics-alert-alertmanager-<TAB> -- sh | ||
| Line 122: | Line 152: | ||
| /alertmanager $ cat /config/alertmanager.yaml | /alertmanager $ cat /config/alertmanager.yaml | ||
| ... | ... | ||
| - | </code><code> | + | </code> |
| + | ==== Подключение ==== | ||
| + | <code> | ||
| + | cmder> kubectl -n vm port-forward svc/vma-victoria-metrics-alert-server 8880 | ||
| cmder> kubectl -n vm port-forward svc/vma-victoria-metrics-alert-alertmanager 9093 | cmder> kubectl -n vm port-forward svc/vma-victoria-metrics-alert-alertmanager 9093 | ||
| </code> | </code> | ||
| - | * Используем [[Система Kubernetes#Базовые объекты k8s]] для стресс тестирования из [[Анализ производительности системы]] | + | ==== Тестирование ==== |
| - | * [[Команда dd]] для нехватки места на диске | + | |
| + | * [[Управление сервисами в Linux#Управление юнитами Systemd]] systemctl reset-failed | ||
| + | * Используем [[Система Kubernetes#Manifest]] развертывания с последующим стресс тестированием из [[Анализ производительности системы]] | ||
| + | * [[Команда dd]] для воспроизведения нехватки места на диске | ||
| ===== Журналы ===== | ===== Журналы ===== | ||
| Line 135: | Line 172: | ||
| <code> | <code> | ||
| - | kube1:~/vm# helm show values vm/victoria-logs-single > vls-values.yaml | + | kube1:~/vm# helm show values vm/victoria-logs-single --version 0.12.0 > vls-values.yaml |
| kube1:~/vm# cat vls-values.yaml | kube1:~/vm# cat vls-values.yaml | ||
| </code><code> | </code><code> | ||
| + | ... | ||
| + | size: 10Gi # replace to 5 | ||
| ... | ... | ||
| vector: | vector: | ||
| Line 145: | Line 184: | ||
| ... | ... | ||
| </code><code> | </code><code> | ||
| - | kube1:~/vm# helm upgrade -i vls oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-single -f vls-values.yaml -n vm | + | kube1:~/vm# helm upgrade -i vls vm/victoria-logs-single -f vls-values.yaml -n vm --version 0.12.0 |
| cmder$ kubectl port-forward svc/vls-victoria-logs-single-server 9428 -n vm | cmder$ kubectl port-forward svc/vls-victoria-logs-single-server 9428 -n vm | ||
| Line 156: | Line 195: | ||
| <code> | <code> | ||
| Grafana -> Connections -> | Grafana -> Connections -> | ||
| - | Add new connection: VictoriaLogs -> | + | Add new connection: VictoriaLogs |
| - | + | ||
| - | Install -> New Datasources | + | |
| - | http://vls-victoria-logs-single-server:9428 -> | + | |
| - | Explore -> LogQL: _time:5m | + | Install -> New Datasources -> VictoriaLogs |
| + | URL: http://vls-victoria-logs-single-server:9428 | ||
| - | Builder-> Filter: kubernetes.pod_name = my-debian | + | Explore -> victoriametrics-logs-datasources |
| + | Code (LogsQL): kubernetes.pod_labels.app: "my-debian" _time: 5m | ||
| </code> | </code> | ||