Стратегии деплоя в Kubernetes

Всем привет! Очередной вебинар в стиле “лучше один раз увидеть, чем сто раз услышать”) На этот раз, речь пойдет о стратегиях развертывания приложений. Добавим в копилку наших знаний понятие Service Mesh и поработаем с Istio

Техническое задание

Столкнуться с необходимостью использования стратегии развертывания приложений
Реализовать canary развертывание с использованием “чистого” kubernetes, ingress и service mesh
Использовать полученные знания для реализации rolling, recreate, blue/green, dark (A/B-тестирование) стратегий

Запись вебинара

Тэги: deployment rollout, canary deployment, ingress annotations, Kubernetes Gateway API, Istio Service Mesh, kiali, nginx proxy header
https://rutube.ru/video/private/45160ea603c4f541cd11ebcc3fdf5109/
https://youtu.be/uEpEDk8EWSg

Методическая подготовка

Шаг 1. Что у нас есть для начала

Место вебинара между Доступ к приложениям в Bare-Metal Kubernetes и Работа с хранилищами в Kubernetes

Шаг 4 (/root/my-webd-deployment.yaml и /root/my-webd-service.yaml красивыми именами объектов ,MetalLB и файл с сетью будут удалены, )
Шаг 5 (NGINX https, сервис будет отключен, DNS gowebd → wan gate)
Шаг 7 (ingress-nginx без allow-snippet-annotations)
Шаг 12 (Приложение развернуто через ArgoCD)

Сделать перед вебинаром

4GB RAM на узел

Сделать на вебинаре

Используем Управление приложениями через kubectl для удаления его из ArgoCD

Шаг 2. canary deployment with pausing-and-resuming-a-deployment

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment

Service ClusterIP
Deployment ver1.1 на 10 реплик
Версии deployment rollout pause/resume

kube1# IP=$(kubectl -n my-ns get svc my-webd-srv -ojsonpath='{.spec.clusterIP}')

kube1# while true; do curl $IP; sleep 0.1; done

Шаг 3. canary deployment with service and labels

kube1# kubectl delete -f my-webd-deployment.yaml -n my-ns; kubectl apply -f my-webd-deployment.yaml -n my-ns

kube1# cat my-webd-canary-replicaset.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-webd-canary-rs
spec:
  selector:
    matchLabels:
      track: my-webd-canary-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        track: my-webd-canary-lab
    spec:
      containers:
      - name: my-webd-canary-con
        image: server.corpX.un:5000/student/gowebd:ver1.2

kube1# kubectl -n my-ns apply -f my-webd-canary-replicaset.yaml

kube1# while true; do curl $IP; sleep 0.1; done
       
kube1# kubectl -n my-ns scale replicaset my-webd-canary-rs --replicas 10
        
kube1# kubectl -n my-ns scale deployment my-webd-dep --replicas 0

Шаг 4. canary deployment with ingress and annotations

Проверить наличие ingress-nginx контроллера

kube1# kubectl -n my-ns delete -f my-webd-canary-replicaset.yaml

kube1# cat my-webd-deployment.yaml

...
  replicas: 1
...

kube1# kubectl apply -f my-webd-deployment.yaml -n my-ns

kube1# cat my-webd-canary-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-webd-canary-srv
spec:
  selector:
    app: my-webd-canary-lab
  ports:
  - protocol: TCP
    port: 80

kube1# cat my-webd-canary-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-canary-dep
spec:
  selector:
    matchLabels:
      app: my-webd-canary-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-canary-lab
    spec:
      containers:
      - name: my-webd-canary-con
        image: server.corpX.un:5000/student/gowebd:ver1.2

kube1# kubectl apply -f my-webd-canary-service.yaml,my-webd-canary-deployment.yaml -n my-ns

kube1# cat my-webd-ingress-and-canary.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-webd-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: gowebd.corpX.un
    http:
      paths:
      - backend:
          service:
            name: my-webd-srv
            port:
              number: 80
        path: /
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-webd-canary-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
    nginx.ingress.kubernetes.io/canary-by-header: x-my-version
    nginx.ingress.kubernetes.io/canary-by-header-value: canary
spec:
  ingressClassName: nginx
  rules:
  - host: gowebd.corpX.un
    http:
      paths:
      - backend:
          service:
            name: my-webd-canary-srv
            port:
              number: 80
        path: /
        pathType: Prefix

kube1# kubectl apply -f my-webd-ingress-and-canary.yaml -n my-ns

kube1# curl kube1 -H "Host: gowebd.corpX.un" -H "x-my-version: canary"

kube1# while true; do curl kube1 -H "Host: gowebd.corpX.un"; sleep 0.1; done

Шаг 5. canary deployment with Gateway API

Простое объяснение CRD в Kubernetes и как его использовать

Istio Getting Started

kube1# kubectl delete -f my-webd-ingress-and-canary.yaml -n my-ns

kube1# kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.2.0" | kubectl apply -f -

MetalLB с autoAssign: true

kube1# curl -L https://istio.io/downloadIstio | sh -

kube1# cp -v /root/istio*/bin/istioctl /usr/local/bin/

kube1# istioctl install

kube1# cat my-webd-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-webd-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same

kube1# kubectl -n my-ns apply -f my-webd-gateway.yaml

kube1# kubectl -n my-ns get gtw my-webd-gateway
my-webd-gateway   istio   192.168.13.66   True         113m

kube1# cat my-webd-route.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-webd-route
spec:
  parentRefs:
  - name: my-webd-gateway
  rules:
  - matches:
    - path:
        type: Exact
        value: /
    backendRefs:
    - name: my-webd-srv
      port: 80
      weight: 90
    - name: my-webd-canary-srv
      port: 80
      weight: 10

kube1# kubectl -n my-ns apply -f my-webd-route.yaml

kube1# kubectl -n my-ns describe httproutes.gateway.networking.k8s.io my-webd-route

kube1# IP=$(kubectl -n my-ns get gtw my-webd-gateway -ojsonpath='{.status.addresses[0].value}')

kube1# while true; do curl $IP; sleep 0.1; done

Шаг 6. canary deployment with Istio VirtualService

kube1:~/istio-1.24.3# kubectl apply -f samples/addons

kube1# kubectl get pods -n istio-system

/home/mobaxterm> ssh root@192.168.13.221 -X
kube1# istioctl dashboard kiali &
kube1# firefox &

или

cmder> ssh -L20001:localhost:20001 root@192.168.13.221
kube1# kubectl port-forward svc/kiali 20001:20001 -n istio-system

или

cmder> kubectl port-forward svc/kiali 20001:20001 -n istio-system

http://localhost:20001/

kube1# kubectl delete ns my-ns; kubectl create ns my-ns
kube1# kubectl label namespace my-ns istio-injection=enabled
kube1# kubectl get ns --show-labels

kube1# cat my-webd-deployment-v1-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-v1-dep
spec:
  selector:
    matchLabels:
      app: my-webd-lab
      version: v1-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        version: v1-lab
    spec:
      containers:
      - name: my-webd-con
        image: server.corp13.un:5000/student/gowebd:ver1.1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-v2-dep
spec:
  selector:
    matchLabels:
      app: my-webd-lab
      version: v2-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        version: v2-lab
    spec:
      containers:
      - name: my-webd-con
        image: server.corp13.un:5000/student/gowebd:ver1.2

kube1# kubectl apply -f my-webd-deployment-v1-v2.yaml -n my-ns

kube1# kubectl -n my-ns describe pod my-webd-v1-<TAB> | grep istio

kube1# cat my-webd-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-webd-srv
spec:
  selector:
    app: my-webd-lab
  ports:
  - protocol: TCP
    port: 80
    name: http  #need for istio

kube1# kubectl apply -f my-webd-service.yaml -n my-ns
    
kube1# kubectl get svc -n istio-system | grep ingr
istio-ingressgateway   LoadBalancer   10.233.37.214   192.168.13.65   15021:31547/TCP,80:32173/TCP,443:31308/TCP       19h

kube1# cat my-webd-istio-gateway.yaml

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: my-webd-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - gowebd.corp13.un
    - webd.corp13.un
#    - "*"

kube1# kubectl apply -f my-webd-istio-gateway.yaml -n my-ns

kube1# cat my-webd-destrul.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-webd-destrul
spec:
  host: my-webd-srv
  subsets:
  - name: v1-subset
    labels:
      version: v1-lab
  - name: v2-subset
    labels:
      version: v2-lab

kube1# kubectl apply -f my-webd-destrul.yaml -n my-ns

kube1# cat virtserv-after-gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtserv-after-gateway
spec:
  hosts:
  - "gowebd.corp13.un"
  gateways:
  - my-webd-gateway
  http:
#  - name: "canary"
#    match:
#    - headers:
#        x-forwarded-for:
#          regex: "192.168.13.*"
##        x-my-version:
##          exact: canary
#    route:
#    - destination:
#        host: frontend
#        subset: v2
  - match:
    - {}
    route:
    - destination:
        host: my-webd-srv
        subset: v1-subset
#        host: frontend
#        subset: v1
      weight: 90
    - destination:
        host: my-webd-srv
        subset: v2-subset
#        host: frontend
#        subset: v2
      weight: 10

kube1# kubectl apply -f virtserv-after-gateway.yaml -n my-ns
      
kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done

kube1# cat frontend.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        version: v1
    spec:
      containers:
      - image: nginx:1.16.1-alpine
        name: nginx
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-v2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        version: v2
    spec:
      containers:
      - image: nginx:1.17.8-alpine
        name: nginx
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen 80 default_server;
        location / {
           proxy_pass http://my-webd-srv;
           proxy_http_version 1.1;
        }
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: frontend
  name: frontend
spec:
  ports:
  - port: 80
    name: http
    protocol: TCP
  selector:
    app: frontend
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: frontend
spec:
  host: frontend
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

kube1# kubectl apply -f frontend.yaml -n my-ns

kube1# cat virtserv-after-gateway.yaml

...
#        host: my-webd-srv
#        subset: v1-subset
        host: frontend
        subset: v1
...
#        host: my-webd-srv
#        subset: v2-subset
        host: frontend
        subset: v2
...

kube1# kubectl apply -f virtserv-after-gateway.yaml -n my-ns

kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done

kube1# cat my-webd-virtserv-src-lab.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-webd
spec:
  hosts:
  - my-webd-srv
  gateways: []
  http:
  - match:
    - sourceLabels:
        app: frontend
        version: v1
    route:
    - destination:
        host: my-webd-srv
        subset: v1-subset
        port:
          number: 80
      weight: 100
  - match:
    - sourceLabels:
        app: frontend
        version: v2
    route:
    - destination:
        host: my-webd-srv
        subset: v2-subset
        port:
          number: 80
      weight: 100

kube1# kubectl apply -f my-webd-virtserv-src-lab.yaml -n my-ns

kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done

Добавление журналов

Envoy Access Logs ... Default access log format

kube1# vim virtserv-after-gateway.yaml

...
        x-forwarded-for:
          regex: "192.168.13.*"
...
    - {}
    route:
    - destination:
        host: frontend
        subset: v1

kube1# cat telemetry.yaml

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: Envoy

kube1# kubectl apply -f telemetry.yaml -n istio-system

kube1# curl 192.168.13.65 -H "Host: gowebd.corp13.un" -H "x-forwarded-for: 192.168.13.10"

kube1# kubectl -n my-ns logs -l app=my-webd-lab -c istio-proxy -f
kube1# kubectl -n my-ns logs pods/my-webd-v2-<TAB> -c istio-proxy -f
kube1# kubectl -n my-ns logs pods/frontend-v2-<TAB> -c istio-proxy -f

kube1# vim virtserv-after-gateway.yaml

...
#        x-my-version:
#          exact: canary
...

kube1# curl 192.168.13.65 -H "Host: gowebd.corp13.un" -H "x-my-version: canary"

gate# systemctl disable haproxy --now

gate# cat /etc/nginx/sites-available/gowebd

    server {
        listen 80;
        server_name gowebd.corp13.un;
        return 301 https://gowebd.corp13.un:443$request_uri;
    }

    server {
        listen 443 ssl;
        server_name gowebd.corp13.un;
        ssl_certificate /root/gowebd.crt;
        ssl_certificate_key /root/gowebd.key;

        location / {
            proxy_pass http://192.168.13.65;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
        }
    }

    server {
        listen 8080;
        server_name gowebd.corp13.un;
        return 301 https://gowebd.corp13.un:8443$request_uri;
    }

    server {
        listen 8443 ssl;
        server_name gowebd.corp13.un;
        ssl_certificate /root/gowebd.crt;
        ssl_certificate_key /root/gowebd.key;

        location / {
            proxy_pass http://192.168.13.65;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-My-Version "canary";
        }
    }

gate# systemctl enable nginx --now

gate.corp13.un:~# cat /etc/iptables/rules.v4
...
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d 172.16.1.13/32 -i eth1 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.13.10:53
-A PREROUTING -s 172.16.1.113/32 -d 172.16.1.13/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.1.13:8080
-A PREROUTING -s 172.16.1.113/32 -d 172.16.1.13/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.1.13:8443
-A POSTROUTING -s 192.168.13.0/24 -o eth1 -j MASQUERADE
COMMIT
...
gate.corp13.un:~# iptables-restore /etc/iptables/rules.v4

server# curl https://gowebd.corp13.un

win client> https://gowebd.corp13.un

Вопросы?

Черновик

istioctl install #--set values.global.proxy.privileged=true

~/gowebd-k8s# cat your-istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout  # Log to stdout (or specify a file path)
    accessLogEncoding: JSON     # Or TEXT, depending on your preference
    accessLogFormat: |
      {
        "start_time": "%START_TIME%",
        "method": "%REQ(:METHOD)%",
        "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        "protocol": "%PROTOCOL%",
        "response_code": "%RESPONSE_CODE%",
        "response_flags": "%RESPONSE_FLAGS%",
        "bytes_received": "%BYTES_RECEIVED%",
        "bytes_sent": "%BYTES_SENT%",
        "duration": "%DURATION%",
        "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
        "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
        "user_agent": "%REQ(USER-AGENT)%",
        "custom_header": "%REQ(X-CUSTOM-HEADER)%"
      }


istioctl install -f your-istio-operator.yaml


kube1:~/gowebd-k8s# cat ingress-preserve-xff.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingress-preserve-xff
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          use_remote_address: false # Отключает перезапись X-Forwarded-For

kube1:~/gowebd-k8s# curl -H "x-forwarded-for: 123.123.123" http://webd.corp13.un

---
Не заработало:
kube1:~/gowebd-k8s# cat enable-header-logging.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: enable-header-logging
spec:
  workloadSelector:
    labels:
      app: my-webd
  configPatches:
#    - applyTo: NETWORK_FILTER
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND  # Или SIDECAR_OUTBOUND для исходящего трафика
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inline_code: |
              function envoy_on_request(request_handle)
                local headers = request_handle:headers()
                for key, value in pairs(headers) do
                  request_handle:logInfo("Request Header: " .. key .. " = " .. value)
                end
              end

              function envoy_on_response(response_handle)
                local headers = response_handle:headers()
                for key, value in pairs(headers) do
                  response_handle:logInfo("Response Header: " .. key .. " = " .. value)
                end
              end




root@node1:~/gowebd-k8s# istioctl proxy-config listeners -n my-ns my-webd-v1-5dfd6f4ff4-trjrh -o json| less

~/gowebd-k8s# kubectl -n my-ns logs -l app=my-webd --all-containers
~/gowebd-k8s# kubectl -n my-ns logs -l app=my-webd -c istio-proxy -f

Методические материалы Лохтурова Вячеслава

Table of Contents

Стратегии деплоя в Kubernetes

Реклама

Техническое задание

Запись вебинара

Методическая подготовка

Шаг 1. Что у нас есть для начала

Шаг 2. canary deployment with pausing-and-resuming-a-deployment

Шаг 3. canary deployment with service and labels

Шаг 4. canary deployment with ingress and annotations

Шаг 5. canary deployment with Gateway API

Шаг 6. canary deployment with Istio VirtualService

Добавление журналов

Вопросы?

Черновик

Методические материалы Лохтурова Вячеслава

User Tools

Site Tools

Table of Contents

Стратегии деплоя в Kubernetes

Реклама

Техническое задание

Запись вебинара

Методическая подготовка

Шаг 1. Что у нас есть для начала

Шаг 2. canary deployment with pausing-and-resuming-a-deployment

Шаг 3. canary deployment with service and labels

Шаг 4. canary deployment with ingress and annotations

Шаг 5. canary deployment with Gateway API

Шаг 6. canary deployment with Istio VirtualService

Добавление журналов

Вопросы?

Черновик

Page Tools