====== Стратегии деплоя в Kubernetes ======

  * [[https://habr.com/ru/companies/flant/articles/471620/|Стратегии деплоя в Kubernetes: rolling, recreate, blue/green, canary, dark (A/B-тестирование)]]
  * [[https://habr.com/ru/companies/flant/articles/440378/|Назад к микросервисам вместе с Istio. Часть 2]]
  

===== Реклама =====

  * Всем привет! Очередной вебинар в стиле "лучше один раз увидеть, чем сто раз услышать") На этот раз, речь пойдет о стратегиях развертывания приложений. Добавим в копилку наших знаний понятие Service Mesh и поработаем с Istio

===== Техническое задание =====

  * Столкнуться с необходимостью использования стратегии развертывания приложений
  * Реализовать canary развертывание с использованием "чистого" kubernetes, ingress и service mesh
  * Использовать полученные знания для реализации rolling, recreate, blue/green, dark (A/B-тестирование) стратегий 

===== Запись вебинара =====

  * Тэги: deployment rollout, canary deployment, ingress annotations, Kubernetes Gateway API, Istio Service Mesh, kiali, nginx proxy header
  * https://rutube.ru/video/private/45160ea603c4f541cd11ebcc3fdf5109/
  * https://youtu.be/uEpEDk8EWSg

===== Методическая подготовка =====

===== Шаг 1. Что у нас есть для начала =====

  * Место вебинара между [[Доступ к приложениям в Bare-Metal Kubernetes]] и [[Работа с хранилищами в Kubernetes]]

  * Шаг 4 (/root/my-webd-deployment.yaml и /root/my-webd-service.yaml красивыми именами объектов ,MetalLB и файл с сетью будут удалены, )
  * Шаг 5 (NGINX https, сервис будет отключен, DNS gowebd -> wan gate)
  * Шаг 7 (ingress-nginx без allow-snippet-annotations)
  * Шаг 12 (Приложение развернуто через ArgoCD)

Сделать перед вебинаром

  * 4GB RAM на узел

Сделать на вебинаре

  * Используем [[Контроллер ArgoCD#Управление приложениями через kubectl]] для удаления его из ArgoCD


===== Шаг 2. canary deployment with pausing-and-resuming-a-deployment =====

  * [[https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment]]

  * [[Система Kubernetes#Service]] ClusterIP
  * [[Система Kubernetes#Deployment]] ver1.1 на 10 реплик
  * [[Система Kubernetes#Версии deployment]] rollout pause/resume

<code>
kube1# IP=$(kubectl -n my-ns get svc my-webd-srv -ojsonpath='{.spec.clusterIP}')

kube1# while true; do curl $IP; sleep 0.1; done
</code>

===== Шаг 3. canary deployment with service and labels =====

  * [[https://kubernetes.io/docs/concepts/workloads/management/#canary-deployments]]
  * [[https://habr.com/ru/companies/nixys/articles/512766/|Canary Deployment в Kubernetes #1: Gitlab CI]]

<code>
kube1# kubectl delete -f my-webd-deployment.yaml -n my-ns; kubectl apply -f my-webd-deployment.yaml -n my-ns

kube1# cat my-webd-canary-replicaset.yaml
</code><code>
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-webd-canary-rs
spec:
  selector:
    matchLabels:
      track: my-webd-canary-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        track: my-webd-canary-lab
    spec:
      containers:
      - name: my-webd-canary-con
        image: server.corpX.un:5000/student/gowebd:ver1.2
</code><code>
kube1# kubectl -n my-ns apply -f my-webd-canary-replicaset.yaml

kube1# while true; do curl $IP; sleep 0.1; done
       
kube1# kubectl -n my-ns scale replicaset my-webd-canary-rs --replicas 10
        
kube1# kubectl -n my-ns scale deployment my-webd-dep --replicas 0
</code>

===== Шаг 4. canary deployment with ingress and annotations =====

  * [[https://medium.com/@muppedaanvesh/implementing-canary-deployment-in-kubernetes-0be4bc1e1aca|Implementing Canary Deployment in Kubernetes (Ingress resource to expose the canary application)]]
  * [[https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md]]

  * Проверить наличие [[Система Kubernetes#ingress-nginx]] контроллера

<code>
kube1# kubectl -n my-ns delete -f my-webd-canary-replicaset.yaml

kube1# cat my-webd-deployment.yaml
</code><code>
...
  replicas: 1
...
</code><code>
kube1# kubectl apply -f my-webd-deployment.yaml -n my-ns

kube1# cat my-webd-canary-service.yaml
</code><code>
apiVersion: v1
kind: Service
metadata:
  name: my-webd-canary-srv
spec:
  selector:
    app: my-webd-canary-lab
  ports:
  - protocol: TCP
    port: 80
</code><code>
kube1# cat my-webd-canary-deployment.yaml
</code><code>
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-canary-dep
spec:
  selector:
    matchLabels:
      app: my-webd-canary-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-canary-lab
    spec:
      containers:
      - name: my-webd-canary-con
        image: server.corpX.un:5000/student/gowebd:ver1.2
</code><code>
kube1# kubectl apply -f my-webd-canary-service.yaml,my-webd-canary-deployment.yaml -n my-ns
</code><code>
kube1# cat my-webd-ingress-and-canary.yaml
</code><code>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-webd-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: gowebd.corpX.un
    http:
      paths:
      - backend:
          service:
            name: my-webd-srv
            port:
              number: 80
        path: /
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-webd-canary-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
    nginx.ingress.kubernetes.io/canary-by-header: x-my-version
    nginx.ingress.kubernetes.io/canary-by-header-value: canary
spec:
  ingressClassName: nginx
  rules:
  - host: gowebd.corpX.un
    http:
      paths:
      - backend:
          service:
            name: my-webd-canary-srv
            port:
              number: 80
        path: /
        pathType: Prefix
</code><code>
kube1# kubectl apply -f my-webd-ingress-and-canary.yaml -n my-ns

kube1# curl kube1 -H "Host: gowebd.corpX.un" -H "x-my-version: canary"

kube1# while true; do curl kube1 -H "Host: gowebd.corpX.un"; sleep 0.1; done
</code>



===== Шаг 5. canary deployment with Gateway API =====

  * [[https://konghq.com/blog/engineering/gateway-api-vs-ingress|Gateway API vs Ingress: The Future of Kubernetes Networking]]
  * [[https://github.com/istio/istio/blob/master/samples/helloworld/gateway-api/README.md|Configure helloworld using the Kubernetes Gateway API]]
  * [[https://medium.com/@kedarnath93/what-is-gateway-api-in-kubernetes-and-how-does-it-differ-from-ingress-api-aa0404d7fc09|What is Gateway API in Kubernetes and How does it differ from Ingress API?]]

  * [[https://habr.com/ru/companies/vk/articles/515138/|Простое объяснение CRD в Kubernetes и как его использовать]]

  * [[https://github.com/kubernetes-sigs/kustomize]]
  * [[https://habr.com/ru/companies/flant/articles/469179/|Краткое введение в Kustomize]]

  * [[https://istio.io/latest/docs/setup/getting-started/|Istio Getting Started]]

<code>
kube1# kubectl delete -f my-webd-ingress-and-canary.yaml -n my-ns

kube1# kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.2.0" | kubectl apply -f -
</code>

  * [[Система Kubernetes#MetalLB]] с autoAssign: true

<code>
kube1# curl -L https://istio.io/downloadIstio | sh -

kube1# cp -v /root/istio*/bin/istioctl /usr/local/bin/

kube1# istioctl install

kube1# cat my-webd-gateway.yaml
</code><code>
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-webd-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same
</code><code>
kube1# kubectl -n my-ns apply -f my-webd-gateway.yaml

kube1# kubectl -n my-ns get gtw my-webd-gateway
my-webd-gateway   istio   192.168.13.66   True         113m

kube1# cat my-webd-route.yaml
</code><code>
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-webd-route
spec:
  parentRefs:
  - name: my-webd-gateway
  rules:
  - matches:
    - path:
        type: Exact
        value: /
    backendRefs:
    - name: my-webd-srv
      port: 80
      weight: 90
    - name: my-webd-canary-srv
      port: 80
      weight: 10
</code><code>
kube1# kubectl -n my-ns apply -f my-webd-route.yaml

kube1# kubectl -n my-ns describe httproutes.gateway.networking.k8s.io my-webd-route

kube1# IP=$(kubectl -n my-ns get gtw my-webd-gateway -ojsonpath='{.status.addresses[0].value}')

kube1# while true; do curl $IP; sleep 0.1; done
</code>

===== Шаг 6. canary deployment with Istio VirtualService =====

  * [[https://istio.io/latest/blog/2017/0.1-canary/|Canary Deployments using Istio]]
  * [[https://habr.com/ru/companies/nixys/articles/513578/|Canary Deployment в Kubernetes #3: Istio]]
  * [[https://habr.com/ru/companies/otus/articles/770254/|Использование Istio для управления трафиком и мониторинга в микросервисах]]
  * [[https://tetrate.io/blog/header-based-routing-in-istio-without-header-propagation/|Header-Based Routing in Istio without Header Propagation]]
<code>
kube1:~/istio-1.24.3# kubectl apply -f samples/addons

kube1# kubectl get pods -n istio-system

/home/mobaxterm> ssh root@192.168.13.221 -X
kube1# istioctl dashboard kiali &
kube1# firefox &

или

cmder> ssh -L20001:localhost:20001 root@192.168.13.221
kube1# kubectl port-forward svc/kiali 20001:20001 -n istio-system

или

cmder> kubectl port-forward svc/kiali 20001:20001 -n istio-system

http://localhost:20001/

kube1# kubectl delete ns my-ns; kubectl create ns my-ns
kube1# kubectl label namespace my-ns istio-injection=enabled
kube1# kubectl get ns --show-labels

kube1# cat my-webd-deployment-v1-v2.yaml
</code><code>
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-v1-dep
spec:
  selector:
    matchLabels:
      app: my-webd-lab
      version: v1-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        version: v1-lab
    spec:
      containers:
      - name: my-webd-con
        image: server.corp13.un:5000/student/gowebd:ver1.1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webd-v2-dep
spec:
  selector:
    matchLabels:
      app: my-webd-lab
      version: v2-lab
  replicas: 1
  template:
    metadata:
      labels:
        app: my-webd-lab
        version: v2-lab
    spec:
      containers:
      - name: my-webd-con
        image: server.corp13.un:5000/student/gowebd:ver1.2
</code><code>
kube1# kubectl apply -f my-webd-deployment-v1-v2.yaml -n my-ns

kube1# kubectl -n my-ns describe pod my-webd-v1-<TAB> | grep istio

kube1# cat my-webd-service.yaml
</code><code>
apiVersion: v1
kind: Service
metadata:
  name: my-webd-srv
spec:
  selector:
    app: my-webd-lab
  ports:
  - protocol: TCP
    port: 80
    name: http  #need for istio
</code><code>
kube1# kubectl apply -f my-webd-service.yaml -n my-ns
    
kube1# kubectl get svc -n istio-system | grep ingr
istio-ingressgateway   LoadBalancer   10.233.37.214   192.168.13.65   15021:31547/TCP,80:32173/TCP,443:31308/TCP       19h

kube1# cat my-webd-istio-gateway.yaml
</code><code>
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: my-webd-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - gowebd.corp13.un
    - webd.corp13.un
#    - "*"
</code><code>
kube1# kubectl apply -f my-webd-istio-gateway.yaml -n my-ns

kube1# cat my-webd-destrul.yaml
</code><code>
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-webd-destrul
spec:
  host: my-webd-srv
  subsets:
  - name: v1-subset
    labels:
      version: v1-lab
  - name: v2-subset
    labels:
      version: v2-lab
</code><code>
kube1# kubectl apply -f my-webd-destrul.yaml -n my-ns

kube1# cat virtserv-after-gateway.yaml
</code><code>
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtserv-after-gateway
spec:
  hosts:
  - "gowebd.corp13.un"
  gateways:
  - my-webd-gateway
  http:
#  - name: "canary"
#    match:
#    - headers:
#        x-forwarded-for:
#          regex: "192.168.13.*"
##        x-my-version:
##          exact: canary
#    route:
#    - destination:
#        host: frontend
#        subset: v2
  - match:
    - {}
    route:
    - destination:
        host: my-webd-srv
        subset: v1-subset
#        host: frontend
#        subset: v1
      weight: 90
    - destination:
        host: my-webd-srv
        subset: v2-subset
#        host: frontend
#        subset: v2
      weight: 10
</code><code>
kube1# kubectl apply -f virtserv-after-gateway.yaml -n my-ns
      
kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done

kube1# cat frontend.yaml
</code><code>
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        version: v1
    spec:
      containers:
      - image: nginx:1.16.1-alpine
        name: nginx
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-v2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        version: v2
    spec:
      containers:
      - image: nginx:1.17.8-alpine
        name: nginx
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen 80 default_server;
        location / {
           proxy_pass http://my-webd-srv;
           proxy_http_version 1.1;
        }
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: frontend
  name: frontend
spec:
  ports:
  - port: 80
    name: http
    protocol: TCP
  selector:
    app: frontend
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: frontend
spec:
  host: frontend
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
</code><code>
kube1# kubectl apply -f frontend.yaml -n my-ns

kube1# cat virtserv-after-gateway.yaml
</code><code>
...
#        host: my-webd-srv
#        subset: v1-subset
        host: frontend
        subset: v1
...
#        host: my-webd-srv
#        subset: v2-subset
        host: frontend
        subset: v2
...
</code><code>
kube1# kubectl apply -f virtserv-after-gateway.yaml -n my-ns

kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done

kube1# cat my-webd-virtserv-src-lab.yaml
</code><code>
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-webd
spec:
  hosts:
  - my-webd-srv
  gateways: []
  http:
  - match:
    - sourceLabels:
        app: frontend
        version: v1
    route:
    - destination:
        host: my-webd-srv
        subset: v1-subset
        port:
          number: 80
      weight: 100
  - match:
    - sourceLabels:
        app: frontend
        version: v2
    route:
    - destination:
        host: my-webd-srv
        subset: v2-subset
        port:
          number: 80
      weight: 100
</code><code>
kube1# kubectl apply -f my-webd-virtserv-src-lab.yaml -n my-ns

kube1# while true; do curl 192.168.13.65 -H "Host: gowebd.corp13.un"; sleep 0.1; done
</code>

==== Добавление журналов ====

  * [[https://istio.io/latest/docs/tasks/observability/logs/access-log/|Envoy Access Logs ... Default access log format]]

<code>
kube1# vim virtserv-after-gateway.yaml
</code><code>
...
        x-forwarded-for:
          regex: "192.168.13.*"
...
    - {}
    route:
    - destination:
        host: frontend
        subset: v1
</code><code>
kube1# cat telemetry.yaml
</code><code>
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: Envoy
</code><code>
kube1# kubectl apply -f telemetry.yaml -n istio-system

kube1# curl 192.168.13.65 -H "Host: gowebd.corp13.un" -H "x-forwarded-for: 192.168.13.10"

kube1# kubectl -n my-ns logs -l app=my-webd-lab -c istio-proxy -f
kube1# kubectl -n my-ns logs pods/my-webd-v2-<TAB> -c istio-proxy -f
kube1# kubectl -n my-ns logs pods/frontend-v2-<TAB> -c istio-proxy -f
</code><code>
kube1# vim virtserv-after-gateway.yaml
</code><code>
...
#        x-my-version:
#          exact: canary
...
</code><code>
kube1# curl 192.168.13.65 -H "Host: gowebd.corp13.un" -H "x-my-version: canary"

gate# systemctl disable haproxy --now

gate# cat /etc/nginx/sites-available/gowebd
</code><code>
    server {
        listen 80;
        server_name gowebd.corp13.un;
        return 301 https://gowebd.corp13.un:443$request_uri;
    }

    server {
        listen 443 ssl;
        server_name gowebd.corp13.un;
        ssl_certificate /root/gowebd.crt;
        ssl_certificate_key /root/gowebd.key;

        location / {
            proxy_pass http://192.168.13.65;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
        }
    }

    server {
        listen 8080;
        server_name gowebd.corp13.un;
        return 301 https://gowebd.corp13.un:8443$request_uri;
    }

    server {
        listen 8443 ssl;
        server_name gowebd.corp13.un;
        ssl_certificate /root/gowebd.crt;
        ssl_certificate_key /root/gowebd.key;

        location / {
            proxy_pass http://192.168.13.65;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-My-Version "canary";
        }
    }
</code><code>
gate# systemctl enable nginx --now

gate.corp13.un:~# cat /etc/iptables/rules.v4
...
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d 172.16.1.13/32 -i eth1 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.13.10:53
-A PREROUTING -s 172.16.1.113/32 -d 172.16.1.13/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.1.13:8080
-A PREROUTING -s 172.16.1.113/32 -d 172.16.1.13/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.1.13:8443
-A POSTROUTING -s 192.168.13.0/24 -o eth1 -j MASQUERADE
COMMIT
...
gate.corp13.un:~# iptables-restore /etc/iptables/rules.v4

server# curl https://gowebd.corp13.un

win client> https://gowebd.corp13.un
</code>
===== Вопросы? =====

===== Черновик =====
<code>
istioctl install #--set values.global.proxy.privileged=true

~/gowebd-k8s# cat your-istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout  # Log to stdout (or specify a file path)
    accessLogEncoding: JSON     # Or TEXT, depending on your preference
    accessLogFormat: |
      {
        "start_time": "%START_TIME%",
        "method": "%REQ(:METHOD)%",
        "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        "protocol": "%PROTOCOL%",
        "response_code": "%RESPONSE_CODE%",
        "response_flags": "%RESPONSE_FLAGS%",
        "bytes_received": "%BYTES_RECEIVED%",
        "bytes_sent": "%BYTES_SENT%",
        "duration": "%DURATION%",
        "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
        "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
        "user_agent": "%REQ(USER-AGENT)%",
        "custom_header": "%REQ(X-CUSTOM-HEADER)%"
      }


istioctl install -f your-istio-operator.yaml


kube1:~/gowebd-k8s# cat ingress-preserve-xff.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingress-preserve-xff
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          use_remote_address: false # Отключает перезапись X-Forwarded-For

kube1:~/gowebd-k8s# curl -H "x-forwarded-for: 123.123.123" http://webd.corp13.un

---
Не заработало:
kube1:~/gowebd-k8s# cat enable-header-logging.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: enable-header-logging
spec:
  workloadSelector:
    labels:
      app: my-webd
  configPatches:
#    - applyTo: NETWORK_FILTER
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND  # Или SIDECAR_OUTBOUND для исходящего трафика
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inline_code: |
              function envoy_on_request(request_handle)
                local headers = request_handle:headers()
                for key, value in pairs(headers) do
                  request_handle:logInfo("Request Header: " .. key .. " = " .. value)
                end
              end

              function envoy_on_response(response_handle)
                local headers = response_handle:headers()
                for key, value in pairs(headers) do
                  response_handle:logInfo("Response Header: " .. key .. " = " .. value)
                end
              end




root@node1:~/gowebd-k8s# istioctl proxy-config listeners -n my-ns my-webd-v1-5dfd6f4ff4-trjrh -o json| less

~/gowebd-k8s# kubectl -n my-ns logs -l app=my-webd --all-containers
~/gowebd-k8s# kubectl -n my-ns logs -l app=my-webd -c istio-proxy -f

</code>