Differences

This shows you the differences between two versions of the page.

--- система_kubernetes [2022/07/12 13:23]
val [Работа со своим репозиторием]
+++ система_kubernetes [2023/09/21 08:18] (current)
val [Установка minikube]
@@ Line 1: / Line 1: @@
 ====== Система Kubernetes ======
+  * [[https://kubernetes.io/ru/docs/home/|Документация по Kubernetes (на русском)]]
   * [[https://youtu.be/sLQefhPfwWE|youtube Введение в Kubernetes на примере Minikube]]
@@ Line 12: / Line 14: @@
   * [[https://github.com/dgkanatsios/CKAD-exercises|A set of exercises that helped me prepare for the Certified Kubernetes Application Developer exam]]
-===== kubectl =====
+===== Инструмент командной строки kubectl =====
   * [[https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands]]
+==== Установка ====
 <code>
-root@gate.corp13.un:~# curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
+# curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
-root@gate.corp13.un:~# chmod +x kubectl
-root@gate.corp13.un:~# mv kubectl /usr/local/bin/
-student@node2:~$ tar zcf kube-config.tar.gz .kube/config .minikube/ca.crt .minikube/profiles/minikube
+# chmod +x kubectl
-gitlab-runner@gate:~$ scp student@node2:kube-config.tar.gz .
+# mv kubectl /usr/local/bin/
+</code>
-gitlab-runner@gate:~$ tar -xvf kube-config.tar.gz
+==== Подключение к кластеру ====
-gitlab-runner@gate:~$ cat .kube/config
+<code>
+gitlab-runner@server:~$ mkdir .kube/
+gitlab-runner@server:~$ scp root@node1:.kube/config .kube/config
+gitlab-runner@server:~$ cat .kube/config
 </code><code>
 ...
-    certificate-authority: /home/gitlab-runner/.minikube/ca.crt
+    server: https://node1:6443
 ...
-    client-certificate: /home/gitlab-runner/.minikube/profiles/minikube/client.crt
+</code><code>
-    client-key: /home/gitlab-runner/.minikube/profiles/minikube/client.key
+gitlab-runner@server:~$ kubectl get all -o wide --all-namespaces
+gitlab-runner@server:~$ kubectl get all -o wide -A
+</code>
+=== Настройка автодополнения ===
+<code>
+gitlab-runner@server:~$ source <(kubectl completion bash)
+</code>
+=== Подключение к другому кластеру ===
+<code>
+gitlab-runner@server:~$ scp root@kube1:.kube/config .kube/config_kube1
+gitlab-runner@server:~$ cat .kube/config_kube1
+</code><code>
+...
+    .kube/config_kube1
 ...
 </code><code>
-gitlab-runner@gate:~$ kubectl get all
+gitlab-runner@server:~$ export KUBECONFIG=~/.kube/config_kube1
+gitlab-runner@server:~$ kubectl get nodes
 </code>
-===== minikube =====
+===== Установка minikube =====
   * [[https://www.linuxtechi.com/how-to-install-minikube-on-ubuntu/|How to Install Minikube on Ubuntu 20.04 LTS / 21.04]]
   * [[https://minikube.sigs.k8s.io/docs/start/|Documentation/Get Started/minikube start]]
-  * Технология Docker [[Технология Docker#Предоставление прав непривилегированным пользователям]]
 <code>
-student@node3:~$ minikube delete
+root@server:~# apt install -y curl wget apt-transport-https
-student@node3:~$ minikube start --driver=docker --insecure-registry "server.corp13.un:5000"
+root@server:~# wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
-ИЛИ
+root@server:~# mv minikube-linux-amd64 /usr/local/bin/minikube
-</code><code>
-student@node2:~$ sudo apt install conntrack
-https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
+root@server:~# chmod +x /usr/local/bin/minikube
-...
+</code>
-wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz
+  * Технология Docker [[Технология Docker#Предоставление прав непривилегированным пользователям]]
+<code>
+gitlab-runner@server:~$ ### minikube delete
+gitlab-runner@server:~$ ### rm -rv .minikube/
+gitlab-runner@server:~$ time minikube start --driver=docker --insecure-registry "server.corpX.un:5000"
+real    29m8.320s
 ...
-student@node2:~$ minikube start --driver=none --insecure-registry "server.corp13.un:5000"
+gitlab-runner@server:~$ minikube status
-</code><code>
-student@node3:~$ minikube status
-student@node3:~$ minikube ip
+gitlab-runner@server:~$ minikube ip
-student@node3:~$ minikube addons list
+gitlab-runner@server:~$ minikube addons list
-student@node3:~$ minikube addons configure registry-creds
+gitlab-runner@server:~$ minikube addons configure registry-creds   #Не нужно для registry попубличных проектов
 ...
 Do you want to enable Docker Registry? [y/n]: y
--- Enter docker registry server url: http://server.corp13.un:5000
+-- Enter docker registry server url: http://server.corpX.un:5000
 -- Enter docker registry username: student
 -- Enter docker registry password:
 ...
-student@node3:~$ minikube addons enable registry-creds
+gitlab-runner@server:~$ minikube addons enable registry-creds
-student@node3:~$ minikube dashboard &
+gitlab-runner@server:~$ minikube kubectl -- get pods -A
+gitlab-runner@server:~$ alias kubectl='minikube kubectl --'
+gitlab-runner@server:~$ kubectl get pods -A
+</code>
+или
+  * [[#Инструмент командной строки kubectl]]
+<code>
+gitlab-runner@server:~$ ###minikube stop
+gitlab-runner@server:~$ ###minikube start
+</code>
+===== Кластер Kubernetes =====
+==== Развертывание через kubeadm ====
+  * [[https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/|kubernetes.io Creating a cluster with kubeadm]]
+  * [[https://infoit.com.ua/linux/kak-ustanovit-kubernetes-na-ubuntu-20-04-lts/|Как установить Kubernetes на Ubuntu 20.04 LTS]]
+  * [[https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/|How to Install Kubernetes Cluster on Ubuntu 22.04]]
+  * [[https://www.linuxtechi.com/install-kubernetes-cluster-on-debian/|https://www.linuxtechi.com/install-kubernetes-cluster-on-debian/]]
+  * [[https://www.cloud4y.ru/blog/installation-kubernetes/|Установка Kubernetes]]
+=== Подготовка узлов ===
+<code>
+node1# ssh-keygen
+node1# ssh-copy-id node2
+node1# ssh-copy-id node3
+node1# bash -c '
+swapoff -a
+ssh node2 swapoff -a
+ssh node3 swapoff -a
+'
+node1# bash -c '
+sed -i"" -e "/swap/s/^/#/" /etc/fstab
+ssh node2 sed -i"" -e "/swap/s/^/#/" /etc/fstab
+ssh node3 sed -i"" -e "/swap/s/^/#/" /etc/fstab
+'
+</code>
+=== Установка ПО ===
+<code>
+node1# bash -c '
+http_proxy=http://proxy.isp.un:3128/ apt -y install apt-transport-https curl
+ssh node2 http_proxy=http://proxy.isp.un:3128/ apt -y install apt-transport-https curl
+ssh node3 http_proxy=http://proxy.isp.un:3128/ apt -y install apt-transport-https curl
+'
+node1# bash -c '
+curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
+ssh node2 "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add"
+ssh node3 "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add"
+'
+node1# bash -c '
+apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
+ssh node2 apt-add-repository \"deb http://apt.kubernetes.io/ kubernetes-xenial main\"
+ssh node3 apt-add-repository \"deb http://apt.kubernetes.io/ kubernetes-xenial main\"
+'
+node1# bash -c '
+http_proxy=http://proxy.isp.un:3128/ apt -y install kubeadm kubelet kubectl kubernetes-cni
+ssh node2 http_proxy=http://proxy.isp.un:3128/ apt -y install kubeadm kubelet kubectl kubernetes-cni
+ssh node3 http_proxy=http://proxy.isp.un:3128/ apt -y install kubeadm kubelet kubectl kubernetes-cni
+'
+</code>
+=== Инициализация master ===
+<code>
+root@node1:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.X.201
+root@node1:~# mkdir -p $HOME/.kube
+root@node1:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+root@node1:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
+root@node1:~# kubectl get pod -o wide --all-namespaces
+root@node1:~# kubectl get --raw='/readyz?verbose'
+</code>
+  * Может понадобиться в случае возникновения ошибки [[https://github.com/containerd/containerd/issues/4581|[ERROR CRI]: container runtime is not running]]
+<code>
+node1# bash -c '
+rm /etc/containerd/config.toml
+systemctl restart containerd
+ssh node2 rm /etc/containerd/config.toml
+ssh node2 systemctl restart containerd
+ssh node3 rm /etc/containerd/config.toml
+ssh node3 systemctl restart containerd
+'
+</code>
+=== Подключение worker ===
+<code>
+root@node2_3:~# curl -k https://node1:6443/livez?verbose
+</code>
+  * [[https://github.com/containerd/containerd/issues/4581|[ERROR CRI]: container runtime is not running]]
+<code>
+root@node2_3:~# kubeadm join 192.168.X.201:6443 --token NNNNNNNNNNNNNNNNNNNN \
+        --discovery-token-ca-cert-hash sha256:NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
+</code>
+=== Проверка состояния ===
+<code>
+root@node1:~# kubectl cluster-info
+root@node1:~# kubectl get nodes -o wide
+</code>
+=== Настройка доступа к Insecure Private Registry ===
+  * [[https://github.com/containerd/containerd/issues/4938|Unable to pull image from insecure registry, http: server gave HTTP response to HTTPS client #4938]]
+  * [[https://github.com/containerd/containerd/issues/3847|Containerd cannot pull image from insecure registry #3847]]
+  * [[https://mrzik.medium.com/how-to-configure-private-registry-for-kubernetes-cluster-running-with-containerd-cf74697fa382|How to Configure Private Registry for Kubernetes cluster running with containerd]]
+  * [[https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header|containerd/docs/PLUGINS.md migrate config v1 to v2]]
+== Docker (устарело, не обязательно) ==
+  * Docker [[Технология Docker#Insecure Private Registry]]
+<code>
+node1# scp server:/etc/docker/daemon.json /etc/docker/daemon.json
+node1# bash -c '
+scp /etc/docker/daemon.json node2:/etc/docker/daemon.json
+scp /etc/docker/daemon.json node3:/etc/docker/daemon.json
+service docker restart
+ssh node2 service docker restart
+ssh node3 service docker restart
+'
+# don't work in cri-tools 1.25, need public project in GitLab (Settings->General->Visibility, project features, permissions)
+### node1# docker login http://server.corpX.un:5000
+### node1# bash -c '
+ssh node2 mkdir -p .docker
+ssh node3 mkdir -p .docker
+scp ~/.docker/config.json node2:.docker/config.json
+scp ~/.docker/config.json node3:.docker/config.json
+'
+</code>
+== сontainerd ==
+<code>
+root@node1:~# mkdir /etc/containerd/
+root@node1:~# cat /etc/containerd/config.toml
+</code><code>
+version = 2
+[plugins."io.containerd.grpc.v1.cri".registry]
+  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
+    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."server.corpX.un:5000"]
+      endpoint = ["http://server.corpX.un:5000"]
+  [plugins."io.containerd.grpc.v1.cri".registry.configs]
+    [plugins."io.containerd.grpc.v1.cri".registry.configs."server.corpX.un:5000".tls]
+      insecure_skip_verify = true
+# don't work in cri-tools 1.25, need public project
+#[plugins."io.containerd.grpc.v1.cri".registry.configs."server.corpX.un:5000".auth]
+#      auth = "c3R1ZGVudDpwYXNzd29yZA=="
+</code><code>
+node1# bash -c '
+ssh node2 mkdir /etc/containerd/
+ssh node3 mkdir /etc/containerd/
+scp /etc/containerd/config.toml node2:/etc/containerd/config.toml
+scp /etc/containerd/config.toml node3:/etc/containerd/config.toml
+systemctl restart containerd
+ssh node2 systemctl restart containerd
+ssh node3 systemctl restart containerd
+'
+root@nodeN:~# containerd config dump | less
+</code>
+Проверка
+<code>
+root@nodeN:~# crictl -r unix:///run/containerd/containerd.sock pull server.corpX.un:5000/student/webd
+</code>
+==== Развертывание через Kubespray ====
+  * [[https://github.com/kubernetes-sigs/kubespray]]
+  * [[https://habr.com/ru/companies/domclick/articles/682364/|Самое подробное руководство по установке высокодоступного (почти ಠ ͜ʖ ಠ ) Kubernetes-кластера]]
+  * [[https://habr.com/ru/companies/X5Tech/articles/645651/|Bare-metal kubernetes-кластер на своём локальном компьютере]]
+  * [[https://internet-lab.ru/k8s_kubespray|Kubernetes — установка через Kubespray]]
+  * [[https://www.mshowto.org/en/ubuntu-sunucusuna-kubespray-ile-kubernetes-kurulumu.html|Installing Kubernetes on Ubuntu Server with Kubespray]]
+<code>
+kube1# ssh-keygen
+kube1# ssh-copy-id kube1;ssh-copy-id kube2;ssh-copy-id kube3;ssh-copy-id kube4;
+kube1# apt update
+kube1# apt install python3-pip -y
+kube1# git clone https://github.com/kubernetes-sigs/kubespray
+kube1# cd kubespray/
+~/kubespray# grep -r containerd_insecure_registries .
+~/kubespray# git log
+~/kubespray# git branch -r
+~/kubespray# ### git checkout origin/release-2.22
+~/kubespray# git tag -l
+~/kubespray# ### git checkout tags/v2.22.1
+~/kubespray# git checkout 4c37399c7582ea2bfb5202c3dde3223f9c43bf59
+~/kubespray# ### git checkout master
+~/kubespray# time pip3 install -r requirements.txt
+real    1m48.202s
+~/kubespray# cp -rfp inventory/sample inventory/mycluster
+~/kubespray# declare -a IPS=(kube1,192.168.X.221 kube2,192.168.X.222 kube3,192.168.X.223)
+~/kubespray# CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+~/kubespray# time ansible-playbook -i inventory/mycluster/hosts.yaml cluster.yml
+real    45m31.796s
+kube1# less ~/.kube/config
+~/kubespray# ###time ansible-playbook -i inventory/mycluster/hosts.yaml reset.yml
+real    7m31.796s
+</code>
+=== Добавление узла через Kubespray ===
+<code>
+~/kubespray# cat inventory/mycluster/hosts.yaml
+</code><code>
 ...
-Opening http://127.0.0.1:NNNNN/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser
+    node4:
+      ansible_host: 192.168.X.204
+      ip: 192.168.X.204
+      access_ip: 192.168.X.204
 ...
-/home/mobaxterm> ssh -L NNNNN:localhost:NNNNN student@192.168.13.230
+    kube_node:
-Теперь, та же ссылка работает на win host системе
+...
+        node4:
+...
+</code><code>
+~/kubespray# ansible-playbook -i inventory/mycluster/hosts.yaml scale.yml
+$ kubectl get nodes -o wide
 </code>
+=== Добавление insecure_registries через Kubespray ===
+<code>
+~/kubespray# cat inventory/mycluster/group_vars/all/containerd.yml
+</code><code>
+...
+containerd_insecure_registries:
+  "server.corpX.un:5000": "http://server.corpX.un:5000"
+containerd_registry_auth:
+  - registry: server.corpX.un:5000
+    username: student
+    password: Pa$$w0rd
+...
+</code><code>
+~/kubespray# time ansible-playbook -i inventory/mycluster/hosts.yaml cluster.yml
+user    46m37.151s
+# less /etc/containerd/config.toml
+</code>
+=== Управление дополнениями через Kubespray ===
+<code>
+~/kubespray# cat inventory/mycluster/group_vars/k8s_cluster/addons.yml
+</code><code>
+...
+helm_enabled: true
+...
+ingress_nginx_enabled: true
+ingress_nginx_host_network: true
+...
+</code>
 ===== Базовые объекты k8s =====
@@ Line 92: / Line 406: @@
   * [[https://kubernetes.io/ru/docs/reference/kubectl/docker-cli-to-kubectl/|kubectl для пользователей Docker]]
   * [[https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/|Run a Stateless Application Using a Deployment]]
 <code>
-$ kubectl create deployment my-debian --image=debian -- "sleep" "3600"
+$ kubectl api-resources
+$ kubectl run my-debian --image=debian -- "sleep" "3600"
 $ kubectl get all
-$ kubectl get deployments
+$ kubectl delete pod my-debian
-$ kubectl get pods
+$ kubectl create deployment my-debian --image=debian -- "sleep" "3600"
+$ kubectl get deployments
+</code>
+  * [[#Настройка автодополнения]]
+<code>
 $ kubectl attach my-debian-NNNNNNNNN-NNNNN
@@ Line 108: / Line 429: @@
 $ kubectl get deployment my-debian -o yaml
+</code>
+  * [[Переменные окружения]]
+<code>
 $ kubectl edit deployment my-debian
+$ kubectl get pods -o wide
 $ kubectl delete deployment my-debian
-</code><code>
+</code>
-  [[https://kubernetes.io/docs/reference/glossary/?all=true#term-manifest|    Kubernetes Documentation Reference Glossary/Manifest]]
+  * [[https://kubernetes.io/docs/reference/glossary/?all=true#term-manifest|Kubernetes Documentation Reference Glossary/Manifest]]
-</code><code>
+<code>
 $ cat my-debian-deployment.yaml
 </code><code>
 apiVersion: apps/v1
-kind: Deployment
+kind: ReplicaSet
+#kind: Deployment
 metadata:
   name: my-debian
@@ Line 125: / Line 451: @@
     matchLabels:
       app: my-debian
+  replicas: 2
   template:
     metadata:
@@ Line 137: / Line 464: @@
       restartPolicy: Always
 </code><code>
-$ kubectl create -f my-debian-deployment.yaml
+$ kubectl apply -f my-debian-deployment.yaml
 ...
 $ kubectl delete -f my-debian-deployment.yaml
 </code>
 ==== namespace для своего приложения ====
+  * [[https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-volumes-example-nfs-persistent-volume.html|How to use an NFS volume]]
 <code>
 $ kubectl create namespace my-ns
@@ Line 147: / Line 477: @@
 $ kubectl get namespaces
-$ ### kubectl create deployment my-webd --image=server.corp13.un:5000/student/webd:latest --replicas=2 -n my-ns
+$ ### kubectl create deployment my-webd --image=server.corpX.un:5000/student/webd:latest --replicas=2 -n my-ns
+$ ### kubectl delete deployment my-webd -n my-ns
+$ cd webd/
 $ cat my-webd-deployment.yaml
@@ Line 155: / Line 489: @@
 metadata:
   name: my-webd
-  namespace: my-ns
 spec:
   selector:
@@ Line 168: / Line 501: @@
       containers:
       - name: my-webd
-        image: server.corp13.un:5000/student/webd:latest
+#        image: server.corpX.un:5000/student/webd
+#        image: server.corpX.un:5000/student/webd:ver1.N
+#        livenessProbe:
+#          httpGet:
+#            port: 80
+#        volumeMounts:
+#        - name: nfs-volume
+#          mountPath: /var/www
+#      volumes:
+#      - name: nfs-volume
+#        nfs:
+#          server: server.corpX.un
+#          path: /var/www
 </code><code>
-$ kubectl apply -f my-webd-deployment.yaml
+$ kubectl apply -f my-webd-deployment.yaml -n my-ns
-$ kubectl get all -n my-ns
+$ kubectl get all -n my-ns -o wide
+$ kubectl describe -n my-ns pod/my-webd-NNNNNNNNNN-NNNNN
 $ kubectl scale deployment my-webd --replicas=3 -n my-ns
+$ kubectl delete pod/my-webd-NNNNNNNNNN-NNNNN -n my-ns
 </code>
@@ Line 180: / Line 532: @@
   * [[https://kubernetes.io/docs/concepts/services-networking/service/|Kubernetes Documentation Concepts Services, Load Balancing, and Networking Service]]
+  * [[https://stackoverflow.com/questions/33069736/how-do-i-get-logs-from-all-pods-of-a-kubernetes-replication-controller|How do I get logs from all pods of a Kubernetes replication controller?]]
 <code>
 $ ### kubectl expose deployment my-webd --type=NodePort --port=80 -n my-ns
+$ ### kubectl delete svc my-webd -n my-ns
 $ cat my-webd-service.yaml
@@ Line 190: / Line 546: @@
 metadata:
   name: my-webd
-  namespace: my-ns
 spec:
-  type: NodePort
+#  type: NodePort
   selector:
     app: my-webd
@@ Line 198: / Line 553: @@
   - protocol: TCP
     port: 80
-    targetPort: 80
+#    nodePort: 30111
-status:
-  loadBalancer: {}
 </code><code>
-$ kubectl apply -f my-webd-service.yaml
+$ kubectl apply -f my-webd-service.yaml -n my-ns
 $ kubectl get svc my-webd -n my-ns
 NAME              TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
-my-webd-svc   NodePort   10.102.135.146   <none>        80:30350/TCP   18h
+my-webd-svc   NodePort   10.102.135.146   <none>        80:NNNNN/TCP   18h
-student@node3:~$ minikube service my-webd -n my-ns --url
+$ kubectl describe svc my-webd -n my-ns
-http://192.168.49.2:30350
-student@node3:~$ curl $(minikube service my-webd -n my-ns --url)
+$ curl http://node1,2,3:NNNNN
+на "самодельном kubeadm" кластере работает не стабильно
+$ minikube service list
+$ minikube service my-webd -n my-ns --url
+http://192.168.49.2:NNNNN
+$ curl $(minikube service my-webd -n my-ns --url)
+$ kubectl logs -l app=my-webd -n my-ns
+(доступны опции -f, --tail=2000, --previous)
+</code>
+==== Удаление объектов ====
+<code>
+$ kubectl get all -n my-ns
+$ kubectl delete -n my-ns -f my-webd-deployment.yaml,my-webd-service.yaml
+или
+$ kubectl delete namespace my-ns
 </code>
 ==== Ingress ====
+  * [[https://kubernetes.github.io/ingress-nginx/deploy/#quick-start|NGINX ingress controller quick-start]]
+=== Minikube ingress-nginx-controller ===
   * [[https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/|Set up Ingress on Minikube with the NGINX Ingress Controller]]
-  * [[https://stackoverflow.com/questions/33069736/how-do-i-get-logs-from-all-pods-of-a-kubernetes-replication-controller|How do I get logs from all pods of a Kubernetes replication controller?]]
+  * [[https://www.golinuxcloud.com/kubectl-port-forward/|kubectl port-forward examples in Kubernetes]]
 <code>
-student@node2:~$ minikube addons enable ingress
+server# cat /etc/bind/corpX.un
+</code><code>
+...
+webd A 192.168.49.2
+</code><code>
+gitlab-runner@server:~$ minikube addons enable ingress
+</code>
-gitlab-runner@gate:~/webd$ cat my-webd-ingress.yaml
+=== Сloud ingress-nginx-controller и port-forward  ===
+  * [[https://stackoverflow.com/questions/61616203/nginx-ingress-controller-failed-calling-webhook|Nginx Ingress Controller - Failed Calling Webhook]]
+<code>
+server# cat /etc/bind/corpX.un
+</code><code>
+...
+webd            A       192.168.X.201
+gowebd          CNAME   webd
+</code><code>
+$ curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/cloud/deploy.yaml
+$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/cloud/deploy.yaml
+$ kubectl get all -n ingress-nginx -o wide
+...
+pod/ingress-nginx-controller...  ... Running ...
+...
+$ kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission   # Понадобилось см. ссылку выше
+root@node1:~# kubectl port-forward --namespace=ingress-nginx --address 0.0.0.0 service/ingress-nginx-controller 80:80
+$ kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/cloud/deploy.yaml
+</code>
+=== Baremetal ingress-nginx-controller и port-forward  ===
+  * [[https://stackoverflow.com/questions/51511547/empty-address-kubernetes-ingress|Empty ADDRESS kubernetes ingress]]
+<code>
+server# cat /etc/bind/corpX.un
+</code><code>
+...
+webd            A       192.168.X.202
+                A       192.168.X.203
+gowebd          CNAME   webd
+</code><code>
+$ curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/baremetal/deploy.yaml | tee ingress-nginx.controller-v1.3.1.baremetal.yaml
+$ cat ingress-nginx.controller-v1.3.1.baremetal.yaml
+</code><code>
+...
+kind: Deployment
+...
+spec:
+...
+  replicas: 3    ### insert this (equial count of worker nodes)
+  template:
+...
+      terminationGracePeriodSeconds: 300
+      hostNetwork: true                    ###insert this
+      volumes:
+...
+</code><code>
+$ kubectl apply -f ingress-nginx.controller-v1.3.1.baremetal.yaml
+$ ### kubectl delete -f ingress-nginx.controller-v1.3.1.baremetal.yaml
+</code>
+=== ingress example ===
+<code>
+gitlab-runner@server:~/webd$ ### kubectl create ingress my-ingress --class=nginx --rule="webd.corpX.un/*=my-webd:80" -n my-ns
+gitlab-runner@server:~$ cat my-ingress.yaml
 </code><code>
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
-  name: my-webd
+  name: my-ingress
-  namespace: my-ns
-  annotations:
-    nginx.ingress.kubernetes.io/rewrite-target: /$1
 spec:
+  ingressClassName: nginx
+#  tls:
+#  - hosts:
+#    - gowebd.corpX.un
+#    secretName: gowebd-tls
   rules:
-    - host: webd.corp13.un
+  - host: webd.corpX.un
-      http:
+    http:
-        paths:
+      paths:
-          - path: /(.*)
+      - backend:
-            pathType: Prefix  # Попробовать: ImplementationSpecific
+          service:
-            backend:
+            name: my-webd
-              service:
+            port:
-                name: my-webd
+              number: 80
-                port:
+        path: /
-                  number: 80
+        pathType: Prefix
+  - host: gowebd.corpX.un
+    http:
+      paths:
+      - backend:
+          service:
+            name: my-gowebd
+            port:
+              number: 80
+        path: /
+        pathType: Prefix
 </code><code>
-$ kubectl apply -f my-webd-ingress.yaml
+$ kubectl apply -f my-ingress.yaml -n my-ns
 $ kubectl get ingress -n my-ns
+NAME      CLASS   HOSTS                             ADDRESS                         PORTS   AGE
+my-webd   nginx   webd.corpX.un,gowebd.corpX.un   192.168.X.202,192.168.X.203   80      14m
-root@gate.corp13.un:~# host webd
-webd.corp13.un is an alias for node2.corp13.un.
-node2.corp13.un has address 192.168.13.220
-$ curl webd.corp13.un
+$ curl webd.corpX.un
+$ curl gowebd.corpX.un
-$ kubectl logs -l app=my-webd -n my-ns
+$ ### kubectl delete ingress my-ingress -n my-ns
 </code>
-==== Удаление объектов ====
+=== ingress-tls ===
+  * [[https://devopscube.com/configure-ingress-tls-kubernetes/|How To Configure Ingress TLS/SSL Certificates in Kubernetes]]
 <code>
-$ kubectl delete -n my-ns -f my-webd-deployment.yaml,my-webd-service.yaml,my-webd-ingress.yaml
+$ kubectl create secret tls gowebd-tls --key gowebd.key --cert gowebd.crt -n my-ns
+$ kubectl get secrets -n my-ns
-или
+$ kubectl get secret/gowebd-tls -o yaml -n my-ns
-$ kubectl delete namespace my-ns
+$ curl https://gowebd.corpX.un -kv
+$ ###kubectl delete secret/gowebd-tls -n my-ns
 </code>
-==== Пример с nfs volume ====
+==== Volumes ====
-  * [[https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-volumes-example-nfs-persistent-volume.html|How to use an NFS volume]]
+<code>
+root@node1:~# # ssh node2 mkdir /disk2
+root@node1:~# ssh node2 touch /disk2/disk2_node2
+root@node1:~# kubectl label nodes node2 disk2=yes
+root@node1:~# kubectl get nodes --show-labels
+root@node1:~# ###kubectl label nodes node2 disk2-
+root@node1:~# cat my-debian-deployment.yaml
+</code><code>
+...
+        args: ["-c", "while true; do echo hello; sleep 3;done"]
+        volumeMounts:
+          - name: my-disk2-volume
+            mountPath: /data
+#        volumeMounts:
+#          - name: data
+#            mountPath: /data
+      volumes:
+        - name: my-disk2-volume
+          hostPath:
+            path: /disk2/
+      nodeSelector:
+        disk2: "yes"
+#      volumes:
+#      - name: data
+#        persistentVolumeClaim:
+#          claimName: my-ha-pvc-sz64m
+</code><code>
+root@node1:~# kubectl apply -f my-debian-deployment.yaml
+root@node1:~# kubectl get all -o wide
+</code>
+  * [[https://qna.habr.com/q/629022|Несколько Claim на один Persistent Volumes?]]
+  * [[https://serveradmin.ru/hranilishha-dannyh-persistent-volumes-v-kubernetes/|Хранилища данных (Persistent Volumes) в Kubernetes]]
+  * [[https://stackoverflow.com/questions/59915899/limit-persistent-volume-claim-content-folder-size-using-hostpath|Limit persistent volume claim content folder size using hostPath]]
 <code>
-$ cat my-webd-nfs-deployment.yaml
+root@node1:~# cat my-ha-pv.yaml
+</code><code>
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: my-pv-node2-sz-128m-num-001
+  labels:
+    type: local
+spec:
+  storageClassName: my-ha-sc
+  capacity:
+    storage: 128Mi
+  accessModes:
+    - ReadWriteMany
+  hostPath:
+    path: /disk2
+  persistentVolumeReclaimPolicy: Retain
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+      - matchExpressions:
+        - key: kubernetes.io/hostname
+          operator: In
+          values:
+          - node2
+</code><code>
+root@node1:~# kubectl apply -f my-ha-pv.yaml
+root@node1:~# kubectl get persistentvolume
 ...
+root@node1:~# cat my-ha-pvc.yaml
+</code><code>
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: my-ha-pvc-sz64m
+spec:
+  storageClassName: my-ha-sc
+  accessModes:
+    - ReadWriteMany
+  resources:
+    requests:
+      storage: 64Mi
+</code><code>
+root@node1:~# kubectl apply -f my-ha-pvc.yaml
+root@node1:~# kubectl get persistentvolumeclaims
+...
+</code>
+==== ConfigMap ====
+  * [[https://www.aquasec.com/cloud-native-academy/kubernetes-101/kubernetes-configmap/|Kubernetes ConfigMap: Creating, Viewing, Consuming & Managing]]
+  * [[https://blog.lapw.at/how-to-enable-ssh-into-a-kubernetes-pod/|How to enable SSH connections into a Kubernetes pod]]
+<code>
+root@node1:~# cat sshd_config
+</code><code>
+PermitRootLogin yes
+PasswordAuthentication no
+ChallengeResponseAuthentication no
+UsePAM no
+</code><code>
+root@node1:~# kubectl create configmap ssh-config --from-file=sshd_config --dry-run=client -o yaml
+...
+server:~# cat .ssh/id_rsa.pub
+...
+root@node1:~# cat my-openssh-server-deployment.yaml
+</code><code>
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ssh-config
+data:
+  sshd_config: |
+    PermitRootLogin yes
+    PasswordAuthentication no
+    ChallengeResponseAuthentication no
+    UsePAM no
+  authorized_keys: |
+    ssh-rsa AAAAB.....C0zOcZ68= root@server.corpX.un
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: my-openssh-server
+spec:
+  selector:
+    matchLabels:
+      app: my-openssh-server
+  template:
+    metadata:
+      labels:
+        app: my-openssh-server
     spec:
       containers:
-      - name: my-webd
+      - name: my-openssh-server
-        image: server.corp13.un:5000/student/webd:latest
+        image: linuxserver/openssh-server
+        command: ["/bin/sh"]
+        args: ["-c", "/usr/bin/ssh-keygen -A; usermod -p '*' root; /usr/sbin/sshd.pam -D"]
+        ports:
+        - containerPort: 22
         volumeMounts:
-        - name: nfs-volume
+        - name: ssh-volume
-          mountPath: /var/www
+          subPath: sshd_config
+          mountPath: /etc/ssh/sshd_config
+        - name: ssh-volume
+          subPath: authorized_keys
+          mountPath: /root/.ssh/authorized_keys
       volumes:
-      - name: nfs-volume
+      - name: ssh-volume
-        nfs:
+        configMap:
-          server: 192.168.13.1
+          name: ssh-config
-          path: /var/www
+---
-</code>
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-openssh-server
+spec:
+  type: NodePort
+  ports:
+  - port: 22
+    nodePort: 32222
+  selector:
+    app: my-openssh-server
+</code><code>
+root@node1:~# kubectl apply -f my-openssh-server-deployment.yaml
+root@node1:~# iptables-save | grep 32222
+root@node1:~# ###kubectl exec -ti my-openssh-server-NNNNNNNN-NNNNN -- bash
+server:~# ssh -p 32222 nodeN
+Welcome to OpenSSH Server
+my-openssh-server-NNNNNNNN-NNNNN:~#
+</code>
 ==== Пример с multi container pod ====
@@ Line 313: / Line 952: @@
       containers:
       - name: my-webd
-        image: server.corp13.un:5000/student/webd:latest
+        image: server.corpX.un:5000/student/webd:latest
         volumeMounts:
         - name: html
@@ Line 359: / Line 998: @@
-==== Установка ====
+==== Установка Helm ====
   * [[https://helm.sh/docs/intro/install/|Installing Helm]]
 <code>
-$ wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
+server# wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
-$ tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
+# tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
-$ sudo mv linux-amd64/helm /usr/local/bin/helm
+# mv linux-amd64/helm /usr/local/bin/helm
 </code>
+==== Работа с готовыми Charts ====
+  * [[https://kubernetes.github.io/ingress-nginx/deploy/|NGINX Ingress Controller Installation Guide]]
+  * [[https://stackoverflow.com/questions/56915354/how-to-install-nginx-ingress-with-hostnetwork-on-bare-metal|stackoverflow How to install nginx-ingress with hostNetwork on bare-metal?]]
+  * [[https://devpress.csdn.net/cloud/62fc8e7e7e66823466190055.html|devpress.csdn.net How to install nginx-ingress with hostNetwork on bare-metal?]]
+<code>
+$ helm upgrade ingress-nginx --install ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
+пока не заработало --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet
+$ helm list --namespace ingress-nginx
+$ helm list -A
+$ kubectl get all -n ingress-nginx -o wide
+$ ### helm delete ingress-nginx --namespace ingress-nginx
+</code>
 ==== Развертывание своего приложения ====
   * [[https://opensource.com/article/20/5/helm-charts|How to make a Helm chart in 10 minutes]]
   * [[https://stackoverflow.com/questions/49812830/helm-upgrade-with-same-chart-version-but-different-docker-image-tag|Helm upgrade with same chart version, but different Docker image tag]]
+  * [[https://stackoverflow.com/questions/69817305/how-set-field-app-version-in-helm3-chart|how set field app-version in helm3 chart?]]
 <code>
-$ helm create webd-chart
+gitlab-runner@server:~/gowebd-k8s$ helm create webd-chart
+$ less webd-chart/templates/deployment.yaml
 $ cat webd-chart/Chart.yaml
@@ Line 387: / Line 1047: @@
 ...
 appVersion: "latest"
+#appVersion: ver1.7   #for vanilla argocd
 </code><code>
 $ cat webd-chart/values.yaml
 </code><code>
 ...
+replicaCount: 2
 image:
-  repository: server.corp13.un:5000/student/webd
+  repository: server.corpX.un:5000/student/webd
   pullPolicy: Always
 ...
@@ Line 399: / Line 1062: @@
 ...
 service:
-  type: NodePort
+#  type: NodePort
 ...
 ingress:
   enabled: true
+  className: "nginx"
 ...
   hosts:
-    - host: webd.corp13.un
+    - host: webd.corpX.un
+...
+#  tls: []
+#  tls:
+#    - secretName: gowebd-tls
+#      hosts:
+#        - gowebd.corpX.un
 ...
 </code><code>
@@ Line 412: / Line 1082: @@
 ...
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+...
+#          livenessProbe:
+#            httpGet:
+#              path: /
+#              port: http
+#          readinessProbe:
+#            httpGet:
+#              path: /
+#              port: http
 ...
 </code><code>
-!!! Был замечен "глюк" DNS, из-за которого не загружался Docker образ, "лечился" предварительным созданием namespace
+$ helm template my-webd webd-chart/ | less
+$ helm install my-webd webd-chart/ -n my-ns --create-namespace --wait
-$ helm install my-webd webd-chart/ --n my-ns --create-namespace --wait
+$ kubectl describe events -n my-ns | less
 $ export HELM_NAMESPACE=my-ns
@@ Line 422: / Line 1103: @@
 $ helm list
-$ helm upgrade my-webd webd-chart/ --set=image.tag=ver1.10
+$ ### helm upgrade my-webd webd-chart/ --set=image.tag=ver1.10
 $ helm history my-webd
@@ Line 436: / Line 1117: @@
   * [[https://github.com/chartmuseum/helm-push/#readme|helm cm-push plugin]]
   * [[https://medium.com/containerum/how-to-make-and-share-your-own-helm-package-50ae40f6c221|How to make and share your own Helm package]]
+  * [[https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html|Gitlab Personal access tokens]]
+  * [[Инструмент GitLab#Подключение через API]] - Role: Mainteiner, api, read_registry, write_registry
 <code>
-$ helm repo add --username student --password NNNNNN-NNNNNNNNNNNNN webd http://192.168.13.1/api/v4/projects/6/packages/helm/stable
+gitlab-runner@server:~/gowebd-k8s$ helm repo add --username student --password NNNNN-NNNNNNNNNNNNNNNNNNN webd http://server.corpX.un/api/v4/projects/N/packages/helm/stable
+"webd" has been added to your repositories
-$ helm repo list
+gitlab-runner@server:~/gowebd-k8s$ ### helm repo remove webd
-$ helm package webd-chart
+gitlab-runner@server:~/gowebd-k8s$ helm repo list
-$ ls *tgz
-$ helm plugin install https://github.com/chartmuseum/helm-push
+gitlab-runner@server:~/gowebd-k8s$ helm package webd-chart
-$ helm cm-push webd-chart-0.1.0.tgz webd
-$ helm search repo webd
+gitlab-runner@server:~/gowebd-k8s$ tar -tf webd-chart-0.1.1.tgz
+gitlab-runner@server:~/gowebd-k8s$ helm plugin install https://github.com/chartmuseum/helm-push
+gitlab-runner@server:~/gowebd-k8s$ helm cm-push webd-chart-0.1.1.tgz webd
+gitlab-runner@server:~/gowebd-k8s$ rm webd-chart-0.1.1.tgz
+kube1:~# helm repo add webd http://server.corpX.un/api/v4/projects/N/packages/helm/stable
+kube1:~# helm repo update
+kube1:~# helm search repo webd
+kube1:~# helm repo update webd
+kube1:~# helm install my-webd webd/webd-chart
+kube1:~# ###helm uninstall my-webd
 </code>
 ==== Работа с публичными репозиториями ====
 <code>
+helm repo add gitlab https://charts.gitlab.io
+helm search repo -l gitlab/gitlab-runner
+helm show values gitlab/gitlab-runner | tee values.yaml
+gitlab-runner@server:~$ diff values.yaml
+</code><code>
+...
+gitlabUrl: http://server.corpX.un/
+...
+runnerRegistrationToken: "NNNNNNNNNNNNNNNNNNNNNNNN"
+...
+,149c142
+<   create: true
+---
+>   create: false
+d317
+<         privileged = true
+c424
+<   allowPrivilegeEscalation: true
+---
+>   allowPrivilegeEscalation: false
+c427
+<   privileged: true
+---
+>   privileged: false
+</code><code>
+gitlab-runner@server:~$ helm upgrade -i gitlab-runner gitlab/gitlab-runner -f values.yaml -n gitlab-runner --create-namespace
+gitlab-runner@server:~$ kubectl get all -n gitlab-runner
+</code><code>
 $ helm search hub -o json wordpress | jq '.' | less
@@ Line 464: / Line 1193: @@
 ===== Дополнительные материалы =====
+==== bare-metal minikube ====
+<code>
+student@node2:~$ sudo apt install conntrack
+https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
+...
+wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz
+...
+student@node2:~$ minikube start --driver=none --insecure-registry "server.corpX.un:5000"
+</code>
+==== minikube dashboard ====
+<code>
+student@node1:~$ minikube dashboard &
+...
+Opening http://127.0.0.1:NNNNN/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser
+...
+/home/mobaxterm> ssh -L NNNNN:localhost:NNNNN student@192.168.X.10
+Теперь, та же ссылка работает на win host системе
+</code>
+==== Подключение к minikube с другой системы ====
+  * Если не minikube, то достаточно только копию .kube/config
+  * [[https://habr.com/ru/company/flant/blog/345580/|см. Настройка GitLab Runner]]
+<code>
+student@node1:~$ tar -cvzf kube-config.tar.gz .kube/config .minikube/ca.crt .minikube/profiles/minikube
+gitlab-runner@server:~$ scp student@node1:kube-config.tar.gz .
+gitlab-runner@server:~$ tar -xvf kube-config.tar.gz
+gitlab-runner@server:~$ cat .kube/config
+</code><code>
+...
+    certificate-authority: /home/gitlab-runner/.minikube/ca.crt
+...
+    client-certificate: /home/gitlab-runner/.minikube/profiles/minikube/client.crt
+    client-key: /home/gitlab-runner/.minikube/profiles/minikube/client.key
+...
+</code>
 ==== kompose ====
+  * [[https://stackoverflow.com/questions/47536536/whats-the-difference-between-docker-compose-and-kubernetes|What's the difference between Docker Compose and Kubernetes?]]
   * [[https://loft.sh/blog/docker-compose-to-kubernetes-step-by-step-migration/|Docker Compose to Kubernetes: Step-by-Step Migration]]
   * [[https://kubernetes.io/docs/tasks/configure-pod-container/translate-compose-kubernetes/|Translate a Docker Compose File to Kubernetes Resources]]
 <code>
-root@gate.corp13.un:~# curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-linux-amd64 -o kompose
+root@gate:~# curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-linux-amd64 -o kompose
-root@gate.corp13.un:~# chmod +x kompose
+root@gate:~# chmod +x kompose
-root@gate.corp13.un:~# sudo mv ./kompose /usr/local/bin/kompose
+root@gate:~# sudo mv ./kompose /usr/local/bin/kompose
 </code>

Методические материалы Лохтурова Вячеслава

User Tools

Site Tools

Differences

Page Tools