User Tools

Site Tools


система_kubernetes

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
система_kubernetes [2022/09/27 14:43]
val [namespace для своего приложения]
система_kubernetes [2024/03/26 13:20] (current)
val [Volumes]
Line 1: Line 1:
 ====== Система Kubernetes ====== ====== Система Kubernetes ======
 +
 +  * [[https://​kubernetes.io/​ru/​docs/​home/​|Документация по Kubernetes (на русском)]]
  
   * [[https://​youtu.be/​sLQefhPfwWE|youtube Введение в Kubernetes на примере Minikube]]   * [[https://​youtu.be/​sLQefhPfwWE|youtube Введение в Kubernetes на примере Minikube]]
Line 7: Line 9:
   * [[https://​habr.com/​ru/​company/​domclick/​blog/​577964/​|Ультимативный гайд по созданию CI/CD в GitLab с автодеплоем в Kubernetes на голом железе всего за 514$ в год ( ͡° ͜ʖ ͡°)]]   * [[https://​habr.com/​ru/​company/​domclick/​blog/​577964/​|Ультимативный гайд по созданию CI/CD в GitLab с автодеплоем в Kubernetes на голом железе всего за 514$ в год ( ͡° ͜ʖ ͡°)]]
   * [[https://​habr.com/​ru/​company/​flant/​blog/​513908/​|Полноценный Kubernetes с нуля на Raspberry Pi]]   * [[https://​habr.com/​ru/​company/​flant/​blog/​513908/​|Полноценный Kubernetes с нуля на Raspberry Pi]]
 +  * [[https://​habr.com/​ru/​companies/​domclick/​articles/​566224/​|Различия между Docker, containerd, CRI-O и runc]]
  
   * [[https://​habr.com/​ru/​company/​vk/​blog/​542730/​|11 факапов PRO-уровня при внедрении Kubernetes и как их избежать]]   * [[https://​habr.com/​ru/​company/​vk/​blog/​542730/​|11 факапов PRO-уровня при внедрении Kubernetes и как их избежать]]
  
   * [[https://​github.com/​dgkanatsios/​CKAD-exercises|A set of exercises that helped me prepare for the Certified Kubernetes Application Developer exam]]   * [[https://​github.com/​dgkanatsios/​CKAD-exercises|A set of exercises that helped me prepare for the Certified Kubernetes Application Developer exam]]
 +
 +  * [[https://​www.youtube.com/​watch?​v=XZQ7-7vej6w|Наш опыт с Kubernetes в небольших проектах / Дмитрий Столяров (Флант)]]
 +
 +  * [[https://​alesnosek.com/​blog/​2017/​02/​14/​accessing-kubernetes-pods-from-outside-of-the-cluster/​|Accessing Kubernetes Pods From Outside of the Cluster]]
  
 ===== Инструмент командной строки kubectl ===== ===== Инструмент командной строки kubectl =====
Line 17: Line 24:
  
 ==== Установка ==== ==== Установка ====
 +
 +=== Linux ===
 <​code>​ <​code>​
 # curl -LO https://​storage.googleapis.com/​kubernetes-release/​release/​`curl -s https://​storage.googleapis.com/​kubernetes-release/​release/​stable.txt`/​bin/​linux/​amd64/​kubectl # curl -LO https://​storage.googleapis.com/​kubernetes-release/​release/​`curl -s https://​storage.googleapis.com/​kubernetes-release/​release/​stable.txt`/​bin/​linux/​amd64/​kubectl
Line 25: Line 34:
 </​code>​ </​code>​
  
-==== Подключение к новому кластеру ====+=== Windows ​===
  
-  * [[https://medium.com/@jacobtomlinson/how-to-merge-kubernetes-kubectl-config-files-737b61bd517d|How to merge Kubernetes ​kubectl ​config files]]+  * [[https://kubernetes.io/docs/tasks/​tools/​install-kubectl-windows/|Install and Set Up kubectl ​on Windows]]
  
 <​code>​ <​code>​
-gitlab-runner@server:​~$ scp root@node1:.kube/config ​.kube/config_new+cmder$ curl -LO "https://dl.k8s.io/release/v1.29.0/bin/​windows/​amd64/​kubectl.exe"​
  
-gitlab-runner@server:​~cp ~/.kube/config ~/.kube/config.bak+cmdermv kubectl.exe /usr/bin 
 +</code>
  
-gitlab-runner@server:​~$ KUBECONFIG=~/​.kube/​config:​~/​.kube/​config_new kubectl config view --flatten > /tmp/config+==== Подключение к кластеру ====
  
-gitlab-runner@server:​~$ mv /​tmp/​config ​~/.kube/config+<​code>​ 
 +mkdir ~/.kube/
  
-gitlab-runner@server:~$ chmod 600 /​home/​gitlab-runner/.kube/config ​   #Helm WARNING+scp root@192.168.X.2N1:.kube/​config ​~/.kube/
  
-gitlab-runner@server:​~$ kubectl ​config get-contexts+cat ~/​.kube/​config 
 +</​code><​code>​ 
 +... 
 +    server: https://​192.168.X.2N1:​6443 
 +... 
 +</​code><​code>​ 
 +kubectl get all -o wide --all-namespaces 
 +kubectl get all -o wide -A 
 +</​code>​ 
 +=== Настройка автодополнения === 
 +<​code>​ 
 +gitlab-runner@server:​~$ ​source <(kubectl ​completion bash) 
 +</​code>​
  
-gitlab-runner@server:​~$ rm ~/​.kube/​config.bak+=== Подключение к другому кластеру ===
  
-gitlab-runner@server:​~$ ​kubectl ​config ​use-context kubernetes-admin@kubernetes+<​code>​ 
 +gitlab-runner@server:​~$ ​scp root@kube1:​.kube/​config ​.kube/​config_kube1 
 + 
 +gitlab-runner@server:​~$ cat .kube/​config_kube1 
 +</​code><​code>​ 
 +... 
 +    .kube/​config_kube1 
 +... 
 +</​code><​code>​ 
 +gitlab-runner@server:~$ export KUBECONFIG=~/​.kube/​config_kube1
  
-gitlab-runner@server:​~$ kubectl get all -o wide --all-namespaces+gitlab-runner@server:​~$ kubectl get nodes
 </​code>​ </​code>​
  
Line 71: Line 103:
  
 gitlab-runner@server:​~$ time minikube start --driver=docker --insecure-registry "​server.corpX.un:​5000"​ gitlab-runner@server:​~$ time minikube start --driver=docker --insecure-registry "​server.corpX.un:​5000"​
-real    ​5m8.320s+real    ​29m8.320s
 ... ...
  
Line 77: Line 109:
  
 gitlab-runner@server:​~$ minikube ip gitlab-runner@server:​~$ minikube ip
- 
-gitlab-runner@server:​~$ minikube kubectl -- get pods -A 
  
 gitlab-runner@server:​~$ minikube addons list gitlab-runner@server:​~$ minikube addons list
  
-gitlab-runner@server:​~$ minikube addons configure registry-creds+gitlab-runner@server:​~$ minikube addons configure registry-creds ​  #Не нужно для registry попубличных проектов
 ... ...
 Do you want to enable Docker Registry? [y/n]: y Do you want to enable Docker Registry? [y/n]: y
Line 91: Line 121:
  
 gitlab-runner@server:​~$ minikube addons enable registry-creds gitlab-runner@server:​~$ minikube addons enable registry-creds
 +
 +gitlab-runner@server:​~$ minikube kubectl -- get pods -A
 +
 +gitlab-runner@server:​~$ alias kubectl='​minikube kubectl --'
 +
 +gitlab-runner@server:​~$ kubectl get pods -A
 </​code>​ </​code>​
 +
 +или
  
   * [[#​Инструмент командной строки kubectl]]   * [[#​Инструмент командной строки kubectl]]
 +
 +<​code>​
 +gitlab-runner@server:​~$ ###minikube stop
 +
 +gitlab-runner@server:​~$ ###minikube start
 +</​code>​
 ===== Кластер Kubernetes ===== ===== Кластер Kubernetes =====
  
-==== Развертывание ==== 
  
 +==== Развертывание через kubeadm ====
 +
 +  * [[https://​kubernetes.io/​docs/​setup/​production-environment/​tools/​kubeadm/​create-cluster-kubeadm/​|kubernetes.io Creating a cluster with kubeadm]]
   * [[https://​infoit.com.ua/​linux/​kak-ustanovit-kubernetes-na-ubuntu-20-04-lts/​|Как установить Kubernetes на Ubuntu 20.04 LTS]]   * [[https://​infoit.com.ua/​linux/​kak-ustanovit-kubernetes-na-ubuntu-20-04-lts/​|Как установить Kubernetes на Ubuntu 20.04 LTS]]
 +  * [[https://​www.linuxtechi.com/​install-kubernetes-on-ubuntu-22-04/​|How to Install Kubernetes Cluster on Ubuntu 22.04]]
 +  * [[https://​www.linuxtechi.com/​install-kubernetes-cluster-on-debian/​|https://​www.linuxtechi.com/​install-kubernetes-cluster-on-debian/​]]
   * [[https://​www.cloud4y.ru/​blog/​installation-kubernetes/​|Установка Kubernetes]]   * [[https://​www.cloud4y.ru/​blog/​installation-kubernetes/​|Установка Kubernetes]]
  
-=== Установка ​ПО и подготовка узлов ===+=== Подготовка узлов ===
  
 <​code>​ <​code>​
Line 109: Line 157:
 node1# ssh-copy-id node3 node1# ssh-copy-id node3
  
 +node1# bash -c '
 +swapoff -a
 +ssh node2 swapoff -a
 +ssh node3 swapoff -a
 +'
 +
 +node1# bash -c '
 +sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab
 +ssh node2 sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab
 +ssh node3 sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab
 +'
 +</​code>​
 +
 +=== Установка ПО ===
 +<​code>​
 node1# bash -c ' node1# bash -c '
 http_proxy=http://​proxy.isp.un:​3128/​ apt -y install apt-transport-https curl http_proxy=http://​proxy.isp.un:​3128/​ apt -y install apt-transport-https curl
Line 129: Line 192:
 node1# bash -c ' node1# bash -c '
 http_proxy=http://​proxy.isp.un:​3128/​ apt -y install kubeadm kubelet kubectl kubernetes-cni http_proxy=http://​proxy.isp.un:​3128/​ apt -y install kubeadm kubelet kubectl kubernetes-cni
-ssh node2 apt -y install kubeadm kubelet kubectl kubernetes-cni +ssh node2 http_proxy=http://​proxy.isp.un:​3128/ ​apt -y install kubeadm kubelet kubectl kubernetes-cni 
-ssh node3 apt -y install kubeadm kubelet kubectl kubernetes-cni+ssh node3 http_proxy=http://​proxy.isp.un:​3128/ ​apt -y install kubeadm kubelet kubectl kubernetes-cni
 ' '
  
-node1# bash -c ' +https://​forum.linuxfoundation.org/​discussion/​864693/​the-repository-http-apt-kubernetes-io-kubernetes-xenial-release-does-not-have-a-release-file 
-swapoff ​-+!!!! Внимание на каждом узле нужно сделать:​ !!!! 
-ssh node2 swapoff ​-a + 
-ssh node3 swapoff ​-a +удалить из /​etc/​apt/​sources.list строчку с kubernetes 
-'+ 
 +mkdir /​etc/​apt/​keyrings 
 + 
 +curl -fsSL https://​pkgs.k8s.io/​core:/​stable:/​v1.28/​deb/​Release.key | sudo gpg --dearmor -o /​etc/​apt/​keyrings/​kubernetes-apt-keyring.gpg 
 + 
 +echo 'deb [signed-by=/​etc/​apt/​keyrings/​kubernetes-apt-keyring.gpg] https://​pkgs.k8s.io/​core:/​stable:/​v1.28/​deb/​ /' | sudo tee /​etc/​apt/​sources.list.d/​kubernetes.list 
 + 
 +apt update 
 + 
 +apt install -y kubeadm=1.28.1-1.1 kubelet=1.28.1-1.1 kubectl=1.28.1-1.1
  
-node1# bash -c ' 
-sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab 
-ssh node2 sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab 
-ssh node3 sed -i""​ -e "/​swap/​s/​^/#/"​ /etc/fstab 
-' 
 </​code>​ </​code>​
  
 === Инициализация master === === Инициализация master ===
  
-  * Может понадобиться в случае возникновения ошибки [[https://​github.com/​containerd/​containerd/​issues/​4581|[ERROR CRI]: container runtime is not running]] 
-<​code>​ 
-node1# bash -c ' 
-rm /​etc/​containerd/​config.toml 
-systemctl restart containerd 
-ssh node2 rm /​etc/​containerd/​config.toml 
-ssh node2 systemctl restart containerd 
-ssh node3 rm /​etc/​containerd/​config.toml 
-ssh node3 systemctl restart containerd 
-' 
-</​code>​ 
 <​code>​ <​code>​
 root@node1:​~#​ kubeadm init --pod-network-cidr=10.244.0.0/​16 --apiserver-advertise-address=192.168.X.201 root@node1:​~#​ kubeadm init --pod-network-cidr=10.244.0.0/​16 --apiserver-advertise-address=192.168.X.201
Line 171: Line 227:
  
 root@node1:​~#​ kubectl get --raw='/​readyz?​verbose'​ root@node1:​~#​ kubectl get --raw='/​readyz?​verbose'​
 +</​code>​
 +  * Может понадобиться в случае возникновения ошибки [[https://​github.com/​containerd/​containerd/​issues/​4581|[ERROR CRI]: container runtime is not running]]
 +<​code>​
 +node1# bash -c '
 +rm /​etc/​containerd/​config.toml
 +systemctl restart containerd
 +ssh node2 rm /​etc/​containerd/​config.toml
 +ssh node2 systemctl restart containerd
 +ssh node3 rm /​etc/​containerd/​config.toml
 +ssh node3 systemctl restart containerd
 +'
 </​code>​ </​code>​
  
Line 190: Line 257:
 </​code>​ </​code>​
  
-==== Настройка доступа к Insecure Private Registry ​====+=== Удаление узла === 
 +<​code>​ 
 +$ kubectl cordon kube3 
 + 
 +$ kubectl drain kube3 --ignore-daemonsets --delete-emptydir-data 
 + 
 +$ kubectl delete node kube3 
 +</​code>​ 
 + 
 +=== Удаление кластера === 
 + 
 +  * [[https://​stackoverflow.com/​questions/​44698283/​how-to-completely-uninstall-kubernetes|How to completely uninstall kubernetes]] 
 + 
 +<​code>​ 
 +node1# bash -c ' 
 +kubeadm reset 
 +ssh node2 kubeadm reset 
 +ssh node3 kubeadm reset 
 +
 +</​code>​ 
 + 
 +=== Настройка доступа к Insecure Private Registry ===
  
   * [[https://​github.com/​containerd/​containerd/​issues/​4938|Unable to pull image from insecure registry, http: server gave HTTP response to HTTPS client #4938]]   * [[https://​github.com/​containerd/​containerd/​issues/​4938|Unable to pull image from insecure registry, http: server gave HTTP response to HTTPS client #4938]]
Line 198: Line 286:
   * [[https://​github.com/​containerd/​containerd/​blob/​main/​docs/​PLUGINS.md#​version-header|containerd/​docs/​PLUGINS.md migrate config v1 to v2]]   * [[https://​github.com/​containerd/​containerd/​blob/​main/​docs/​PLUGINS.md#​version-header|containerd/​docs/​PLUGINS.md migrate config v1 to v2]]
  
-  * Docker [[Технология Docker#​Insecure Private Registry]]+== сontainerd ==
  
 <​code>​ <​code>​
-node1# scp server:/​etc/​docker/​daemon.json /​etc/​docker/​daemon.json +root@node1:~mkdir /etc/containerd/
- +
-serverbash -c ' +
-scp /etc/docker/​daemon.json node2:/​etc/​docker/​daemon.json +
-scp /​etc/​docker/​daemon.json node3:/​etc/​docker/​daemon.json +
-service docker restart +
-ssh node2 service docker restart +
-ssh node3 service docker restart +
-+
- +
-# don't work in cri-tools 1.25, need public project +
-### node1# docker login http://​server.corpX.un:​5000 +
- +
-### node1# bash -c ' +
-ssh node2 mkdir -p .docker +
-ssh node3 mkdir -p .docker +
-scp ~/​.docker/​config.json node2:​.docker/​config.json +
-scp ~/​.docker/​config.json node3:​.docker/config.json +
-'+
  
 root@node1:​~#​ cat /​etc/​containerd/​config.toml root@node1:​~#​ cat /​etc/​containerd/​config.toml
Line 229: Line 299:
     [plugins."​io.containerd.grpc.v1.cri"​.registry.mirrors."​server.corpX.un:​5000"​]     [plugins."​io.containerd.grpc.v1.cri"​.registry.mirrors."​server.corpX.un:​5000"​]
       endpoint = ["​http://​server.corpX.un:​5000"​]       endpoint = ["​http://​server.corpX.un:​5000"​]
-  ​[plugins."​io.containerd.grpc.v1.cri"​.registry.configs] + 
-    [plugins."​io.containerd.grpc.v1.cri"​.registry.configs."​server.corpX.un:​5000"​.tls] +# no need 
-      insecure_skip_verify = true+#  ​[plugins."​io.containerd.grpc.v1.cri"​.registry.configs] 
 +   ​[plugins."​io.containerd.grpc.v1.cri"​.registry.configs."​server.corpX.un:​5000"​.tls] 
 +     ​insecure_skip_verify = true 
 # don't work in cri-tools 1.25, need public project # don't work in cri-tools 1.25, need public project
 #​[plugins."​io.containerd.grpc.v1.cri"​.registry.configs."​server.corpX.un:​5000"​.auth] #​[plugins."​io.containerd.grpc.v1.cri"​.registry.configs."​server.corpX.un:​5000"​.auth]
Line 237: Line 310:
 </​code><​code>​ </​code><​code>​
 node1# bash -c ' node1# bash -c '
 +ssh node2 mkdir /​etc/​containerd/​
 +ssh node3 mkdir /​etc/​containerd/​
 scp /​etc/​containerd/​config.toml node2:/​etc/​containerd/​config.toml scp /​etc/​containerd/​config.toml node2:/​etc/​containerd/​config.toml
 scp /​etc/​containerd/​config.toml node3:/​etc/​containerd/​config.toml scp /​etc/​containerd/​config.toml node3:/​etc/​containerd/​config.toml
Line 250: Line 325:
  
 <​code>​ <​code>​
-root@nodeN:​~#​ crictl -r unix:///​run/​containerd/​containerd.sock pull server.corpX.un:​5000/​student/​webd+root@nodeN:​~#​ crictl -r unix:///​run/​containerd/​containerd.sock pull server.corpX.un:​5000/​student/​gowebd 
 +</​code>​ 
 + 
 +==== Развертывание через Kubespray ==== 
 + 
 +  * [[https://​github.com/​kubernetes-sigs/​kubespray]] 
 +  * [[https://​habr.com/​ru/​companies/​domclick/​articles/​682364/​|Самое подробное руководство по установке высокодоступного (почти ಠ ͜ʖ ಠ ) Kubernetes-кластера]] 
 +  * [[https://​habr.com/​ru/​companies/​X5Tech/​articles/​645651/​|Bare-metal kubernetes-кластер на своём локальном компьютере]] 
 +  * [[https://​internet-lab.ru/​k8s_kubespray|Kubernetes — установка через Kubespray]] 
 +  * [[https://​www.mshowto.org/​en/​ubuntu-sunucusuna-kubespray-ile-kubernetes-kurulumu.html|Installing Kubernetes on Ubuntu Server with Kubespray]] 
 +  * [[https://​kubernetes.io/​docs/​setup/​production-environment/​tools/​kubespray/​|Installing Kubernetes with Kubespray]] 
 + 
 +  * [[https://​stackoverflow.com/​questions/​29882263/​browse-list-of-tagged-releases-in-a-repo]] 
 + 
 +<​code>​ 
 +kube1# ssh-keygen 
 + 
 +kube1# ssh-copy-id kube1;​ssh-copy-id kube2;​ssh-copy-id kube3;​ssh-copy-id kube4; 
 + 
 +kube1# apt update 
 + 
 +kube1# apt install python3-pip -y 
 + 
 +kube1# git clone https://​github.com/​kubernetes-sigs/​kubespray 
 + 
 +kube1# cd kubespray/​ 
 + 
 +~/​kubespray#​ grep -r containerd_insecure_registries . 
 +~/​kubespray#​ git log 
 + 
 +~/​kubespray#​ git branch -r 
 +~/​kubespray#​ ### git checkout origin/​release-2.22 ​  # debian11 - 2.24 
 + 
 +~/​kubespray#​ git tag -l 
 +~/​kubespray#​ ### git checkout tags/​v2.22.1 
 + 
 +~/​kubespray#​ git checkout 4c37399c7582ea2bfb5202c3dde3223f9c43bf59 
 + 
 +~/​kubespray#​ ### git checkout master 
 +</​code>​ 
 + 
 +  * Может потребоваться [[Язык программирования Python#​Виртуальная среда Python]] 
 +  * Может потребоваться [[https://​github.com/​kubernetes-sigs/​kubespray/​issues/​10688|"​The conditional check '​groups.get('​kube_control_plane'​)'​ failed. The error was: Conditional is marked as unsafe, and cannot be evaluated."​ #10688]] 
 + 
 +<​code>​ 
 +~/​kubespray#​ time pip3 install -r requirements.txt 
 +real    1m48.202s 
 + 
 +~/​kubespray#​ cp -rfp inventory/​sample inventory/​mycluster 
 + 
 +~/​kubespray#​ declare -a IPS=(kube1,​192.168.X.221 kube2,​192.168.X.222 kube3,​192.168.X.223) 
 + 
 +~/​kubespray#​ CONFIG_FILE=inventory/​mycluster/​hosts.yaml python3 contrib/​inventory_builder/​inventory.py ${IPS[@]} 
 + 
 +~/​kubespray#​ less inventory/​mycluster/​hosts.yaml 
 + 
 +~/​kubespray#​ time ansible-playbook -i inventory/​mycluster/​hosts.yaml cluster.yml 
 +real    45m31.796s 
 + 
 +kube1# less ~/​.kube/​config 
 + 
 +~/​kubespray#​ ###time ansible-playbook -i inventory/​mycluster/​hosts.yaml reset.yml 
 +real    7m31.796s 
 +</​code>​ 
 + 
 +=== Добавление узла через Kubespray === 
 +<​code>​ 
 +~/​kubespray#​ cat inventory/​mycluster/​hosts.yaml 
 +</​code><​code>​ 
 +... 
 +    node4: 
 +      ansible_host:​ 192.168.X.204 
 +      ip: 192.168.X.204 
 +      access_ip: 192.168.X.204 
 +... 
 +    kube_node:​ 
 +... 
 +        node4: 
 +... 
 +</​code><​code>​ 
 + 
 +~/​kubespray#​ time ansible-playbook -i inventory/​mycluster/​hosts.yaml --limit=kube4 scale.yml 
 +real    17m37.459s 
 + 
 +$ kubectl get nodes -o wide 
 +</​code>​ 
 + 
 +=== Добавление insecure_registries через Kubespray === 
 +<​code>​ 
 +~/​kubespray#​ cat inventory/​mycluster/​group_vars/​all/​containerd.yml 
 +</​code><​code>​ 
 +... 
 +containerd_insecure_registries:​ 
 +  "​server.corpX.un:​5000":​ "​http://​server.corpX.un:​5000"​ 
 +containerd_registry_auth:​ 
 +  - registry: server.corpX.un:​5000 
 +    username: student 
 +    password: Pa$$w0rd 
 +... 
 +</​code><​code>​ 
 +~/​kubespray#​ time ansible-playbook -i inventory/​mycluster/​hosts.yaml cluster.yml 
 +user    46m37.151s 
 + 
 +# less /​etc/​containerd/​config.toml 
 +</​code>​ 
 + 
 +=== Управление дополнениями через Kubespray === 
 +<​code>​ 
 +~/​kubespray#​ cat inventory/​mycluster/​group_vars/​k8s_cluster/​addons.yml 
 +</​code><​code>​ 
 +... 
 +helm_enabled:​ true 
 +... 
 +ingress_nginx_enabled:​ true 
 +ingress_nginx_host_network:​ true 
 +...
 </​code>​ </​code>​
 ===== Базовые объекты k8s ===== ===== Базовые объекты k8s =====
Line 262: Line 452:
  
 <​code>​ <​code>​
-$ kubectl ​create deployment ​my-debian --image=debian -- "​sleep"​ "​3600"​+$ kubectl ​api-resources 
 + 
 +$ kubectl run my-debian --image=debian -- "​sleep"​ "​3600"​
  
 $ kubectl get all $ kubectl get all
  
-$ kubectl get deployments+kubeN# crictl ps | grep debi 
 +kubeN# crictl images 
 +nodeN# ctr ns ls 
 +nodeN# ctr -n=k8s.io image ls | grep debi
  
-$ kubectl ​get pods -o wide+$ kubectl ​delete pod my-debian
  
 +$ kubectl create deployment my-debian --image=debian -- "​sleep"​ "​3600"​
 +
 +$ kubectl get deployments
 +</​code>​
 +  * [[#​Настройка автодополнения]]
 +<​code>​
 $ kubectl attach my-debian-NNNNNNNNN-NNNNN $ kubectl attach my-debian-NNNNNNNNN-NNNNN
  
Line 276: Line 477:
  
 $ kubectl get deployment my-debian -o yaml $ kubectl get deployment my-debian -o yaml
 +</​code>​ 
 +  * [[Переменные окружения]] EDITOR 
 +<​code>​
 $ kubectl edit deployment my-debian $ kubectl edit deployment my-debian
 +
 +$ kubectl get pods -o wide
  
 $ kubectl delete deployment my-debian $ kubectl delete deployment my-debian
Line 286: Line 491:
 </​code><​code>​ </​code><​code>​
 apiVersion: apps/v1 apiVersion: apps/v1
-kind: Deployment+kind: ReplicaSet 
 +#kind: Deployment
 metadata: metadata:
   name: my-debian   name: my-debian
Line 293: Line 499:
     matchLabels:​     matchLabels:​
       app: my-debian       app: my-debian
 +  replicas: 2
   template:   template:
     metadata:     metadata:
Line 305: Line 512:
       restartPolicy:​ Always       restartPolicy:​ Always
 </​code><​code>​ </​code><​code>​
-$ kubectl ​create ​-f my-debian-deployment.yaml+$ kubectl ​apply -f my-debian-deployment.yaml
 ... ...
 $ kubectl delete -f my-debian-deployment.yaml $ kubectl delete -f my-debian-deployment.yaml
Line 312: Line 519:
  
   * [[https://​matthewpalmer.net/​kubernetes-app-developer/​articles/​kubernetes-volumes-example-nfs-persistent-volume.html|How to use an NFS volume]]   * [[https://​matthewpalmer.net/​kubernetes-app-developer/​articles/​kubernetes-volumes-example-nfs-persistent-volume.html|How to use an NFS volume]]
 +  * [[https://​hub.docker.com/​_/​httpd|The Apache HTTP Server Project - httpd Docker Official Image]]
  
 <​code>​ <​code>​
Line 330: Line 538:
 metadata: metadata:
   name: my-webd   name: my-webd
-  namespace: my-ns 
 spec: spec:
   selector:   selector:
Line 346: Line 553:
 #        image: server.corpX.un:​5000/​student/​webd #        image: server.corpX.un:​5000/​student/​webd
 #        image: server.corpX.un:​5000/​student/​webd:​ver1.N #        image: server.corpX.un:​5000/​student/​webd:​ver1.N
 +
 +#        imagePullPolicy:​ "​Always"​
 +
 +#        image: httpd
 +#        lifecycle:
 +#          postStart:
 +#            exec:
 +#              command: ["/​bin/​sh",​ "​-c",​ "echo Hello from apache2 on $(hostname) > /​usr/​local/​apache2/​htdocs/​index.html"​]
 +
 +#        env:
 +#        - name: APWEBD_HOSTNAME
 +#          value: "​apwebd.corpX.un"​
 +#        - name: KEYCLOAK_HOSTNAME
 +#          value: "​keycloak.corpX.un"​
 +#        - name: REALM_NAME
 +#          value: "​corpX"​
  
 #        livenessProbe:​ #        livenessProbe:​
Line 360: Line 583:
 #          path: /var/www #          path: /var/www
 </​code><​code>​ </​code><​code>​
-$ kubectl apply -f my-webd-deployment.yaml+$ kubectl apply -f my-webd-deployment.yaml ​-n my-ns
  
 $ kubectl get all -n my-ns -o wide  $ kubectl get all -n my-ns -o wide 
Line 388: Line 611:
 metadata: metadata:
   name: my-webd   name: my-webd
-  namespace: my-ns 
 spec: spec:
-  ​type: NodePort+#  ​type: NodePort 
 +#  type: LoadBalancer 
 +#  loadBalancerIP:​ 192.168.X.64
   selector:   selector:
     app: my-webd     app: my-webd
Line 398: Line 622:
 #    nodePort: 30111 #    nodePort: 30111
 </​code><​code>​ </​code><​code>​
-$ kubectl apply -f my-webd-service.yaml+$ kubectl apply -f my-webd-service.yaml ​-n my-ns
  
 +$ kubectl logs -l app=my-webd -n my-ns 
 +(доступны опции -f, --tail=2000,​ --previous)
 +</​code>​
 +=== NodePort ===
 +<​code>​
 $ kubectl get svc my-webd -n my-ns $ kubectl get svc my-webd -n my-ns
 NAME              TYPE       ​CLUSTER-IP ​      ​EXTERNAL-IP ​  ​PORT(S) ​       AGE NAME              TYPE       ​CLUSTER-IP ​      ​EXTERNAL-IP ​  ​PORT(S) ​       AGE
Line 407: Line 636:
  
 $ curl http://​node1,​2,​3:​NNNNN $ curl http://​node1,​2,​3:​NNNNN
- +на "​самодельном kubeadm"​ кластере работает не стабильно 
 +</​code>​ 
 +== NodePort Minikube == 
 +<​code>​
 $ minikube service list $ minikube service list
  
Line 415: Line 646:
  
 $ curl $(minikube service my-webd -n my-ns --url) $ curl $(minikube service my-webd -n my-ns --url)
 +</​code>​
  
 +=== LoadBalancer ===
  
-$ kubectl ​logs -l app=my-webd -n my-ns  +== MetalLB == 
-(доступны опции ​-f--tail=2000, ​--previous)+ 
 +  * [[https://​www.adaltas.com/​en/​2022/​09/​08/​kubernetes-metallb-nginx/​|Ingresses and Load Balancers in Kubernetes with MetalLB and nginx-ingress]] 
 + 
 +  * [[https://​metallb.universe.tf/​installation/​|Installation]] 
 +  * [[https://​metallb.universe.tf/​configuration/​_advanced_ipaddresspool_configuration/​|Advanced AddressPool configuration]] 
 +  * [[https://​metallb.universe.tf/​configuration/​_advanced_l2_configuration/​|Advanced L2 configuration]] 
 + 
 +<​code>​ 
 +$ kubectl ​apply -f https://​raw.githubusercontent.com/​metallb/​metallb/​v0.14.3/​config/​manifests/​metallb-native.yaml 
 + 
 +$ kubectl ​-n metallb-system get all 
 + 
 +$ cat first-pool.yaml 
 +</​code><​code>​ 
 +--- 
 +apiVersion: metallb.io/​v1beta1 
 +kind: IPAddressPool 
 +metadata: 
 +  name: first-pool 
 +  namespace: metallb-system 
 +spec: 
 +  addresses:​ 
 +  - 192.168.13.64/​28 
 +  autoAssign: false 
 +--- 
 +apiVersion: metallb.io/​v1beta1 
 +kind: L2Advertisement 
 +metadata: 
 +  name: first-pool-advertisement 
 +  namespace: metallb-system 
 +spec: 
 +  ipAddressPools:​ 
 +  - first-pool 
 +  interfaces:​ 
 +  - eth0 
 +</​code><​code>​ 
 +$ kubectl apply -f first-pool.yaml 
 + 
 +$ ### kubectl delete ​-f first-pool.yaml && rm first-pool.yaml 
 + 
 +$ ### kubectl apply -f https://​raw.githubusercontent.com/​metallb/​metallb/​v0.14.3/​config/​manifests/​metallb-native.yaml
 </​code>​ </​code>​
 +
 +=== ClusterIP ===
 +<​code>​
 +kube1# host my-webd.my-ns.svc.cluster.local 169.254.25.10
 +...10.102.135.146...
 +
 +server# ssh -p 32222 nodeN
 +
 +my-openssh-server-NNNNNNNN-NNNNN:​~#​ curl my-webd.my-ns.svc.cluster.local
 +  ИЛИ
 +my-openssh-server-NNNNNNNN-NNNNN:​~#​ curl my-webd-webd-chart.my-ns.svc.cluster.local
 +</​code>​
 +
 +== port-forward ==
 +
 +  * [[#​Инструмент командной строки kubectl]]
 +
 +<​code>​
 +node1/​kube1#​ kubectl port-forward -n my-ns --address 0.0.0.0 services/​my-webd 1234:80
 +
 +cmder> kubectl port-forward -n my-ns services/​my-webd 1234:80
 +</​code>​
 +
 +  * http://​192.168.X.2N1:​1234
 +  * http://​localhost:​1234
 +
 +<​code>​
 +node1/​kube1#​ kubectl -n my-ns delete pod/​my-webd...
 +</​code>​
 +
 +== kubectl proxy ==
 +
 +  * [[#​Инструмент командной строки kubectl]]
 +
 +<​code>​
 +kube1:~# kubectl proxy --address='​0.0.0.0'​ --accept-hosts='​^*$'​
 +
 +cmder> kubectl proxy
 +</​code>​
 +
 +  * http://​192.168.X.2N1:​8001/​api/​v1/​namespaces/​my-ns/​services/​my-webd:​80/​proxy/​
 +  * http://​localhost:​8001/​api/​v1/​namespaces/​my-ns/​services/​my-webd:​80/​proxy/​
 +
  
 ==== Удаление объектов ==== ==== Удаление объектов ====
 <​code>​ <​code>​
 +$ kubectl get all -n my-ns
 +
 $ kubectl delete -n my-ns -f my-webd-deployment.yaml,​my-webd-service.yaml $ kubectl delete -n my-ns -f my-webd-deployment.yaml,​my-webd-service.yaml
  
Line 431: Line 749:
  
 ==== Ingress ==== ==== Ingress ====
 +
 +  * [[https://​kubernetes.github.io/​ingress-nginx/​deploy/#​quick-start|NGINX ingress controller quick-start]]
 +
 +=== Minikube ingress-nginx-controller ===
  
   * [[https://​kubernetes.io/​docs/​tasks/​access-application-cluster/​ingress-minikube/​|Set up Ingress on Minikube with the NGINX Ingress Controller]]   * [[https://​kubernetes.io/​docs/​tasks/​access-application-cluster/​ingress-minikube/​|Set up Ingress on Minikube with the NGINX Ingress Controller]]
Line 436: Line 758:
  
 <​code>​ <​code>​
-server# ​host webd +server# ​cat /etc/bind/corpX.un 
-webd.corpX.un ​has address 192.168.49.+</​code><​code>​ 
-  или +... 
-webd.corpX.un has address ​192.168.X.201+webd 192.168.49.
 +</​code><​code>​ 
 +gitlab-runner@server:​~$ minikube addons enable ingress 
 +</​code>​
  
 +=== Baremetal ingress-nginx-controller ​ ===
  
-gitlab-runner@server:~$ minikube addons enable ​ingress+  * [[https://​github.com/​kubernetes/​ingress-nginx/​tags]] Версии 
 +  * [[https://​stackoverflow.com/​questions/​61616203/​nginx-ingress-controller-failed-calling-webhook|Nginx Ingress Controller - Failed Calling Webhook]] 
 +  * [[https://​stackoverflow.com/​questions/​51511547/​empty-address-kubernetes-ingress|Empty ADDRESS kubernetes ingress]] 
 +  * [[https://​kubernetes.github.io/​ingress-nginx/​deploy/#​bare-metal-clusters|ingress-nginx/​deploy/​bare-metal-clusters]] 
 + 
 +<​code>​ 
 +server# cat /​etc/​bind/​corpX.un 
 +</​code><​code>​ 
 +... 
 +webd            A       ​192.168.X.202 
 +                A       ​192.168.X.203 
 +gowebd ​         CNAME   ​webd 
 +</​code><​code>​ 
 +node1# curl https://​raw.githubusercontent.com/​kubernetes/​ingress-nginx/​controller-v1.3.1/​deploy/​static/​provider/​baremetal/​deploy.yaml | tee ingress-nginx.controller-v1.3.1.baremetal.yaml 
 + 
 +node1# cat ingress-nginx.controller-v1.3.1.baremetal.yaml 
 +</​code><​code>​ 
 +... 
 +kind: Deployment 
 +... 
 +spec: 
 +... 
 +  replicas: 3    ### insert this (equial count of worker nodes) 
 +  template: 
 +... 
 +      terminationGracePeriodSeconds:​ 300 
 +      hostNetwork:​ true                    ###insert this 
 +      volumes: 
 +... 
 +</​code><​code>​ 
 +node1# kubectl apply -f ingress-nginx.controller-v1.3.1.baremetal.yaml 
 + 
 +node1# kubectl get all -n ingress-nginx 
 + 
 +node1# ### kubectl delete -f ingress-nginx.controller-v1.3.1.baremetal.yaml
 </​code>​ </​code>​
-  * [[https://​kubernetes.github.io/​ingress-nginx/​deploy/#​quick-start|NGINX ingress ​controller ​quick-start]]+ 
 +=== Управление конфигурацией ​ingress-nginx-controller ​ ===
 <​code>​ <​code>​
-root@node1:~kubectl ​port-forward --namespace=ingress-nginx ​--address 0.0.0.0 service/​ingress-nginx-controller ​80:80+master-1:~kubectl ​exec -ingress-nginx ​pods/​ingress-nginx-controller-<​TAB>​ -- cat /​etc/​nginx/​nginx.conf | tee nginx.conf
  
 +master-1:~$ kubectl edit -n ingress-nginx configmaps ingress-nginx-controller
 +</​code><​code>​
 +...
 +data:
 +  use-forwarded-headers:​ "​true"​
 +...
 +</​code>​
  
-gitlab-runner@server:~/webd$ ### kubectl create ingress my-webd --class=nginx --rule="​webd.corpX.un/​*=my-webd:​80"​ -n my-ns+=== Итоговый вариант с DaemonSet === 
 +<​code>​ 
 +node1# diff ingress-nginx.controller-v1.8.2.baremetal.yaml.orig ingress-nginx.controller-v1.8.2.baremetal.yaml 
 +</​code><​code>​ 
 +323a324 
 +>   ​use-forwarded-headers"​true"​ 
 +391c392,​393 
 +< kind: Deployment 
 +--- 
 +> #kind: Deployment 
 +> kind: DaemonSet 
 +409,​412c411,​414 
 +<   ​strategy:​ 
 +<     ​rollingUpdate:​ 
 +<       ​maxUnavailable:​ 1 
 +<     type: RollingUpdate 
 +--- 
 +> #  strategy: 
 +> #    rollingUpdate:​ 
 +> #      maxUnavailable:​ 1 
 +> #    type: RollingUpdate 
 +501a504 
 +>       ​hostNetwork:​ true 
 +</code><​code>​ 
 +node1# kubectl -n ingress-nginx describe service/​ingress-nginx-controller 
 +... 
 +Endpoints: ​               192.168.X.221:​80,​192.168.X.222:​80,​192.168.X.223:​80 
 +... 
 +</​code>​ 
 + 
 +=== ingress example === 
 + 
 +<​code>​ 
 +node1# ​### kubectl create ingress my-ingress ​--class=nginx --rule="​webd.corpX.un/​*=my-webd:​80"​ -n my-ns
  
-gitlab-runner@server:​~/​webd$ ​cat my-webd-ingress.yaml+node1# ​cat my-ingress.yaml
 </​code><​code>​ </​code><​code>​
 apiVersion: networking.k8s.io/​v1 apiVersion: networking.k8s.io/​v1
 kind: Ingress kind: Ingress
 metadata: metadata:
-  name: my-webd +  name: my-ingress
-  namespace: my-ns+
 spec: spec:
   ingressClassName:​ nginx   ingressClassName:​ nginx
 +#  tls:
 +#  - hosts:
 +#    - gowebd.corpX.un
 +#    secretName: gowebd-tls
   rules:   rules:
   - host: webd.corpX.un   - host: webd.corpX.un
Line 471: Line 875:
         path: /         path: /
         pathType: Prefix         pathType: Prefix
-status+  - hostgowebd.corpX.un 
-  ​loadBalancer{}+    http: 
 +      paths: 
 +      - backend: 
 +          service: 
 +            name: my-gowebd 
 +            port: 
 +              number: 80 
 +        path: / 
 +        pathType: Prefix
 </​code><​code>​ </​code><​code>​
-kubectl apply -f my-webd-ingress.yaml+node1# ​kubectl apply -f my-ingress.yaml ​-n my-ns
  
  
-kubectl get ingress -n my-ns +node1# ​kubectl get ingress -n my-ns 
-NAME      CLASS   ​HOSTS ​           ADDRESS ​  ​PORTS   AGE +NAME      CLASS   ​HOSTS ​                            ​ADDRESS ​                        ​PORTS   AGE 
-my-webd ​  ​nginx ​  ​webd.corpX.un ​            ​80      ​11s+my-webd ​  ​nginx ​  ​webd.corpX.un,​gowebd.corpX.un ​  ​192.168.X.202,​192.168.X.203 ​  80      ​14m 
  
 $ curl webd.corpX.un $ curl webd.corpX.un
 +$ curl gowebd.corpX.un
 +$ curl https://​gowebd.corpX.un #-kv
  
-$ ### kubectl delete ingress my-webd -n my-ns+curl http://​nodeN/​ -H "Host: webd.corpX.un"​ 
 +$ curl --connect-to "":"":​kubeN:​443 https://​gowebd.corpX.un #-vk 
 + 
 +$ kubectl logs -n ingress-nginx -l app.kubernetes.io/​name=ingress-nginx -f 
 + 
 +node1# ​### kubectl delete ingress my-ingress ​-n my-ns
 </​code>​ </​code>​
  
 +=== secrets tls ===
 +
 +  * [[https://​devopscube.com/​configure-ingress-tls-kubernetes/​|How To Configure Ingress TLS/SSL Certificates in Kubernetes]]
 +
 +<​code>​
 +$ kubectl create secret tls gowebd-tls --key gowebd.key --cert gowebd.crt -n my-ns
 +    ​
 +$ kubectl get secrets -n my-ns
 +
 +$ kubectl get secret/​gowebd-tls -o yaml -n my-ns
 +
 +$ ###kubectl delete secret/​gowebd-tls -n my-ns
 +</​code>​
 +
 +==== Volumes ====
 +
 +=== PersistentVolume и PersistentVolumeVolumeClaim ===
 +<​code>​
 +root@node1:​~#​ ssh node2 mkdir /disk2
 +
 +root@node1:​~#​ ### ssh kube3 chmod 777 /disk2/
 +
 +root@node1:​~#​ ssh node2 touch /​disk2/​disk2_node2
 +
 +root@node1:​~#​ kubectl label nodes node2 disk2=yes
 +
 +root@node1:​~#​ kubectl get nodes --show-labels
 +
 +root@node1:​~#​ ###kubectl label nodes node2 disk2-
 +
 +root@node1:​~#​ cat my-debian-deployment.yaml
 +</​code><​code>​
 +...
 +        args: ["​-c",​ "while true; do echo hello; sleep 3;​done"​]
 +
 +        volumeMounts:​
 +          - name: my-disk2-volume
 +            mountPath: /data
 +
 +#        volumeMounts:​
 +#          - name: data
 +#            mountPath: /data
 +
 +      volumes:
 +        - name: my-disk2-volume
 +          hostPath:
 +            path: /disk2/
 +      nodeSelector:​
 +        disk2: "​yes"​
 +
 +#      volumes:
 +#      - name: data
 +#        persistentVolumeClaim:​
 +#          claimName: my-ha-pvc-sz64m
 +
 +      restartPolicy:​ Always
 +</​code><​code>​
 +root@node1:​~#​ kubectl apply -f my-debian-deployment.yaml
 +
 +root@node1:​~#​ kubectl get all -o wide
 +</​code>​
 +
 +  * [[https://​qna.habr.com/​q/​629022|Несколько Claim на один Persistent Volumes?]]
 +  * [[https://​serveradmin.ru/​hranilishha-dannyh-persistent-volumes-v-kubernetes/​|Хранилища данных (Persistent Volumes) в Kubernetes]]
 +  * [[https://​stackoverflow.com/​questions/​59915899/​limit-persistent-volume-claim-content-folder-size-using-hostpath|Limit persistent volume claim content folder size using hostPath]]
 +  * [[https://​stackoverflow.com/​questions/​63490278/​kubernetes-persistent-volume-hostpath-vs-local-and-data-persistence|Kubernetes persistent volume: hostpath vs local and data persistence]]
 +  * [[https://​www.alibabacloud.com/​blog/​kubernetes-volume-basics-emptydir-and-persistentvolume_594834|Kubernetes Volume Basics: emptyDir and PersistentVolume]]
 +
 +<​code>​
 +root@node1:​~#​ cat my-ha-pv.yaml
 +</​code><​code>​
 +apiVersion: v1
 +kind: PersistentVolume
 +metadata:
 +  name: my-pv-node2-sz-128m-num-001
 +  labels:
 +    type: local
 +spec:
 +## comment storageClassName for keycloak
 +  storageClassName:​ my-ha-sc
 +  capacity:
 +    storage: 128Mi
 +#    storage: 8Gi
 +#  volumeMode: Filesystem
 +  accessModes:​
 +    - ReadWriteMany
 +#    - ReadWriteOnce
 +  hostPath:
 +    path: /disk2
 +  persistentVolumeReclaimPolicy:​ Retain
 +  nodeAffinity:​
 +    required:
 +      nodeSelectorTerms:​
 +      - matchExpressions:​
 +        - key: kubernetes.io/​hostname
 +          operator: In
 +          values:
 +          - node2
 +#          - kube3
 +</​code><​code>​
 +root@node1:​~#​ kubectl apply -f my-ha-pv.yaml
 +
 +root@node1:​~#​ kubectl get persistentvolume
 +  или
 +root@node1:​~#​ kubectl get pv
 +...
 +root@node1:​~#​ ###kubectl delete pv my-pv-node2-sz-128m-num-001
 +
 +root@node1:​~#​ cat my-ha-pvc.yaml
 +</​code><​code>​
 +apiVersion: v1
 +kind: PersistentVolumeClaim
 +metadata:
 +  name: my-ha-pvc-sz64m
 +spec:
 +  storageClassName:​ my-ha-sc
 +#  storageClassName:​ local-path
 +  accessModes:​
 +    - ReadWriteMany
 +  resources:
 +    requests:
 +      storage: 64Mi
 +</​code><​code>​
 +root@node1:​~#​ kubectl apply -f my-ha-pvc.yaml
 +
 +root@node1:​~#​ kubectl get persistentvolumeclaims
 +  или
 +root@node1:​~#​ kubectl get pvc
 +...
 +
 +root@node1:​~#​ ### kubectl delete pvc my-ha-pvc-sz64m
 +</​code>​
 +
 +=== Dynamic Volume Provisioning ===
 +
 +  * [[https://​kubernetes.io/​docs/​concepts/​storage/​dynamic-provisioning/​|Dynamic Volume Provisioning]]
 +
 +== rancher local-path-provisioner ==
 +
 +  * [[https://​github.com/​rancher/​local-path-provisioner|rancher local-path-provisioner]]
 +  * [[https://​artifacthub.io/​packages/​helm/​ebrianne/​local-path-provisioner|This chart bootstraps a  deployment on a  cluster using the  package manager]]
 +
 +<​code>​
 +$ kubectl apply -f https://​raw.githubusercontent.com/​rancher/​local-path-provisioner/​v0.0.26/​deploy/​local-path-storage.yaml
 +
 +$ kubectl get sc
 +
 +$ curl https://​raw.githubusercontent.com/​rancher/​local-path-provisioner/​v0.0.26/​deploy/​local-path-storage.yaml | less
 +/​DEFAULT_PATH_FOR_NON_LISTED_NODES
 +
 +ssh root@kube1 'mkdir /​opt/​local-path-provisioner'​
 +ssh root@kube2 'mkdir /​opt/​local-path-provisioner'​
 +ssh root@kube3 'mkdir /​opt/​local-path-provisioner'​
 +ssh root@kube1 'chmod 777 /​opt/​local-path-provisioner'​
 +ssh root@kube2 'chmod 777 /​opt/​local-path-provisioner'​
 +ssh root@kube3 'chmod 777 /​opt/​local-path-provisioner'​
 +</​code>​
 +
 +  * Сервис Keycloak в [[Сервис Keycloak#​Kubernetes]]
 +
 +<​code>​
 +$ kubectl get pvc -n my-keycloak-ns
 +
 +$ kubectl get pv
 +
 +$ ###kubectl -n my-keycloak-ns delete pvc data-my-keycloak-postgresql-0
 +</​code>​
 +== longhorn ==
 +
 +<​code>​
 +kubeN:~# apt install open-iscsi
 +</​code>​
 +  * [[https://​github.com/​longhorn/​longhorn]]
 +<​code>​
 +Setting->​General
 +
 +Pod Deletion Policy When Node is Down: delete-statefuset-pod
 +</​code>​
 +
 +Подключение через kubectl proxy
 +
 +  * [[https://​stackoverflow.com/​questions/​45172008/​how-do-i-access-this-kubernetes-service-via-kubectl-proxy|How do I access this Kubernetes service via kubectl proxy?]]
 +
 +<​code>​
 +cmder> kubectl proxy
 +
 +http://​localhost:​8001/​api/​v1/​namespaces/​longhorn-system/​services/​longhorn-frontend:​80/​proxy/​
 +</​code>​
 +
 +Подключение через ingress
 +
 +!!! Добавить пример с аутентификацией !!!
 +<​code>​
 +student@server:​~/​longhorn$ cat ingress.yaml
 +apiVersion: networking.k8s.io/​v1
 +kind: Ingress
 +metadata:
 +  name: longhorn-ingress
 +  namespace: longhorn-system
 +spec:
 +  ingressClassName:​ nginx
 +  rules:
 +  - host: lh.corp13.un
 +    http:
 +      paths:
 +      - backend:
 +          service:
 +            name: longhorn-frontend
 +            port:
 +              number: 80
 +        path: /
 +        pathType: Prefix
 +</​code>​
 +
 +Использование snapshot-ов
 +
 +  * [[https://​github.com/​longhorn/​longhorn/​issues/​63?​ref=https%3A%2F%2Fgiter.vip|What should be the best procedure to recover a snapshot or a backup in rancher2/​longhorn ?]]
 +
 +<​code>​
 +kube1:~# kubectl -n my-keycloak-ns scale --replicas 0 statefulset my-keycloak
 +
 +kube1:~# kubectl -n my-keycloak-ns scale --replicas 0 statefulset my-keycloak-postgresql
 +
 +kube1:~# kubectl -n my-keycloak-ns scale --replicas 1 statefulset my-keycloak-postgresql
 +
 +kube1:~# kubectl -n my-keycloak-ns scale --replicas 1 statefulset my-keycloak
 +</​code>​
 +
 +Использование backup-ов
 +<​code>​
 +Setting -> General -> Backup Target -> nfs://​server.corp13.un:/​disk2
 +</​code>​
 +  * бекапим,​ удаляем чарт, удаляем pv/pvc, восстанавливаем Volume из бекапа,​ создаем для него PV/PVC как было, устанавливаем чарт, он подхватывает pv/pvc
 +
 +==== ConfigMap ====
 +
 +  * [[https://​www.aquasec.com/​cloud-native-academy/​kubernetes-101/​kubernetes-configmap/​|Kubernetes ConfigMap: Creating, Viewing, Consuming & Managing]]
 +  * [[https://​blog.lapw.at/​how-to-enable-ssh-into-a-kubernetes-pod/​|How to enable SSH connections into a Kubernetes pod]]
 +
 +<​code>​
 +root@node1:​~#​ cat sshd_config
 +</​code><​code>​
 +PermitRootLogin yes
 +PasswordAuthentication no
 +ChallengeResponseAuthentication no
 +UsePAM no
 +</​code><​code>​
 +root@node1:​~#​ kubectl create configmap ssh-config --from-file=sshd_config --dry-run=client -o yaml
 +...
 +
 +server:~# cat .ssh/​id_rsa.pub
 +...
 +
 +root@node1:​~#​ cat my-openssh-server-deployment.yaml
 +</​code><​code>​
 +apiVersion: v1
 +kind: ConfigMap
 +metadata:
 +  name: ssh-config
 +data:
 +  sshd_config:​ |
 +    PermitRootLogin yes
 +    PasswordAuthentication no
 +    ChallengeResponseAuthentication no
 +    UsePAM no
 +  authorized_keys:​ |
 +    ssh-rsa AAAAB.....C0zOcZ68= root@server.corpX.un
 +---
 +apiVersion: apps/v1
 +kind: Deployment
 +metadata:
 +  name: my-openssh-server
 +spec:
 +  selector:
 +    matchLabels:​
 +      app: my-openssh-server
 +  template:
 +    metadata:
 +      labels:
 +        app: my-openssh-server
 +    spec:
 +      containers:
 +      - name: my-openssh-server
 +        image: linuxserver/​openssh-server
 +        command: ["/​bin/​sh"​]
 +        args: ["​-c",​ "/​usr/​bin/​ssh-keygen -A; usermod -p '​*'​ root; /​usr/​sbin/​sshd.pam -D"]
 +        ports:
 +        - containerPort:​ 22
 +        volumeMounts:​
 +        - name: ssh-volume
 +          subPath: sshd_config
 +          mountPath: /​etc/​ssh/​sshd_config
 +        - name: ssh-volume
 +          subPath: authorized_keys
 +          mountPath: /​root/​.ssh/​authorized_keys
 +      volumes:
 +      - name: ssh-volume
 +        configMap:
 +          name: ssh-config
 +---
 +apiVersion: v1
 +kind: Service
 +metadata:
 +  name: my-openssh-server
 +spec:
 +  type: NodePort
 +  ports:
 +  - port: 22
 +    nodePort: 32222
 +  selector:
 +    app: my-openssh-server
 +</​code><​code>​
 +root@node1:​~#​ kubectl apply -f my-openssh-server-deployment.yaml
 +
 +root@node1:​~#​ iptables-save | grep 32222
 +
 +root@node1:​~#​ ###kubectl exec -ti my-openssh-server-NNNNNNNN-NNNNN -- bash
 +
 +server:~# ssh -p 32222 nodeN
 +Welcome to OpenSSH Server
 +my-openssh-server-NNNNNNNN-NNNNN:​~#​ nslookup my-openssh-server.default.svc.cluster.local
 +</​code>​
 ==== Пример с multi container pod ==== ==== Пример с multi container pod ====
  
Line 513: Line 1255:
       containers:       containers:
       - name: my-webd       - name: my-webd
-        image: server.corp13.un:​5000/​student/​webd:​latest+        image: server.corpX.un:​5000/​student/​webd:​latest
         volumeMounts:​         volumeMounts:​
         - name: html         - name: html
Line 562: Line 1304:
  
   * [[https://​helm.sh/​docs/​intro/​install/​|Installing Helm]]   * [[https://​helm.sh/​docs/​intro/​install/​|Installing Helm]]
 +  * [[https://​github.com/​helm/​helm/​releases|helm releases]]
  
 <​code>​ <​code>​
-server# wget https://​get.helm.sh/​helm-v3.9.0-linux-amd64.tar.gz+# wget https://​get.helm.sh/​helm-v3.9.0-linux-amd64.tar.gz
  
 # tar -zxvf helm-v3.9.0-linux-amd64.tar.gz # tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
Line 572: Line 1315:
  
 ==== Работа с готовыми Charts ==== ==== Работа с готовыми Charts ====
 +
 +=== ingress-nginx ===
  
   * [[https://​kubernetes.github.io/​ingress-nginx/​deploy/​|NGINX Ingress Controller Installation Guide]]   * [[https://​kubernetes.github.io/​ingress-nginx/​deploy/​|NGINX Ingress Controller Installation Guide]]
 +  * [[https://​stackoverflow.com/​questions/​56915354/​how-to-install-nginx-ingress-with-hostnetwork-on-bare-metal|stackoverflow How to install nginx-ingress with hostNetwork on bare-metal?​]]
 +  * [[https://​devpress.csdn.net/​cloud/​62fc8e7e7e66823466190055.html|devpress.csdn.net How to install nginx-ingress with hostNetwork on bare-metal?​]]
 +  * [[https://​github.com/​kubernetes/​ingress-nginx/​blob/​main/​charts/​ingress-nginx/​values.yaml]]
  
 <​code>​ <​code>​
-curl https://raw.githubusercontent.com/​kubernetes/​ingress-nginx/controller-v1.3.1/​deploy/​static/​provider/​cloud/​deploy.yaml+helm upgrade ingress-nginx --install ingress-nginx \ 
 +--set controller.hostNetwork=true,​controller.publishService.enabled=false,​controller.kind=DaemonSet,​controller.config.use-forwarded-headers=true \ 
 +--repo ​https://kubernetes.github.io/​ingress-nginx --namespace ingress-nginx --create-namespace
  
-kubectl apply -f https://​raw.githubusercontent.com/​kubernetes/​ingress-nginx/controller-v1.3.1/​deploy/​static/​provider/​cloud/​deploy.yaml+helm list --namespace ​ingress-nginx 
 +$ helm list -A
  
-$ kubectl ​delete ​-f https://​raw.githubusercontent.com/​kubernetes/​ingress-nginx/controller-v1.3.1/​deploy/​static/​provider/​cloud/​deploy.yaml+$ kubectl ​get all -ingress-nginx -o wide
  
-$ helm upgrade --install ingress-nginx ingress-nginx --repo https://​kubernetes.github.io/​ingress-nginx --namespace ingress-nginx ​--create-namespace+$ helm delete ​ingress-nginx --namespace ingress-nginx
  
-$ helm list --namespace ingress-nginx 
  
-$ ### helm delete ​ingress-nginx --namespace ingress-nginx+mkdir ingress-nginx;​ cd ingress-nginx 
 + 
 +$ helm template ingress-nginx --repo https://​kubernetes.github.io/​ingress-nginx --namespace ingress-nginx | tee t1.yaml 
 + 
 +$ helm show values ingress-nginx --repo https://​kubernetes.github.io/​ingress-nginx | tee values.yaml.orig 
 + 
 +$ cat values.yaml 
 +</​code><​code>​ 
 +controller:​ 
 +  hostNetwork:​ true 
 +  publishService:​ 
 +    enabled: false 
 +  kind: DaemonSet 
 + ​config:​ 
 +   ​use-forwarded-headers:​ true 
 +   ​allow-snippet-annotations:​ true 
 +</​code><​code>​ 
 +helm template ​ingress-nginx -f values.yaml --repo https://​kubernetes.github.io/​ingress-nginx -n ingress-nginx | tee t2.yaml 
 + 
 +$ helm upgrade ingress-nginx -i ingress-nginx -f values.yaml --repo https://​kubernetes.github.io/​ingress-nginx -n ingress-nginx --create-namespace 
 + 
 +$ kubectl exec -n ingress-nginx ​pods/​ingress-nginx-controller-<​TAB>​ -- cat /​etc/​nginx/​nginx.conf | tee nginx.conf | grep use_forwarded_headers 
 + 
 +$ kubectl -n ingress-nginx describe service/​ingress-nginx-controller 
 +... 
 +Endpoints: ​               192.168.X.221:​80,​192.168.X.222:​80,​192.168.X.223:​80 
 +... 
 + 
 +# kubectl get clusterrole -A | grep -i ingress 
 +# kubectl get clusterrolebindings -A | grep -i ingress 
 +# kubectl get validatingwebhookconfigurations -A | grep -i ingress
 </​code>​ </​code>​
 ==== Развертывание своего приложения ==== ==== Развертывание своего приложения ====
Line 592: Line 1372:
   * [[https://​opensource.com/​article/​20/​5/​helm-charts|How to make a Helm chart in 10 minutes]]   * [[https://​opensource.com/​article/​20/​5/​helm-charts|How to make a Helm chart in 10 minutes]]
   * [[https://​stackoverflow.com/​questions/​49812830/​helm-upgrade-with-same-chart-version-but-different-docker-image-tag|Helm upgrade with same chart version, but different Docker image tag]]   * [[https://​stackoverflow.com/​questions/​49812830/​helm-upgrade-with-same-chart-version-but-different-docker-image-tag|Helm upgrade with same chart version, but different Docker image tag]]
 +  * [[https://​stackoverflow.com/​questions/​69817305/​how-set-field-app-version-in-helm3-chart|how set field app-version in helm3 chart?]]
  
 <​code>​ <​code>​
-$ helm create webd-chart+gitlab-runner@server:​~/​gowebd-k8s$ helm create webd-chart 
 + 
 +$ less webd-chart/​templates/​deployment.yaml
  
 $ cat webd-chart/​Chart.yaml $ cat webd-chart/​Chart.yaml
Line 604: Line 1387:
 ... ...
 appVersion: "​latest"​ appVersion: "​latest"​
 +#​appVersion:​ ver1.7 ​  #for vanilla argocd
 </​code><​code>​ </​code><​code>​
 $ cat webd-chart/​values.yaml $ cat webd-chart/​values.yaml
 </​code><​code>​ </​code><​code>​
 ... ...
 +replicaCount:​ 2
 +
 image: image:
   repository: server.corpX.un:​5000/​student/​webd   repository: server.corpX.un:​5000/​student/​webd
Line 616: Line 1402:
 ... ...
 service: service:
-  ​type: NodePort+#  ​type: NodePort
 ... ...
 ingress: ingress:
Line 623: Line 1409:
 ... ...
   hosts:   hosts:
-    - host: webd.corp13.un+    - host: webd.corpX.un
 ... ...
 +#  tls: []
 +#  tls:
 +#    - secretName: gowebd-tls
 +#      hosts:
 +#        - gowebd.corpX.un
 +...
 +#​APWEBD_HOSTNAME:​ "​apwebd.corp13.un"​
 +#​KEYCLOAK_HOSTNAME:​ "​keycloak.corp13.un"​
 +#​REALM_NAME:​ "​corp13"​
 </​code><​code>​ </​code><​code>​
 $ less webd-chart/​templates/​deployment.yaml $ less webd-chart/​templates/​deployment.yaml
Line 630: Line 1425:
 ... ...
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
 +#          env:
 +#          - name: APWEBD_HOSTNAME
 +#            value: "{{ .Values.APWEBD_HOSTNAME }}"
 +#          - name: KEYCLOAK_HOSTNAME
 +#            value: "{{ .Values.KEYCLOAK_HOSTNAME }}"
 +#          - name: REALM_NAME
 +#            value: "{{ .Values.REALM_NAME }}"
 +...
 +#          livenessProbe:​
 +#            httpGet:
 +#              path: /
 +#              port: http
 +#          readinessProbe:​
 +#            httpGet:
 +#              path: /
 +#              port: http
 ... ...
 </​code><​code>​ </​code><​code>​
 +$ helm template my-webd webd-chart/ | less
 +
 $ helm install my-webd webd-chart/ -n my-ns --create-namespace --wait $ helm install my-webd webd-chart/ -n my-ns --create-namespace --wait
 +
 +$ kubectl describe events -n my-ns | less
  
 $ export HELM_NAMESPACE=my-ns $ export HELM_NAMESPACE=my-ns
Line 638: Line 1453:
 $ helm list $ helm list
  
-$ helm upgrade my-webd webd-chart/ --set=image.tag=ver1.10+### helm upgrade my-webd webd-chart/ --set=image.tag=ver1.10
  
 $ helm history my-webd $ helm history my-webd
Line 653: Line 1468:
   * [[https://​medium.com/​containerum/​how-to-make-and-share-your-own-helm-package-50ae40f6c221|How to make and share your own Helm package]]   * [[https://​medium.com/​containerum/​how-to-make-and-share-your-own-helm-package-50ae40f6c221|How to make and share your own Helm package]]
   * [[https://​docs.gitlab.com/​ee/​user/​profile/​personal_access_tokens.html|Gitlab Personal access tokens]]   * [[https://​docs.gitlab.com/​ee/​user/​profile/​personal_access_tokens.html|Gitlab Personal access tokens]]
 +  * [[Инструмент GitLab#​Подключение через API]] - Role: Mainteiner, api, read_registry,​ write_registry
 <​code>​ <​code>​
-$ helm repo add --username student --password ​NNNNNN-NNNNNNNNNNNNN ​webd http://192.168.13.1/​api/​v4/​projects/​6/​packages/​helm/​stable+gitlab-runner@server:​~/​gowebd-k8s$ helm repo add --username student --password ​NNNNN-NNNNNNNNNNNNNNNNNNN ​webd http://server.corpX.un/​api/​v4/​projects/​N/​packages/​helm/​stable 
 +"​webd"​ has been added to your repositories
  
-$ helm repo list+gitlab-runner@server:​~/​gowebd-k8s### helm repo remove webd
  
-$ helm package webd-chart +gitlab-runner@server:​~/​gowebd-k8shelm repo list
-ls *tgz+
  
-$ helm plugin install https://​github.com/​chartmuseum/​helm-push +gitlab-runner@server:~/gowebd-k8s$ helm package ​webd-chart
-$ helm cm-push ​webd-chart-0.1.0.tgz webd+
  
-... С другого кластера подключаем (аналогично) наш репозиторий и ...+gitlab-runner@server:​~/​gowebd-k8s$ tar -tf webd-chart-0.1.1.tgz
  
-$ helm search repo webd+gitlab-runner@server:​~/​gowebd-k8s$ helm plugin install https://​github.com/​chartmuseum/​helm-push
  
-$ helm repo update ​webd+gitlab-runner@server:​~/​gowebd-k8s$ helm cm-push webd-chart-0.1.1.tgz ​webd
  
-$ helm install my-webd webd/​webd-chart+gitlab-runner@server:​~/​gowebd-k8srm webd-chart-0.1.1.tgz 
 +</​code><​code>​ 
 +kube1:~# helm repo add webd http://​server.corpX.un/​api/​v4/​projects/​N/​packages/​helm/​stable 
 + 
 +kube1:~# helm repo update 
 + 
 +kube1:~# helm search repo webd 
 + 
 +kube1:~# helm repo update webd 
 + 
 +kube1:​~# ​helm install my-webd webd/​webd-chart 
 + 
 +kube1:~# ###helm uninstall my-webd 
 +</​code><​code>​ 
 +kube1:~# mkdir gowebd; cd gowebd 
 + 
 +kube1:​~/​gowebd#​ ###helm pull webd-chart --repo https://​server.corp13.un/​api/​v4/​projects/​1/​packages/​helm/​stable 
 + 
 +kube1:​~/​gowebd#​ helm show values webd-chart --repo https://​server.corp13.un/​api/​v4/​projects/​1/​packages/​helm/​stable | tee values.yaml.orig 
 + 
 +kube1:​~/​gowebd#​ cat values.yaml 
 +</​code><​code>​ 
 +replicaCount:​ 3 
 +image: 
 +  tag: "​ver1.1"​ 
 +#​REALM_NAME:​ "​corp"​ 
 +</​code><​code>​ 
 +kube1:​~/​gowebd#​ helm upgrade my-webd -i webd-chart -f values.yaml -n my-ns --create-namespace --repo https://​server.corp13.un/​api/​v4/​projects/​1/​packages/​helm/​stable 
 + 
 +$ curl http://​kubeN -H "Host: gowebd.corpX.un"​ 
 + 
 +kube1:​~/​gowebd#​ ###helm uninstall my-webd -n my-ns
 </​code>​ </​code>​
  
 ==== Работа с публичными репозиториями ==== ==== Работа с публичными репозиториями ====
 <​code>​ <​code>​
 +helm repo add gitlab https://​charts.gitlab.io
 +
 +helm search repo -l gitlab/​gitlab-runner
 +
 +helm show values gitlab/​gitlab-runner | tee values.yaml
 +
 +gitlab-runner@server:​~$ diff values.yaml values.yaml.orig
 +</​code><​code>​
 +...
 +gitlabUrl: http://​server.corpX.un/​
 +...
 +runnerRegistrationToken:​ "​NNNNNNNNNNNNNNNNNNNNNNNN"​
 +...
 +148,149c142
 +<   ​create:​ true
 +---
 +>   ​create:​ false
 +325d317
 +<         ​privileged = true
 +432c424
 +<   ​allowPrivilegeEscalation:​ true
 +---
 +>   ​allowPrivilegeEscalation:​ false
 +435c427
 +<   ​privileged:​ true
 +---
 +>   ​privileged:​ false
 +</​code><​code>​
 +gitlab-runner@server:​~$ helm upgrade -i gitlab-runner gitlab/​gitlab-runner -f values.yaml -n gitlab-runner --create-namespace
 +
 +gitlab-runner@server:​~$ kubectl get all -n gitlab-runner
 +</​code><​code>​
 $ helm search hub -o json wordpress | jq '​.'​ | less $ helm search hub -o json wordpress | jq '​.'​ | less
  
Line 682: Line 1559:
 $ helm show values bitnami/​wordpress $ helm show values bitnami/​wordpress
 </​code>​ </​code>​
 +
 +===== Kubernetes Dashboard =====
 +
 +  * https://​kubernetes.io/​docs/​tasks/​access-application-cluster/​web-ui-dashboard/​
 +  * https://​github.com/​kubernetes/​dashboard/​blob/​master/​docs/​user/​access-control/​creating-sample-user.md
 +
 +<​code>​
 +$ kubectl apply -f https://​raw.githubusercontent.com/​kubernetes/​dashboard/​v2.7.0/​aio/​deploy/​recommended.yaml
 +
 +$ cat dashboard-user-role.yaml
 +</​code><​code>​
 +---
 +apiVersion: v1
 +kind: ServiceAccount
 +metadata:
 +  name: admin-user
 +  namespace: kubernetes-dashboard
 +---
 +apiVersion: rbac.authorization.k8s.io/​v1
 +kind: ClusterRoleBinding
 +metadata:
 +  name: admin-user
 +roleRef:
 +  apiGroup: rbac.authorization.k8s.io
 +  kind: ClusterRole
 +  name: cluster-admin
 +subjects:
 +- kind: ServiceAccount
 +  name: admin-user
 +  namespace: kubernetes-dashboard
 +---
 +apiVersion: v1
 +kind: Secret
 +metadata:
 +  name: admin-user
 +  namespace: kubernetes-dashboard
 +  annotations:​
 +    kubernetes.io/​service-account.name:​ "​admin-user"​
 +type: kubernetes.io/​service-account-token
 +</​code><​code>​
 +$ kubectl apply -f dashboard-user-role.yaml
 +
 +$ kubectl -n kubernetes-dashboard create token admin-user
 +
 +$ kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={"​.data.token"​} | base64 -d ; echo
 +
 +cmder$ kubectl proxy
 +</​code>​
 +
 +  * http://​localhost:​8001/​api/​v1/​namespaces/​kubernetes-dashboard/​services/​https:​kubernetes-dashboard:/​proxy/​
  
 ===== Дополнительные материалы ===== ===== Дополнительные материалы =====
Line 696: Line 1623:
 ... ...
  
-student@node2:​~$ minikube start --driver=none --insecure-registry "​server.corp13.un:​5000"​+student@node2:​~$ minikube start --driver=none --insecure-registry "​server.corpX.un:​5000"​
 </​code>​ </​code>​
  
Line 737: Line 1664:
  
 <​code>​ <​code>​
-root@gate.corp13.un:~# curl -L https://​github.com/​kubernetes/​kompose/​releases/​download/​v1.26.0/​kompose-linux-amd64 -o kompose +root@gate:​~#​ curl -L https://​github.com/​kubernetes/​kompose/​releases/​download/​v1.26.0/​kompose-linux-amd64 -o kompose 
-root@gate.corp13.un:~# chmod +x kompose +root@gate:​~#​ chmod +x kompose 
-root@gate.corp13.un:~# sudo mv ./kompose /​usr/​local/​bin/​kompose+root@gate:​~#​ sudo mv ./kompose /​usr/​local/​bin/​kompose
 </​code>​ </​code>​
  
система_kubernetes.1664279012.txt.gz · Last modified: 2022/09/27 14:43 by val