Инструмент GitLab

• https://ru.wikipedia.org/wiki/GitLab - Википедия

• Youtube. RomNero. GitLab. Devops система
• GitLab бесплатен? gitlab-ee vs gitlab-ce

• RAM от 4Gb

• Install self-managed GitLab
• Доступно из РФ: https://packages.gitlab.com/gitlab/gitlab-ce

server# apt-get install -y curl ca-certificates perl

server# curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | bash

server# time EXTERNAL_URL="http://$(hostname)" apt-get install gitlab-ce
...
real    38m49.787s  !!! Загрузка может прерываться, надо повторять команду !!!
..

• Install GitLab using Docker Compose
• Технология Docker
• docker-compose

# cat docker-compose.yml

version: '3.6'
services:
  web:
    image: 'gitlab/gitlab-ce:latest'
#    image: 'gitlab/gitlab-ce:16.7.4-ce.0'
    restart: always
    hostname: 'server.corpX.un'
    environment:
      GITLAB_ROOT_PASSWORD: "strongpassword"
      GITLAB_OMNIBUS_CONFIG: |
        prometheus_monitoring['enable'] = false
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['registry_host'] = "server.corpX.un"
        external_url 'http://server.corpX.un'
        registry_external_url 'http://server.corpX.un'
        gitlab_rails['registry_port'] = "5000"
        registry['registry_http_addr'] = "server.corpX.un:5000"
#        external_url 'https://server.corpX.un'
#        registry_external_url 'https://server.corpX.un:5000'
#        gitlab_rails['registry_port'] = "5050"
#        registry['registry_http_addr'] = "server.corpX.un:5050"
    ports:
      - '80:80'
#      - '443:443'
      - '2222:22'
      - '5000:5000'
    volumes:
      - '/etc/gitlab:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
    shm_size: '256m'
    logging:
      driver: "json-file"
      options:
        max-size: "2048m"

# ### cat /etc/gitlab/ssl/gitlab.bmstu.ru.{crt,key}

# docker-compose up -d

# docker logs root_web_1 -n 10 -f

### docker-compose stop
### rm -r /srv/gitlab/ /etc/gitlab/

• https://galaxy.ansible.com/ui/repo/published/hifis/toolkit/content/role/gitlab/

• http://server.corpX.un/

• Токен доступа: Settings → Access Tokens (Project access tokens), в примере достаточно role: Reporter, Scopes: api
• Номер проекта: Settings → General (Where do I find the project ID for the GitLab API?)
• How to download a single file from GitLab?

root@node1,2,3:~# curl "http://server.corpX.un/api/v4/projects/2/repository/files/docker-compose.yml/raw" | tee docker-compose.yml

  или, для НЕ публичных проектов
root@node1,2,3:~# curl --header "PRIVATE-TOKEN: NNNNNNNNNNNNNNNNNNNNN" "http://server.corpX.un/api/v4/projects/4/repository/files/docker-compose.yml/raw?ref=master" | tee docker-compose.yml

• ansible-pull

client1:~/ansible-pull-gpo# cat readme.md

sudo -i

export BR=main; bash <(curl -s http://gate.corp13.un/api/v4/projects/1/repository/files/start.sh/raw?ref=$BR)

# cat /etc/gitlab/gitlab.rb

...
external_url 'http://server.corpX.un'
...

### docker exec -it root_web_1 bash

# gitlab-ctl show-config

# time gitlab-ctl reconfigure
...
real    2m34.726s
...

• The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if you’re using the built-in Let’s Encrypt integration
• Настройка работы Gitlab с registry без ssl - Sysadmin

# cat /etc/gitlab/gitlab.rb

...
registry_external_url 'http://server.corpX.un'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "server.corpX.un"
gitlab_rails['registry_port'] = "5000"
registry['registry_http_addr'] = "server.corpX.un:5000"
...

• Проверка конфигурации и перезапуск

# cat /etc/gitlab/gitlab.rb

...
grafana['http_addr'] = '0.0.0.0'
...

• Проверка конфигурации и перезапуск

# cat /etc/gitlab/gitlab.rb

...
prometheus_monitoring['enable'] = false
...

• Проверка конфигурации и перезапуск

# time rm -rf /var/opt/gitlab/prometheus/data/*

• Configure HTTPS manually
• Configure Custom SSL to Secure GitLab Server

mkdir -p /etc/gitlab/ssl/

cp wild.crt -v /etc/gitlab/ssl/$(hostname).crt
cp wild.key -v /etc/gitlab/ssl/$(hostname).key

# cat /etc/gitlab/gitlab.rb

...
external_url 'https://server.corpX.un'
...
# nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
# nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
...
letsencrypt['enable'] = false
...

• Проверка конфигурации и перезапуск

• Username - login, Name - ФИО

# cat /etc/gitlab/initial_root_password

• gitlab initial root password reset

# gitlab-rake "gitlab:password:reset[root]"

• Integrate LDAP with GitLab
• Установка и настройка OpenLDAP
• Хранение учетных записей UNIX в LDAP !!! с атрибутом почты и паролем

# cat /etc/gitlab/gitlab.rb

...
gitlab_rails['ldap_enabled'] = true

gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
  main:
    label: 'LDAP'
    host: 'server.corpX.un'
#    host: 'server2.corpX.un'
    port: 389
#    uid: 'uid'
    uid: 'sAMAccountName'
#    bind_dn: 'cn=admin,dc=corpX,dc=un'
#    password: 'secret'
    bind_dn: 'cn=Administrator,cn=Users,dc=corpX,dc=un'
    password: 'Pa$$w0rd'
    encryption: 'plain'
#    active_directory: false
    active_directory: true
    base: 'dc=corpX,dc=un'
EOS
...

• Проверка конфигурации и перезапуск

• Install GitLab Runner manually on GNU/Linux
• https://val.bmstu.ru/unix/Git/gitlab-runner_amd64.deb (16.10.0)

# wget http://gate.isp.un/unix/Git/gitlab-runner_amd64.deb

##2 часа## curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"

# dpkg -i gitlab-runner_amd64.deb

# gitlab-runner register --help

# export CI_SERVER_URL=http://server.corpX.un

# gitlab-runner register
...
Enter the GitLab instance URL: http://server.corpX.un
Enter the registration token: ...
...
Enter tags for the runner: dhcptest, dhcpdeploy
  или
Enter tags for the runner: openvpn1deploy
...
Enter an executor: shell
...

или

# gitlab-runner register -n --executor "shell" -u http://server.corpX.un -r "NNNNNNNNNNNNNNNNNNNNNNNNNNNN"

или по инструкции в “New instance runner”

Перезапускать не нужно

# gitlab-runner verify

# cat /etc/gitlab-runner/config.toml
log_level = "debug"
...

# systemctl restart gitlab-runner

• Использование Docker in Docker в GitLab

gate:~### docker stop gitlab-runner; docker rm gitlab-runner
gate:~### rm /srv/gitlab-runner/config/config.toml

gate:~# docker run -d --name gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

• Включаем Docker Insecure Private Registry

gate:~# docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --url "http://server.corpX.un/" \
  --registration-token "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN" \
  --executor "docker" \
  --docker-image "docker:stable" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
  --description "dood-runner"

• Можно отключить Docker Insecure Private Registry

gate:~# docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --url "http://server.corpX.un/" \
  --registration-token "NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN" \
  --executor "docker" \
  --docker-image "docker:stable" \
  --docker-privileged \
  --description "dind-runner"

gate:~# docker volume ls

gate:~# docker volume inspect ...

# cp wild.crt /srv/gitlab-runner/config/

docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
...
  --url "https://server.corp20.un/" \
  --tls-ca-file "/etc/gitlab-runner/wild.crt" \
...

• CI/CD templates
• GitLab: understanding pipelines, stages, jobs and organising them efficiently for speed and feedback loop
• How to disable auto pipelines in gitlab

IDE GitLab->New File: .gitlab-ci.yml

или

CI/CD -> Editor -> Configure Pipelines

или

Build -> Pipeline editor -> Configure Pipelines

#stages:
#  - build
#  - test
#  - deploy

test1-job:
  stage: test
  script:
    - echo $(date) "Do test dhcpd" >> /tmp/Bash.gitlab-ci.log
    - make test
  tags:
    - dhcptest

deploy1-job:
  stage: deploy
  script:
    - echo $(date) "Do deploy dhcpd" >> /tmp/Bash.gitlab-ci.log
    - sudo make install
  tags:
    - dhcpdeploy

• Limit Gitlab CI pipelines to specific branches
• Get Branch name in gitlab ci

Administrator@Ra-master ~/openvpn1 (test)
λ touch .gitlab-ci.yml
  или
Build -> Pipeline editor -> Configure Pipelines

deploy_test:
  stage: deploy
  script:
    - ansible-playbook openvpn1.yaml -i inventory.yaml -e "variable_host=test_nodes"
  tags:
    - openvpn1deploy
  only:
    - test

deploy_prod:
  stage: deploy
  script:
    - ansible-playbook openvpn1.yaml -i inventory.yaml
  tags:
    - openvpn1deploy
  only:
#    - master
#    - main

• Технология Docker Предоставление прав непривилегированным пользователям

• Use Docker to build Docker images
• Predefined variables reference
• Add a CI/CD variable to a project

# Можно назначить в GitLab (Settings -> CI/CD -> Variables)
# export MY_CI_REGISTRY=server.corpX.un:5000 
# export MY_CI_REGISTRY_IMAGE=student/webd
# или использовать встроенные CI_REGISTRY и CI_REGISTRY_IMAGE 
# поскольку используем этот же проект GitLab как Registry

# в GitLab будет установлено автоматически после git commit -m "ver 1.2" и git push
# export CI_COMMIT_MESSAGE="ver 1.2"   

gitlab-runner@server:~/webd$ cat build.sh

#!/bin/sh

VER="$(echo $CI_COMMIT_MESSAGE | sed 's/[^a-zA-Z0-9\.]//g')"

# needed once
# docker login -u $MY_CI_REGISTRY_USER -p $MY_CI_REGISTRY_PASSWORD $MY_CI_REGISTRY
# docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

docker build -t webd webd

#docker run --rm -e MYMODE=TEST webd || exit 1

#docker tag webd $MY_CI_REGISTRY/$MY_CI_REGISTRY_IMAGE:$VER
#docker tag webd $MY_CI_REGISTRY/$MY_CI_REGISTRY_IMAGE
docker tag webd $CI_REGISTRY_IMAGE:$VER
docker tag webd $CI_REGISTRY_IMAGE

# previously need: docker login ...

#docker push $MY_CI_REGISTRY/$MY_CI_REGISTRY_IMAGE:$VER
#docker push $MY_CI_REGISTRY/$MY_CI_REGISTRY_IMAGE
docker push $CI_REGISTRY_IMAGE:$VER
docker push $CI_REGISTRY_IMAGE

gitlab-runner@server:~/webd$ cat .gitlab-ci.yml

stages:
  - lintertest
  - build
#  - deploy

lintertest1:
  stage: lintertest
  script:
#    - echo $(date) "Do a test webd here" >> /tmp/Bash.gitlab-ci.log
    - shellcheck webd/webd
  tags:
    - shellcheck

build1:
  stage: build
  script:
#    - echo $(date) "Do a build webd here" >> /tmp/Bash.gitlab-ci.log
#    - env | tee -a /tmp/Bash.gitlab-ci.log
    - sh build.sh
  tags:
    - webdbuild

#deploy1:
#  stage: deploy
#  script:
#    - sh deploy.sh
#  tags:
#    - webddeploy

### OR .gitlab-ci.yml for gowebd-k8s project running from another pipeline ###

#deploy1:
#  stage: deploy
#  variables:
#    HELM_NAMESPACE: "my-ns"
#  rules:
#    - if: '$CI_PIPELINE_SOURCE == "pipeline" && $VER'
#  script:
#    - env
#    - envsubst < my-webd-deployment-env.yaml | kubectl apply -f - -n my-ns
#    - helm upgrade -i my-webd webd-chart/ --set=image.tag=$VER --create-namespace

gitlab-runner@server:~/webd$ cp my-webd-deployment.yaml my-webd-deployment-env.yaml
  или
gitlab-runner@server:~/gowebd-k8s$ scp root@node1:my-webd-deployment.yaml my-webd-deployment-env.yaml

gitlab-runner@server:~/webd$ cat my-webd-deployment-env.yaml

...
        image: server.corpX.un:5000/student/webd:$VER
...

# в GitLab будет устанавлено автоматически
gitlab-runner@gate:~/webd$ export CI_COMMIT_MESSAGE="ver 1.2"

gitlab-runner@gate:~/webd$ cat deploy.sh

#!/bin/sh

#alias kubectl='minikube kubectl --'

kubectl apply -f my-webd-deployment.yaml -n my-ns

#export VER="$(echo $CI_COMMIT_MESSAGE | sed 's/[^a-zA-Z0-9\.]//g')"

#envsubst < my-webd-deployment-env.yaml | kubectl apply -f - -n my-ns

kubectl apply -f my-webd-service.yaml -n my-ns


#export HELM_NAMESPACE=my-ns
#helm upgrade --install my-webd webd-chart/ --set=image.tag=$VER --create-namespace

gitlab-runner@server:~/$ kubectl describe replicaset.apps/my-webd-NNNNNNNNNNN -n my-ns

• Build Golang Docker images with GitLab CI Pipelines
• Best practices for building docker images with GitLab CI

• How to run a script from file in another project using include in GitLab CI?
• Gitlab триггеры и для каких тестов их стоит использовать?

• Container Image Build Tools: Docker vs. Buildah vs. kaniko
• Use kaniko to build Docker images
• A Tale of Two Container Image Tools: Skopeo and Crane

student@client1:~/gowebd$ cat .gitlab-ci.yml

stages:
  - build
  - push
#  - deploy

#variables:
#  DOCKER_TLS_CERTDIR: ""

#services:
#  - name: docker:dind
#    command:
#      [
#        '--insecure-registry=server.corpX.un:5000',
#      ]

before_script:
  - env
#  - docker info
  - echo -n $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY

Build:
  stage: build
#  image:
#    name: gcr.io/kaniko-project/executor:v1.9.0-debug
#    entrypoint: [""]
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - >
      docker build
      --pull
      --cache-from $CI_REGISTRY_IMAGE:latest
      --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

#    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
#    - /kaniko/executor
#      --insecure --skip-tls-verify
#      --context "${CI_PROJECT_DIR}"
#      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
#      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}"

Push latest:
#  image:
#    name: gcr.io/go-containerregistry/crane:debug
#    entrypoint: [""]
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    - main
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest

#    - crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
#    - crane --insecure cp $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest

Push tag:
#  image:
#    name: gcr.io/go-containerregistry/crane:debug
#    entrypoint: [""]
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    - tags
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

#   - crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
#   - crane --insecure cp $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

#Deploy:
#  variables:
#    VER: "$CI_COMMIT_REF_NAME"
#  stage: deploy
#  only:
#    - tags
#  trigger:
#    project: student/gowebd-k8s

• https://github.com/zmartzone/mod_auth_openidc/wiki/GitLab-OAuth2
• Управление доступом к HTTP серверу с использованием OpenID аутентификации
• Admin Area→ Applications

Name: test-cgi
Redirect URI: http://gate.corp13.un/cgi-bin/test-cgi  !!! Если URL каталога, то без финального "/" !!!
Trusted: Yes
Confidential: Yes
Scopes: openid

Application ID: ...
Secret: ...
Callback URL = Redirect URI

• You can use GitLab as a client application with OpenID Connect as an OmniAuth provider
• use self-signed to integate gitlab with keycloak but see error: certificate verify failed (self signed certificate))
• Using Keycloak as SSO for Gitlab with pre-existing users (no autocreate)

# cp server.crt /etc/gitlab/trusted-certs/
  или
# cp ca.crt /etc/gitlab/trusted-certs/

# cat /etc/gitlab/gitlab.rb

...
gitlab_rails['omniauth_providers'] = [
  {
    name: "openid_connect", # do not change this parameter
    label: "Keycloak", # optional label for login button, defaults to "Openid Connect"
    args: {
      name: "openid_connect",
      scope: ["openid", "profile", "email"],
      response_type: "code",
#     issuer:  "https://keycloak.example.com/realms/myrealm",
      issuer:  "https://keycloak.corpX.un/realms/corpX",
      client_auth_method: "query",
      discovery: true,
      uid_field: "preferred_username",
      pkce: true,
      client_options: {
#        identifier: "<YOUR CLIENT ID>",
        identifier: "any-client",
#        secret: "<YOUR CLIENT SECRET>",
        secret: "anystring",
#        redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback"
        redirect_uri: "https://gate.corpX.un/users/auth/openid_connect/callback"
      }
    }
  }
]
...

• Проверка конфигурации и перезапуск
• User → Profile → Account → Select a service to sign in with → Keycloak