Методические материалы Лохтурова Вячеслава

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Решение Ceph ======

  * [[https://youtu.be/OB5jysMLjIM|Разворачиваем распределенное отказоустойчивое хранилище Ceph за 45 минут]]


  * [[https://ru.wikipedia.org/wiki/Ceph|Ceph — свободная программная объектная сеть хранения (Материал из Википедии)]]
  * [[https://ceph.io/get/|Официальный сайт объектной системы хранения Ceph/GETTING STARTED]]

  * [[https://habr.com/ru/post/313644/|Знакомство с хранилищем Ceph в картинках (Хабр)]]
  * [[https://habr.com/ru/company/oleg-bunin/blog/431536/|Ceph. Анатомия катастрофы (Хабр)]]

  * [[http://onreader.mdl.ru/LearningCeph/content/index.html|Практическое руководство по разработке, реализации и управлению вашей управляемой программным обеспечением, массивно масштабируемой системы хранения Ceph]]
  * [[http://onreader.mdl.ru/MasteringCeph.2ed/content/index.html|Полное руководство Ceph, 2е изд.]]
  * [[http://onreader.mdl.ru/CephCookbook/content/index.html|Книга рецептов Ceph]]

===== Развертывание =====

  * [[https://docs.ceph.com/en/latest/install/manual-deployment/|MANUAL DEPLOYMENT]]
  * [[https://www.server-world.info/en/note?os=Debian_11&p=ceph14&f=1|Debian 11 : Ceph Nautilus]]

==== Установка ====
<code>
nodeN# apt install ceph

nodeN# ceph -v
</code>

==== MON ====

  * [[https://docs.ceph.com/en/latest/man/8/ceph-mon/|CEPH-MON – CEPH MONITOR DAEMON]]

=== Начальная конфигурация ===
<code>
nodeN# cat /etc/ceph/ceph.conf
</code><code>
[global]
	fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
	mon initial members = node3,node4,node5
	mon host = 192.168.X.3,192.168.X.4,192.168.X.5
	public network = 192.168.X.0/24
	auth cluster required = none
	auth service required = none
	auth client required = none
</code><code>
scp /etc/ceph/ceph.conf node4:/etc/ceph/
scp /etc/ceph/ceph.conf node5:/etc/ceph/
</code><code>
monmaptool --create --add node3 192.168.X.3 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
monmaptool --add node4 192.168.X.4 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
monmaptool --add node5 192.168.X.5 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
</code><code>
monmaptool --print /tmp/monmap
</code><code>
scp /tmp/monmap node4:/tmp/
scp /tmp/monmap node5:/tmp/
</code><code>
node3# bash -c '
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node3
ssh node4 sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node4
ssh node5 sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node5
'
</code><code>
node3# bash -c '
sudo -u ceph ceph-mon --mkfs -i node3 --monmap /tmp/monmap
ssh node4 sudo -u ceph ceph-mon --mkfs -i node4 --monmap /tmp/monmap
ssh node5 sudo -u ceph ceph-mon --mkfs -i node5 --monmap /tmp/monmap
'
</code><code>
node3# ls -l /var/lib/ceph/mon/ceph-node3
</code><code>
node3# bash -c '
systemctl start ceph-mon@node3
systemctl enable ceph-mon@node3
ssh node4 systemctl start ceph-mon@node4
ssh node4 systemctl enable ceph-mon@node4
ssh node5 systemctl start ceph-mon@node5
ssh node5 systemctl enable ceph-mon@node5
'
</code>

  * [[https://www.suse.com/support/kb/doc/?id=000019960|Cluster status shows a "mons are allowing insecure global_id reclaim" health warning]]

<code>
node3# ceph mon enable-msgr2

node3# ceph config set mon mon_warn_on_insecure_global_id_reclaim false
node3# ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
</code><code>
# ceph -s

# ps ax | grep ceph
</code>
=== Поиск проблемных узлов ===
<code>
# ceph health detail
</code><code>
MON_DOWN 1/3 mons down, quorum node4,node5
    mon.node3 (rank 0) addr 192.168.X.3:6789/0 is down (out of quorum)
</code>

==== MGR ====

  * [[https://docs.ceph.com/en/latest/mgr/index.html|CEPH MANAGER DAEMON]]

<code>
node3# bash -c '
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node3
systemctl start ceph-mgr@node3
systemctl enable ceph-mgr@node3

ssh node4 sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node4
ssh node4 systemctl start ceph-mgr@node4
ssh node4 systemctl enable ceph-mgr@node4

ssh node5 sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node5
ssh node5 systemctl start ceph-mgr@node5
ssh node5 systemctl enable ceph-mgr@node5
'
</code><code>
# ceph -s

debian13# ceph health detail
debian13# ceph health mute MGR_MODULE_DEPENDENCY
</code>

==== OSD ====

  * [[https://docs.ceph.com/en/latest/man/8/ceph-osd/|CEPH-OSD – CEPH OBJECT STORAGE DAEMON]]

=== Начальная конфигурация ===
<code>
# lsblk
</code><code>
node3# bash -c '
ceph-volume lvm create --data /dev/sdb
ssh node4 ceph-volume lvm create --data /dev/sdb
ssh node5 ceph-volume lvm create --data /dev/sdb
'
</code><code>
# vgs

# ps ax | grep ceph

# ceph -s

# ceph osd stat

# ceph osd tree

# ceph osd df
</code>

=== Добавление узла OSD ===

<code>
node3# scp /etc/ceph/ceph.conf node6:/etc/ceph/

node3# ssh node6 ceph-volume lvm create --data /dev/sdb

node3# watch ceph -s
</code>

=== Удаление узла OSD ===

<code>
ceph osd safe-to-destroy osd.0
OSD(s) 0 are safe to destroy without reducing data durability.

ceph osd destroy 0 --yes-i-really-mean-it
ceph osd purge 0 --yes-i-really-mean-it
</code>

=== Управление приоритетом использования узла OSD ===

<code>

ceph osd crush reweight osd.2 0
ceph osd crush reweight osd.2 0.00389

node3# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
 0   hdd       0  0.83000 4.00GiB 1.01GiB 2.99GiB 25.30 0.37   0
 1   hdd 0.00389  0.83000 4.00GiB 2.78GiB 1.21GiB 69.66 1.02  74
 2   hdd 0.00389  0.83000 4.00GiB 3.04GiB  980MiB 76.04 1.11  81
 3   hdd 0.00389  1.00000 4.00GiB 3.43GiB  575MiB 85.96 1.26  99
 4   hdd 0.00389  1.00000 4.00GiB 3.38GiB  631MiB 84.58 1.24 102

ceph osd reweight osd.2 1
</code>

==== OSD POOL ====

  * [[https://docs.ceph.com/en/latest/rados/operations/pools/|POOLS]]

=== Расчет параметров и создание POOL ===

  * [[https://docs.ceph.com/en/latest/rados/operations/placement-groups/|PLACEMENT GROUPS]]

<code>
Total PGs = OSDs * 100 / pool_size

4*100/3 = 133,3...
</code><code>
# ceph osd pool create test-pool1 128

# ceph -s
</code><code>
...
    health: HEALTH_WARN
            1 pool(s) do not have an application enabled
...
    pgs:     128 active+clean
...
</code><code>
# ceph pg dump

# ceph osd lspools
</code>
=== Изменение параметров POOL ===
<code>
ceph osd pool set test-pool1 pg_num 256
ceph osd pool set test-pool1 pgp_num 160  ...  192  ...  224  ...  256

ceph osd pool get test-pool1 size
ceph osd pool get test-pool1 min_size

ceph osd pool set test-pool1 size 2
ceph osd pool set test-pool1 size 3
</code>

==== RBD POOL ====

  * Reliable Autonomic Distributed Object Store = RADOS
  * [[https://docs.ceph.com/en/latest/rbd/rados-rbd-cmds/|BASIC BLOCK DEVICE COMMANDS]]
  * [[https://docs.ceph.com/en/latest/man/8/rbd/|rbd – manage rados block device (RBD) images]]

<code>
# ceph df

# rbd pool init test-pool1

# ceph -s

# rbd create -p test-pool1 rbd1 --size 1G

# rbd list test-pool1

# rbd info test-pool1/rbd1
</code>
<code>
# rbd resize --size 3584M test-pool1/rbd1

если уже использовался, может понадобиться:

node3# bash -c '
systemctl restart tgt
ssh node4 systemctl restart tgt
ssh node5 systemctl restart tgt
'
</code>

==== TGT-RBD ====

  * [[https://habr.com/ru/post/315646/|Кластер хранилища Ceph на VMWare за 10 минут]]
<code>
nodeN# apt install tgt-rbd

nodeN# cat /etc/tgt/conf.d/ceph.conf
</code><code>
<target virtual-ceph-test-pool1-rbd1:iscsi>
    driver iscsi
    bs-type rbd
    backing-store test-pool1/rbd1
    initiator-address ALL
</target>
</code><code>
scp /etc/tgt/conf.d/ceph.conf node4:/etc/tgt/conf.d/
scp /etc/tgt/conf.d/ceph.conf node5:/etc/tgt/conf.d/
</code><code>
node3# bash -c '
systemctl restart tgt
ssh node4 systemctl restart tgt
ssh node5 systemctl restart tgt
'
</code><code>
systemctl status tgt

===
https://ceph.io/planet/find-the-osd-location/

# ceph osd find 0


ceph health detail


===


===
https://medium.com/flant-com/3-cases-from-our-ceph-experience-c46efea0e527
Case #1. Gracefully taking out OSDs from the Ceph cluster
...
ceph tell 'osd.*' injectargs --osd-max-backfills=1
...
</code>
Методические материалы Лохтурова Вячеслава

User Tools

Site Tools

Page Tools
