Shut down the cluster for maintenance
All clients must stop operation on the cluster (obviously…).
The following flags must be set for the OSD daemons:
noout: OSDs will not automatically be marked out after the configured interval
nobackfill: Backfilling of PGs is suspended
norecover: Recovery of PGs is suspended
norebalance: OSD will choose not to backfill unless PG is also degraded
nodown: OSD failure reports are being ignored, such that the monitors will not mark OSDs down
pause: Pauses reads and writes
ceph osd set noout
ceph osd set nobackfill
ceph osd set norecover
ceph osd set norebalance
ceph osd set nodown
ceph osd set pause
Bootstrap Ceph daemons with systemd and containers
ceph config set global public_network 192.168.122.0/24
ceph config set global cluster_network 10.4.0.0/24
The crash module collects information about daemon crashdumps and stores it in the Ceph cluster for later analysis.
root@cephadmin:/etc/ceph# ceph crash ls
INFO:cephadm:Using recent ceph image ceph/ceph:v15
ID ENTITY NEW
2020-06-16T20:54:01.009899Z_f4ad9af2-fb8a-4844-b892-c59c53062ff8 mgr.cephadmin.ciozlx *
root@cephadmin:/etc/ceph# ceph crash archive 2020-06-16T20:54:01.009899Z_f4ad9af2-fb8a-4844-b892-c59c53062ff8
ceph log last cephadm
ceph orch ps
ceph orch daemon stop mgr.cephhost01.urlllo
ceph orch daemon restart mgr.cephadmin.ciozlx
root@cephadmin:~# ceph orch stop osd
stop osd.0 from host 'cephhost01'
stop osd.3 from host 'cephhost01'
stop osd.4 from host 'cephhost02'
stop osd.1 from host 'cephhost02'
stop osd.2 from host 'cephhost03'
stop osd.5 from host 'cephhost03'
root@cephadmin:~# ceph orch stop mon
stop mon.cephadmin from host 'cephadmin'
stop mon.cephhost01 from host 'cephhost01'
stop mon.cephhost02 from host 'cephhost02'
stop mon.cephhost03 from host 'cephhost03'
Every node must:
- run docker for cephadm to work
- have either DNS resolution or a
/etc/hostsentry of all other nodes
- have all other nodes in their
- have the cluster SSH key in their
Ceph uses docker for its daemons and the containers have names like
ceph-55f960fa-af0f-11ea-987f-09d125b534ca-osd.0 which contains the
fsid is a unique identifier for the cluster, and stands for File System ID from the days when the Ceph Storage Cluster was principally for the Ceph Filesystem. Ceph now supports native interfaces, block devices, and object storage gateway interfaces too, so
fsid is a bit of a misnomer.
(docker-container)@container / $ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
The debian package with