Starting with Ceph version Jewel (10.2.X) all daemons (MON and OSD) will run under the privileged user ceph. Prior to Jewel daemons were running under root which is a potential security issue.
This means data has to change ownership before a daemon running the Jewel code can run.
Chown data
As the Release Notes state you will have to chown all your data to ceph:ceph in /var/lib/ceph.
chown -R ceph:ceph /var/lib/ceph
On a system with multiple OSDs this might take a lot of time, using GNU Parallel you can save yourself a lot of time.
Static UID
The ceph User and Group have been assigned static UID and GIDs in the major distributions:
- Fedora/CentOS/RHEL: 167:167
- Debian/Ubuntu: 64045/64045
Chown in parallel
Using these commands you can chown the data in /var/lib/ceph much faster.
WARNING: Make sure the OSDs are stopped on the system before you continue!
Now you can run these commands (Ubuntu in this case):
find /var/lib/ceph/osd -maxdepth 1 -mindepth 1 -type d|parallel chown -R 64045:64045 chown 64045:64045 /var/lib/ceph chown 64045:64045 /var/lib/ceph/* chown 64045:64045 /var/lib/ceph/bootstrap-*/*
The first command will take the longest. I tested it on a system with 24 OSDs all containing about 800GB of data. That took roughly 20 minutes.