Many new disks like the Seagate He8 disks are using a technique called Shingled Magnetic Recording to increase capacity.
As these disks offer a very low price per Gigabyte they seem interesting to use in a Ceph cluster.
Due to the nature of SMR these disks are very, very, very bad when it comes to Random Write performance. Random I/O is something that Ceph does a lot on the backing disks.
This results in disks spiking to 100% utilization very quickly causing all kinds of trouble with OSDS going down and committing suicide.
Do NOT use them
The solution is very simple. Do not use SMR disks in Ceph but stick to the traditional PMR disks in your Ceph cluster.
In the future we might see SMR support in the new BlueStore of Ceph, but at this moment no work has been done, so don’t expect anything soon.