Category: Uncategorized

Ceph Monitors are laggy or clock might be skewed

This weekend I got to investigate a Ceph cluster which had issues where the Monitors were constantly performing new elections.

After some investigation on of the three monitors was eating 100% CPU on a single core and kept printing this in the logs:

mon.charlie@2(peon).paxos(paxos updating c 106399655..106400232) lease_expire from mon.0 [2a00:XXX:121:XXX::6789:1]:6789/0 is 2.380296 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)

Digging further I found that the LevelDB store in /var/lib/ceph/mon/X/store.db was 2.5GB in size.

Compact on Start

You can tell the monitor to compact the LevelDB database on start. Add the following to your ceph.conf:

[mon]
mon compact on start = true

Now restart the monitor and it will compact the LevelDB database.

The CPU usage now dropped and the monitors were happy again.

Last SuperCharger in Sweden, going to Norway

Leaving Sweden

In the past two days we visited Stockholm and continued our journey towards Norway.

We both never visited Stockholm before, so yesterday morning we visited Stockholm. We saw Gamla Stan (Old City) and visited the Abba Museum. (My dad is a Abba fan!)

From there on we continued to the Sollentuna and Gävle SuperChargers. We spend the night in Gävle and today we continued towards the Krokum SuperCharger via the Sundsvall.

Nothing really special actually. It started snowing and the temperatures started to drop, it’s -11 Celcius right now and snow is still falling.

Heading to Norway

Tomorrow we are going to drive from Krokum to the Grong SuperCharger in Norway. This is a 250km trip via what it seems some very small towns.

It looks like we have to cross mountains as well, so we’ll do a 100% charge at the Krokum SuperCharger before heading to Grong in Norway.

250km should be doable under the worst conditions possible with a full battery, so it’s looking good. We’ll be in Norway tomorrow!

Energy Consumption

I’m keeping a detailed spreadsheet with the energy consumption during the trip and we are currently at 225Wh/km over 2200km. That is really not bad, I expected higher consumption.

Flash LSI 2308 to IT-mode on a SuperMicro X10SL7-F mainboard

For a new SSD-only Ceph deployment I got my hands on a couple of SuperMicro SYS-1018D-73MTF 1U server with 8 Intel SSDs.

Great machines, but by default SuperMicro ships them with the IR (Integrated RAID) firmware and I wanted to use the IT (JBOD) firmware.

The X10SL7 mainboard has a UEFI bios, so flashing the controller was rather easy.

First, download the latest firmware for the LSI 2308 controller from SuperMicro. At this moment it is the PH19-IT firmware.

Now we have to create a small disk image with a FAT32 filesystem to use as a virtual drive for the IPMI.

I run Ubuntu on my desktops and laptops, so this is how I created that small disk image:

NOTE: You can generate the ISO with the steps below, or download it directly.

mkfs.msdos -C ipmi.iso 2880
sudo mkdir /media/ipmi/
sudo mount -o loop ipmi.iso /media/ipmi/
wget ftp://ftp.supermicro.com/Driver/SAS/LSI/2308/Firmware/IT/PH19-IT.zip
sudo unzip PH19-IT.zip UEFI/* -d /media/ipmi/
sudo umount /media/ipmi/

Now open your IPMI and attach ipmi.iso to your server.

Reboot the machine and hit the F11 key to open a boot menu and start the build in UEFI shell.

When in the UEFI shell enter these commands:

fs0:
cd UEFI
SMC2308T.NSH

At some point it will ask for the 9 remaining digits of the SAS address. Simply enter 000000000 (9 zeros) and you’re fine.

Reboot the system and the controller should now be in IT mode