Placement Groups with Ceph Luminous stay in activating state

Placement Groups stuck in activating

When migrating from FileStore with BlueStore with Ceph Luminuous you might run into the problem that certain Placement Groups stay stuck in the activating state.

44    activating+undersized+degraded+remapped

PG Overdose

This is a side-effect of the new PG overdose protection in Ceph Luminous.

Too many PGs on your OSDs can cause serious performance or availability problems.

You can see the amount of Placement Groups per OSD using this command:

$ ceph osd df

Increase Max PG per OSD

The default value is a maximum of 200 PGs per OSD and you should stay below that! However, if you are hit by PGs in the activating state you can set this configuration value:

mon_max_pg_per_osd = 500

Then restart the OSDs and MONs which are serving the affected by this.

Usually you shouldn’t run into this, but if this hits you in the middle of a migration or upgrade this might save you.

Building a Lithium battery for my electric scooter


I am not a battery expert nor am I claiming to be one! I got all my information from the internet and I think designed and build my battery correctly.

Always use common sense and do not blindly trust the information I (or anybody else) am/is putting on the internet.

Novox C20

In 2010 I bought a Novox C20 25km/h electric scooter. Novox was (they are gone) Dutch manufacturer of electric scooters. It seems that they (partially) bought the scooters in China and rebranded them to their brand Novox. It seems that many parts of it were actually from the Chinese manufacturer Tysong.

I previously wrote a dutch post and a english post about this scooter.

The model I have is the 25km/h version equipped with a 2.5kW electric motor with 4 12V 40Ah batteries.

Realistically this scooter had a range of about 35 ~ 40km before running out of juice.


In the Netherlands we have two types of scooters. We call them snorfiets (25km/h, blue license plate) or bromfiets (45km/h, yellow plate). My scooter is a snorfiets which means I do not have to wear a helmet.

The primary purpose for the scooter is going to the beach in summer, use it when it’s nice weather and when I want to go somewhere which is a bit to far to go by bicycle.

4x 12V 40Ah

Originally the scooter had 4 12V 40Ah batteries which (in theory) is roughly 1.9kWh of capacity. These were lead acid batteries and each weight 12kg, so the total weight of the batteries was nearly 50kg.

Over the years the range and performance declined until the batteries died completely. After reading about DIY batteries using 18650 Lithium batteries I decided to build my own!

Designing the battery

After I decided to design and build my own battery I went out to gather information. I watched endless hours of videos on YouTube and I also bought a book:

With the information from YouTube, that book and other sources I designed a 14S15P battery.


A 14S15P configuration means 14 Cells in Series and 15 Cells Parallel. The total amount of cells would be: 210 (14*15).

The nominal voltage of the battery would be 50.4V (3.6*14) and the maximum voltage 58.8V (4.2*14).

The existing controller in the scooter would not be able to handle that voltage so I would have to swap the controller. That’s something I’ll explain in a upcoming post.

On AliExpress I searched for 18650 battery cell holders for a 14S5P pack and ordered three sets. Combining them would give me a 14S15P battery pack which would physically fit in my scooter.

Samsung INR18650-35E

During my search for the correct cell I selected the Samsung INR18650-35E cell. This cell has a capacity of 3.500mAh and 210 of them would sum up to a total capacity of 2646Wh (3.6 * 3500 * 210 / 1000) or 2.6kWh for my scooter.

Each cell can deliver 10A of current so 15 of them in parallel would be able to provide 150A of current. With a nominal voltage of 50V that would be 7500W or 7.5kW.

As the motor in my scooter is rated for 2.5kW I would only need 50A of current. 50A or 2.5kW was my design target. Anything above it would be a bonus.

After searching I found the Dutch shop Nkon which has a good reputation and was able to deliver the INR18650-35E for a good price. So I ordered 220 cells (10 spare).

Next to the cells I ordered 15m of 0.15mm Nickel strip to use when spotwelding all the cells together.

Battery Management System

The next important thing was a Battery Management System (BMS). A BMS makes sure the cells stay in a healthy state by doing various things:

  • Protecting them against under -or overvoltage
  • Protecting against a high charge or discharge current
  • Making sure the cells are balanced

There are good and bad BMS out there. At first I thought I’d buy one on AliExpress but after reading on the dutch forum of Tweakers in the topic about electric scooters I purchased a TinyBMS s516 from Energus Power Solutions. It’s not the cheapest, but it does do its job: Protecting my 210 cells!

In addition to the BMS I ordered:

  • USB programming cable
  • Temperature sensors
  • Bluetooth module
  • Balancing Wires


To connect all the cells together you need to use spotwelding as soldering would damage the cells badly due to the heat.

At first I bought a Sunko 709A spotwelder on AliExpress. Short story: A waste of money!

After searching a bit more I found a Arduino based spotwelder made by Malectrics in Germany.

This spotwelder works great with a Bosch 12V 44A 440A car battery. I made a short video and put it on YouTube.

In addition a picture of the spotwelder connected to the battery.

I never used a spotwelder before nor did I build a battery. I started with building a small 3S5P 12V battery to test my welding skills.

Safety First! Remember to wear gloves and eye protection when working with a spotwelder and/or batteries. Also make sure there are no loose tools or wires on your workplace and again, use common sense!

I am now using this battery as a DIY powerbank with a 12V and USB output.

Building it

After I got some practice on building the small battery I just started working on the big battery. Cutting nickel to length and I started to build the blocks.

This took a lot of time, probably over 40 hours as I slowly build my first big battery and still needed to learn.

I ended up stacking two strips of 0.15mm nickel on top of each other to handle my 50A target current.

You can also see the balancing wires of the BMS go to the positive terminal of each series. The manual of the BMS has clear instructions on how to wire them.

From the Nickel I went to 8AWG wire which I connected to a SB50 connector. Using that connector I would connect the battery Plus and Minus to the BMS and the BMS to the controller. The SB50 connector is rated for 50A and thus meets my requirements. I got the connectors from a local supplier.

Using a tape, lexan and some screw the end result (without BMS) looked like this:

And with my TinyBMS connected to it:

The tiny red wires are the balancing wires used by the BMS to monitor the cells in series and balance charge them when needed.

BMS settings

Using a Windows tool you can monitor and configure the BMS using a USB cable (which you have to buy).

There are various settings you can change, but the main values I set were:

  • Low voltage cut-off: 3.25V
  • High voltage cut-off: 4.15V
  • Maximum discharge currect: 75A

Although the cells can range from 3.2V to 4.2V I decided to take a 0.05V safety margin to increase the lifetime of the cells. I might lower the maximum voltage per cell to 4.1V in the future, but that’s something I still have to decide.

Using the BMS tool you can see the voltage of each individual series of cells and see them change while charging the battery.

Charging and Charger

To charge this battery I also needed a new charger as the old charger was not suited for Lithium and the voltage was too low.

I bought a 58.8V/5A charger on AliExpress which I’m using for now. It does seem to do the job just fine for now. The BMS isn’t complaining (yet).

Using XT90 connectors I connect my charger to the BMS although the scooter has a XLR input. But internally I’m connecting all those wires using XT90 connectors.

I got my XT90 connectors at Hobbyking as they also have a local warehouse in the Netherlands and have a good reputation.


The four old batteries had a combined weight of nearly 50kg where the new battery weighs slightly less than 12kg with everything connected.

That’s a weight reduction of 38kg! That will benefit acceleration, handling and range positively.

End result

The end result is a 2,6kWh battery consisting out of 210 cells connected in a 14S15P configuration with a nominal/maximum voltag of 50.4V/58.8V.

My initial calculations and tests tell me that when driving ~30km/h (Yes, I did some tuning!) the motor draws about 20A. That’s about 1000W at 50V.

Since the battery is 2600Wh I should be able to drive for roughly 2 hours and 30 minutes at 30km/h before the battery is empty.

30km for 2.5 hours would mean a range of 75km which is more then enough for my driving habits with this scooter.

With the original batteries I had a range of roughly 40km, but I never tried if it actually got that far. I think I improved the range with nearly 75% with this new battery.

In this picture you see the battery installed in the scooter and hooked up to the controller. A post about the controller and connecting it all will follow later.


The total costs for the cells, BMS, wire and connectors is about EUR 940,00 which can be broken down into:

  • Cells: EUR 650,00
  • BMS: EUR 200,00
  • Wires: EUR 40,00
  • Connectors: EUR 50,00
  • Time: 50 hours!

For me it was a cool project and something I could learn a lot from.

BIG Thanks

There are a few people and sources I have to thank big-time for their information on the internet.

  • jehugarcia on YouTube
  • User on
  • All the other people sharing information!

I wouldn’t have been able to do this without those people. Thanks!


The battery is currently installed in the scooter and I’m still in the process of fine-tuning the controller.

My initial test drives tell me that the performance improved a lot! Acceleration to 30km/h is a lot quicker and I feel the motor has a lot more torque than it had before. This is mainly due to the battery being able to provide the current required to power the motor. The weight reduction can also be felt clearly and also contributes to the improved acceleration.

One of the cool things is that the scooter now has regenerative breaking thanks to the new controller.

A post regarding the controller and connecting it all will follow. Stay tuned!

Quick overview of Ceph version running on OSDs

When checking a Ceph cluster it’s useful to know which versions you OSDs in the cluster are running.

There is a very simple on-line command to do this:

ceph osd metadata|jq '.[].ceph_version'|sort|uniq -c

Running this on a cluster which is currently being upgraded to Jewel to Luminous it shows:

     10 "ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)"
   1670 "ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)"
    426 "ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)"
     66 "ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)"

So 66 OSDs are running Luminous and 2106 OSDs are running Jewel.

Starting with Luminous there is also this command:

ceph features

This shows us all daemon and client versions in the cluster:

    "mon": {
        "group": {
            "features": "0x1ffddff8eea4fffb",
            "release": "luminous",
            "num": 5
    "osd": {
        "group": {
            "features": "0x7fddff8ee84bffb",
            "release": "jewel",
            "num": 426
        "group": {
            "features": "0x1ffddff8eea4fffb",
            "release": "luminous",
            "num": 66
    "client": {
        "group": {
            "features": "0x7fddff8ee84bffb",
            "release": "jewel",
            "num": 357
        "group": {
            "features": "0x1ffddff8eea4fffb",
            "release": "luminous",
            "num": 7

Getting connection URL from (Flask) SQLAlchemy

While working with Flask‘s SQLAlchemy extension I ran into the case where I wanted to obtain the host I was connected to combined with the username and password.

As I didn’t want to parse the URL I provided earlier to SQLAlchemy (and I didn’t have it available at that part of the code) I looked for a method to obtain it from SQLAlchemy.

It turns out this is not (or poorly) documented, but the Engine object has a attribute called url.

>>> db.engine.url

This is actually a sqlalchemy.engine.url.URL object which contains attributes with the connection information:

>>> url = db.engine.url
>>> url.username
>>> url.password
>>> url.database

This was the information I needed and allowed me to use this information elsewhere.

4 years of Tesla Model S ownership

4 years

Last year I wrote a blogpost about 3 years of Model S ownership. Well, it’s a year later, so it’s time to write a post about 4 years of ownership 🙂

Do I still like my Model S? Best car I’ve ever owned!

The numbers

Some numbers about last year:

  • Last year my Model S had driven 141.466km, and another year later the counter is at 187.058km. A total of 45.592km in one year.
  • My average energy consumption still hovers somewhere around 200Wh/km.
  • A full charge (100%) yields 380km and a 90% charge 341km.


The Model S is still my only car and I still take roadtrips with it. This year I took a few roadtrips:

  • 1.400km roadtrip to the UK and back (SuperChargers at the channel tunnel, yay!)
  • 3.500km to Milan/Rome for a summer vacation.
  • 2.000km roadtrip to Munich for Oktoberfest.


A few pictures how my Model S still looks like after 4 years. Personally I can’t find any real signs of wear on the exterior nor interior.

Using a Destination Charger in Tuscany, Italy

Tesla charging stations (22kW) at our new office

Break-in in Rome

While parked in Rome people broke in to the car and stole clothing. And punctured the rear wheels…

Accident in Milan

On the way back from Rome a small truck changed lane and hit us. Nobody got hurt, just some damage to the car.


After 4 years my Model S is still doing just fine! No wear and tear, no severe battery degradation or major failures. The car just works!

I’ll probably keep this Model S until it turns 5 next year and buy a new one. Will I buy a Tesla again? Yes, no doubt!

Using the new dashboard in ceph-mgr

The upcoming Ceph Luminous (12.2.0) release features the new ceph-mgr daemon which has a few default plugins. One of these plugins is a dashboard to give you a graphical overview of your cluster.

Enabling Module

To enable the dashboard you have to enable the module in your /etc/ceph/ceph.conf on all machines running the ceph-mgr daemon. These are usually your Monitors.

Add this to the configuration:

mgr_modules = dashboard

Don’t restart your ceph-mgr daemon yet. More configuration changes have to be made first.

Setting server address and port

A server address and optionally a port have to be configured as a config-key.

By setting the value to :: the dashboard will be available on all IPv4 and IPv6 addresses on port 7000 (default):

ceph config-key put mgr/dashboard/server_addr ::

Restart daemons

Now restart all ceph-mgr daemons on your hosts:

systemctl restart ceph-mgr@

Accessing the dashboard

The default port is 7000, so now go to the IP-Address of the active ceph-mgr and open the see the dashboard.

You can find the active ceph-mgr in the ceph status:

root@alpha:~# ceph -s
    id:     30d838cd-955f-42e5-bddb-5609e1c880f8
    health: HEALTH_OK
    mon: 3 daemons, quorum alpha,bravo,charlie
    mgr: charlie(active), standbys: alpha, bravo
    osd: 3 osds: 3 up, 3 in
    pools:   1 pools, 64 pgs
    objects: 0 objects, 0 bytes
    usage:   3173 MB used, 27243 MB / 30416 MB avail
    pgs:     64 active+clean

In this case charlie is the active mgr which in my case has IPv6 Address 2001:db8::102.

Point your browser to: http://[2001:db8::102]:7000 and you will see the dashboard.

Apache CloudStack and MySQL 5.7

SQL Mode

Starting with MySQL 5.7 the default SQL mode is far more strict then it was before.


This can cause problems for applications which need other SQL modes. Apache CloudStack is one of these applications.

The best thing would be to modify the SQL queries executed by CloudStack, but that’s not that easy.

Changing the mode

Luckily the SQL mode can be changed in either the my.conf or as a session variable.

In the my.cnf one can add:



You should now be able to run a CloudStack management server on MySQL 5.7!


In the future CloudStack should only be using SQL queries which comply with the new more strict SQL mode. In the meantine a issue and Pull Request have been created to track this situation.

Docker containers with IPv6 behind NAT


In production IPv6 should always be used without NAT. Only use IPv6 and NAT for testing purposes. There is no valid reason to use IPv6 with NAT in any production environment.

IPv6 and NAT

IPv6 is designed to remove the need for NAT and that is a very, very good thing. NAT breaks Peer-to-Peer connections and that is exactly what is one of the great things of IPv6. Every device on the internet gets it’s own public IP-Address again.

Docker and IPv6

Support for IPv6 in Docker has been there for a while now. It is disabled by default however. The documentation describes on how to enable it.

I wanted to enable IPv6 on my Docker setup on my laptop running Ubuntu, but as my laptop is a mobile device the IPv6 prefix I have changes when I move to a different location. IPv6 Prefix Delegation isn’t available at every IPv6-enabled location either, so I wanted to figure out if I could enable IPv6 in my Docker setup locally and use NAT to have my containers reach the internet over IPv6.

At home I have IPv6 via ZeelandNet and at the office we have a VDSL connection from XS4All. When I’m on a remote location I enable our OpenVPN tunnel which has IPv6 enabled. This way I always have IPv6 available.

The Docker documentation shows that enabling IPv6 is very easy. I modified the systemd service file of docker and added a fixed IPv6 CIDR:

ExecStart=/usr/bin/dockerd --ipv6 --fixed-cidr-v6="fd00::/64" -H fd://

fd00::/64 is a Site-Local IPv6 subnet (deprecated) which can be safely used.

I then added a NAT rule into ip6tables so that it would NAT for me:

sudo ip6tables -t nat -A POSTROUTING -s fd00::/64 -j MASQUERADE


My Docker containers now get a IPv6 Address as can be seen below:

root@da80cf3d8532:~# ip -6 a
1: lo:  mtu 65536 state UNKNOWN qlen 1
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
15: eth0@if16:  mtu 1500 state UP 
    inet6 fd00::242:ac11:2/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

In this case the address is fd00::242:ac11:2 which as assigned by Docker.

Since my laptop has IPv6 I can now ping from my Docker container.

root@da80cf3d8532:~# ping6 -c 3 -n
PING (2a00:f10:101:0:46e:c2ff:fe00:93): 56 data bytes
64 bytes from 2a00:f10:101:0:46e:c2ff:fe00:93: icmp_seq=0 ttl=61 time=14.368 ms
64 bytes from 2a00:f10:101:0:46e:c2ff:fe00:93: icmp_seq=1 ttl=61 time=16.132 ms
64 bytes from 2a00:f10:101:0:46e:c2ff:fe00:93: icmp_seq=2 ttl=61 time=15.790 ms
--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 14.368/15.430/16.132/0.764 ms

Again, this should ONLY be used for testing purposes. For production IPv6 Prefix Delegation is the route to go down.

Do not use SMR disks with Ceph

Many new disks like the Seagate He8 disks are using a technique called Shingled Magnetic Recording to increase capacity.

As these disks offer a very low price per Gigabyte they seem interesting to use in a Ceph cluster.


Due to the nature of SMR these disks are very, very, very bad when it comes to Random Write performance. Random I/O is something that Ceph does a lot on the backing disks.

This results in disks spiking to 100% utilization very quickly causing all kinds of trouble with OSDS going down and committing suicide.

Do NOT use them

The solution is very simple. Do not use SMR disks in Ceph but stick to the traditional PMR disks in your Ceph cluster.

In the future we might see SMR support in the new BlueStore of Ceph, but at this moment no work has been done, so don’t expect anything soon.

Testing Ceph BlueStore with the Kraken release

Ceph version Kraken (11.2.0) has been released and the Release Notes tell us that the new BlueStore backend for the OSDs is now available.


The current backend for the OSDs is the FileStore which mainly uses the XFS filesystem to store it’s data. To overcome several limitations of XFS and POSIX in general the BlueStore backend was developed.

It will provide more performance (mainly writes), data safety due to checksumming and compression.

Users are encouraged to test BlueStore starting with the Kraken release for non-production and non-critical data sets and report back to the community.

Deploying with BlueStore

To deploy OSDs with BlueStore you can use the ceph-deploy by using the –bluestore flag.

I created a simple test cluster with three machines: alpha, bravo and charlie.

Each machine will be running a ceph-mon and ceph-osd proces.

This is the sequence of ceph-deploy commands I used to deploy the cluster

ceph-deploy new alpha bravo charlie
ceph-deploy mon create alpha bravo charlie

Now, edit the ceph.conf file in the current directory and add:

enable_experimental_unrecoverable_data_corrupting_features = bluestore

With this setting we allow the use of BlueStore and we can now deploy our OSDs:

ceph-deploy --overwrite-conf osd create --bluestore alpha:sdb bravo:sdb charlie:sdb

Running BlueStore

This tiny cluster how runs three OSDs with BlueStore:

root@alpha:~# ceph -s
    cluster c824e460-2f09-4994-8b2f-108aedc52d19
     health HEALTH_OK
     monmap e2: 3 mons at {alpha=[2001:db8::100]:6789/0,bravo=[2001:db8::101]:6789/0,charlie=[2001:db8::102]:6789/0}
            election epoch 14, quorum 0,1,2 alpha,bravo,charlie
        mgr active: charlie standbys: alpha, bravo
     osdmap e14: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v24: 64 pgs, 1 pools, 0 bytes data, 0 objects
            43356 kB used, 30374 MB / 30416 MB avail
                  64 active+clean
root@alpha:~# ceph osd tree
-1 0.02907 root default                                       
-2 0.00969     host alpha                                     
 0 0.00969         osd.0         up  1.00000          1.00000 
-3 0.00969     host bravo                                     
 1 0.00969         osd.1         up  1.00000          1.00000 
-4 0.00969     host charlie                                   
 2 0.00969         osd.2         up  1.00000          1.00000 

On alpha I see that osd.0 only has a small partition for a bit of configuration and the rest is used by BlueStore.

root@alpha:~# df -h /var/lib/ceph/osd/ceph-0
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        97M  5.4M   92M   6% /var/lib/ceph/osd/ceph-0
root@alpha:~# lsblk 
sda      8:0    0    8G  0 disk 
├─sda1   8:1    0  7.5G  0 part /
├─sda2   8:2    0    1K  0 part 
└─sda5   8:5    0  510M  0 part [SWAP]
sdb      8:16   0   10G  0 disk 
├─sdb1   8:17   0  100M  0 part /var/lib/ceph/osd/ceph-0
└─sdb2   8:18   0  9.9G  0 part 
sdc      8:32   0   10G  0 disk 
root@alpha:~# cat /var/lib/ceph/osd/ceph-0/type

The OSDs should work just like OSDs running FileStore, but they should perform better.