Wido den Hollander – Page 30

SQL connection error after upgrade to CloudStack 4.3.0

I just upgraded a small cluster of mine from CloudStack 4.2.1 to 4.3.0 and after installing the packages on my Ubuntu system the management server wouldn’t start due to a SQL error:

2014-03-25 20:52:13,643 INFO  [c.c.u.d.T.Transaction] (main:null) Is Data Base High Availiability enabled? Ans : false
2014-03-25 20:52:13,736 ERROR [c.c.u.d.Merovingian2] (main:null) Unable to get a new db connection
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true
	at java.sql.DriverManager.getConnection(DriverManager.java:635)
	at java.sql.DriverManager.getConnection(DriverManager.java:195)

I quickly remembered a licensing issue around JDBC which delayed 4.3.0 and I was right. The management server was missing the right JAR/package for the SQL connection.

A quick apt-get install fixed it:

$ sudo apt-get install libmysql-java

This should have been a dependency of the ‘cloudstack-management’ package, but that somehow slipped through. I already applied a patch in the master branch and I’ll make sure it gets into 4.3.1 and 4.4.0.

So if you are running Ubuntu and are upgrading to CloudStack 4.3.0 and run into this issue, simply install the package and it’s fixed.

Safely backing up your Ceph monitors

So you might wonder: Why do I need to make a backup of my Ceph monitors? I have multiple monitors.

That’s true, but would you run into the very unfortunate situation where you loose all you monitors, you loose all your data. The monitors contain very important metadata (pgmap, osdmap, crushmap) to run your cluster. If you loose that metadata, you practially loose all your data.

Ceph’s monitors use Google’s LevelDB to store all their information. When looking at a monitors data directory you’ll see something like this:

[root@mon1:/var/lib/ceph/mon/ceph-alpha]$ ls -alR
.:
total 16
drwxr-xr-x 3 root root 4096 Sep 23  2013 .
drwxr-xr-x 3 root root 4096 Mar 24 11:04 ..
-rw-r--r-- 1 root root   55 Sep 23  2013 keyring
drwxr-xr-x 2 root root 4096 Mar 25 14:09 store.db

./store.db:
total 236172
drwxr-xr-x 2 root root    4096 Mar 25 14:09 .
drwxr-xr-x 3 root root    4096 Sep 23  2013 ..
-rw-r--r-- 1 root root 2116576 Mar  1 01:35 1400870.sst
-rw-r--r-- 1 root root 2111248 Mar  1 01:40 1400992.sst
...
...
-rw-r--r-- 1 root root 1149227 Mar 25 14:09 2026520.sst
-rw-r--r-- 1 root root      17 Mar 25 04:34 CURRENT
-rw-r--r-- 1 root root       0 Sep 23  2013 LOCK
-rw-r--r-- 1 root root 2196679 Mar 25 14:09 LOG
-rw-r--r-- 1 root root 3829307 Mar 25 04:33 LOG.old
-rw-r--r-- 1 root root  983040 Mar 25 14:09 MANIFEST-2016290
[root@mon1:/var/lib/ceph/mon/ceph-alpha]$

So it’s very tempting to simply run your favorite backup tool and back up this directory. Usually it’s less then 500MB, so it’s very simple to do so.

It’s however not a wise idea to do so, since you have to be sure the LevelDB database is in a consistent state before backing it up.

In a production cluster you will probably have a least three monitors, so stopping a monitor is not a big problem.

A simple backup solution would be:

service ceph stop mon
tar czf /var/backups/ceph-mon-backup_$(date +'%a').tar.gz /var/lib/ceph/mon
service ceph start mon

Put that in a Shell script and have CRON run it every 24 hours. Make sure not all three monitors create their backup at the same time, but this works just fine.

You now have a tarball which you can upload to any offsite location to make sure your monitors are safe.

Another solution would be to run the monitors on a ZFS on Linux filesystem and use ZFS’s snapshot functionalities. But you can’t be 100% sure that your LevelDB database is in a consistent state at that point.

The safest solution at this moment is to fully stop the monitor, create the backup and start the monitor again. Just make sure you don’t stop all monitors at the same time.

Middelburg to Wales and back

I play paintball as a sport/hobby and this weekend there is a event in Wales which I’m going to attend.

As a Model S owner I obviously wanted to go there with my Tesla Model S, but it’s a 800km single-trip, so I needed to charge somewhere. After some searching I found the Ecotricity network in the UK. A network with 50kW chargers along most of the main Motorways in the UK.

The ecotricity chargers have a 50kW CHAdeMO connector (DC) and a 43kW Type 2 (AC) connector. Since Tesla hasn’t released their CHAdeMO adapter yet I’ll have to charge using the AC connector. The onboard chargers of the Model S are capable of 3x32A (22kW), but they are temporarily limited to 3x26A (18kW). So we could have charged with 50kW instead of 18kW if we had the adapter already.

This morning I left my home in Middelburg and headed to the Calais (France) for the ferry crossing to the UK.

At Calais Port

The first stop was at Calais. No charging facilities there. Thus far we (driving with a friend of mine) had driven 217km with a total usage of 47.3kWh. That comes down to 217Wh/km. It was quite windy and cold (5C) this morning, so that explains the higher usage. We continuesly drove with a speed of 100km/h.

After the ferry to Dover it was time to go to the first charger. The initial idea was to go to a charger along the M20, but over the last few weeks the Ecotricity website said it was online, offline, online, offline. At the moment it’s marked as offline, so we went to a charger along the M2 instead.

From Dover it was just 60km to the charger. We still had 136km of range left, so without any problems we drove to the charger.

Charger at M2

There we are, charging at the Ecotricity charger! Worked just fine. Hit the buttons, swipe the card, choose AC and plug it in!

We are now happily charging at 26A@238V:

charging-m2-screenshot

As you can see, the chargers goes up to 62A, but it’s the Model S which is limited to 26A right now.

The next charging station is underneath a Windmill near Reading. Just 160km from here. So a bit more charging and we are on our way!

Towards the Windmill
After a charge it was time to head to the Green Business Park in Reading for a charge underneath a windmill.

The charging station wasn’t hard to find, simply look for the windmill. There was a Nissean Leaf already charging, but the station has two outlets, so we could plug in and charge.

leaf-and-model-s-windmill

At the windmill there is a nice park with some information signs about the windmill:

2MW is a huge amount of power, should be enough to give me a full charge 🙂

We stayed here for about 30 minutes. Besides the park there isn’t that much to do at the windmill, so we left towards the next services area to get dinner and a longer charge.

At the next service area we still had 160km or range left.

We’re taking a defensive strategy this trip, we don’t want to run low on power, so we do multiple shorter charges instead of 2 long ones. You never know what happens during the trip!

The voltage is 247V. So with 3x26A we are looking at a 19.2kW charge! The more kiloWatts that go into the car, the shorter the charge is.

Made it to Wales
A day late I’m writing to be able to tell you I made it to Wales! We charged 73km before our destination to 200km of range so we got at the location with enough charge.

We weren’t sure if we could charge at the Bluestone Park Wales, but they allowed us to plug into a 13A socket which was enough to fully charge the car in 15 hours. We are full again and can begin our trip back to the Netherlands on Monday.

In total we drove 740km with an average of 199Wh/km, so we used 148kWh of energy to get here.

Heading back to Dover
On Sunday evening we left Wales to drive 140km to our first Ecotricity charger to get something to eat and a short charge.

The paintball event was finished around 16:00, so we left Wales around 17:00.

After some “diner” at the Burger King we headed down the M4 for a overnight stay in the Travelodge motel in Chieveley which also has a Ecotricity charger.

We fully charged the car and had a good night of sleep. Got up again at 07:00 and at 08:00 we were on our way again to our last charger before the ferry at Dover.

After this it’s just 280km to our home, so no more need for charging.

Last stretch home
From the last charging station at the M2 it was another 290km to get home. We charged the battery to about 320km before leaving for the ferry.

In Calais it was quite windy, so our energy consumption went up pretty steep. In the end we reached Middelburg with just 17km left in the battery.

Trip log
I kept a full log of the trip and energy consumption which is available here.

In total we used 302kWh for almost 1500km.

The Ecotricity network
One word: Awesome!

I seriously love it that they are solving the chicken-egg problem by simply putting those chargers out there now. The charging is for free right now, but I would have paid if required.

If this network keeps expanding and Tesla delivers the CHAdeMO adapter it will only get better! My next visit to the UK will probably be fully electric again thanks to Ecotricity!

In the end this is what my Model S shows on his charging map: