Author: Wido den Hollander

Ceph distributed storage with CloudStack

As we are nearing the CloudStack 4.0 release I figured it was time I’d write something about the Ceph integration in CloudStack 4.0

In the beginning of this year we (my company) decided we wanted to use CloudStack for our cloud product, but we also wanted to use Ceph for the storage. CloudStack lacked the support for Ceph, so I decided I’d implement that.

Fast forward 4 months, a long flight to California, becoming a committer and PPMC member of CloudStack, various patches for libvirt(-java) and here we are, 25 September 2012!

RBD, the RADOS Block Device from Ceph enables you to stripe disks for (virtual) machines across your Ceph cluster. This not only gives high performance, it gives you virtually unlimited scalability (without downtime!) and redundancy. Something your NetApp, EMC or EqualLogic SAN can’t give you.

Although I’m a very big fan of Nexenta (use it a lot) it also has it’s limitations. A SAS environment won’t keep scaling for ever and SAS is expensive! Yes, ZFS is truly awesome, but you can’t compare it to the distributed powers Ceph has.

The current implementation of RBD in CloudStack is for Primary Storage only, but that’s mainly what you want, it has a couple of limitations though:

  • You still need either NFS or Local Storage for your System VMs
  • Snapshotting isn’t enabled (see below!)
  • It only works with KVM (Using RBD in Qemu)

If you are happy with that you’ll able to allocate hundreds of TB’s to your CloudStack cluster like it was nothing.

What do you need to use RBD for Primary Storage?

  • CloudStack 4.0 (RC2 is out now)
  • Hypervisors with Ubuntu 12.04.1
  • librbd and librados on your hypervisors
  • Libvirt 0.10.0 (Needs manual installation)
  • Qemu compiled with RBD enabled

There is no need for special configuration on your Hypervisor, that’s all controlled by the Management Server. I’d however recommend that you test the Ceph connectivity first:

rbd -m <monitor address> –user <cephx id> –key <cephx key> ls

If that works you can go ahead and add the RBD Primary Storage pool to your CloudStack cluster. It should be there when adding a new storage pool.

It behaves like any storage pool in CloudStack, except the fact that it is running on the next generation of storage 🙂

About the snapshots, this will be implemented in a later version, probably 4.2. It mainly has to do with the way how CloudStack currently handles snapshots. A major overhaul of the storage code is planned and as part of that I’ll implement snapshotting.

Testing is needed! So if you have the time, please test and report back!

You can find me on the Ceph and CloudStack IRC channels and mailinglists, feel free to contact me. Remember that I’m in GMT +2 (Netherlands).

Installing the socket of my Open EVSE

While the shed in my backyard is still under constructing I started installing the Type 2 socket on the outer wall on the side of my parking spot.

For my EVSE I’m using the Open EVSE project. Although we use a different connector in Europe, the signaling is the same as with the J1772 connector. One other advantage is that the European connector has support for 3-phase power.

My shed has a outer and inner wall with the main supports in between them, I’m trying to run most cabling inside the wall for the aesthetics. For the socket it was also the easiest way.

A couple of pictures of what I did so far (still took me 5 hours though! Those stiff cables don’t make it any easier!).

 

 

 

This is the Type 2 socket for installation in a wall. This picture is without the interlocking actuator. The actuator is used for locking the connector in the socket while charging. It’s to prevent you from taking out a connector which might be carrying 22kW of power, that would give some fireworks!

 

 

 

 

 

 

 

This is the socket with the actuator installed on top of it. It’s a very simple 12V motor which pushes a pin into the socket and locking the connector in place.

I haven’t found a way yet to control this with the Open EVSE project, but I’m positive I’ll find a way to do so.

Locking is done by putting 12V on it for 300ms and unlocking is done by simply reversing the poles. Any suggestions how to do this with Open EVSE are welcome!

 

 

 

This now had to be installed in the outer wall of my shed. This wall is 11mm thick while the connector is build for 9mm at max. It took some power tools to get about 4mm off that wall!

 

 

 

This is how it looks on the outer wall after installation. You can see the actual socket still lying there, but the outer cap has been installed.

The socket itself is very well build by Mennekes. Rubbers everywhere to prevent water from coming in, but it also has water drainage output on the bottom. Should somebody plug in a connector full of water the excess water can flow out of the socket.

It’s a pretty expensive socket (EUR 300,00) but they did their job of creating a descent one!

 

 

 

This picture shows the installed socket from the back with the main power lines connected.

WARNING: These cables are NOT energized! The power flow is controlled by the EVSE with a relais. Take note of that should you consider installing your own EVSE!

The PP (Proximity Pilot) and CP (Control Pilot) pins are not connected yet, you however see the CAT6 cable already there. I’m using that cable for the CP and PP pins as well as for controlling the actuator.

The power cable is a 5G6 cable. This means 5 wires of 6mm2 each. My goal is to have this EVSE installation be able to deliver 3-phase 32A (22kW). 4mm2 cabling would maybe have done the job, but safety first! It also means less voltage drop, so more kW to the car!

The flexible PVC hose at the bottom is water drainage from the socket, should water enter the socket it can flow out of the socket through there.

 

 

 

 

This is how it all looks after installation with the actuator installed.

One PVC pipe carries the the main power cable while the other carries the CAT6 cable for controlling the actuator and connecting the PP and CP pin.

The pipes run through the structure towards the main power panel (yet to be installed) where I’ll also install the Open EVSE.

Between the outer and inner wall there will be insulation, but I have to prevent this from making contact with the socket. I still have build something for that around the socket, I want to make sure the insulation doesn’t catch fire for some reason. (Although I don’t know if it actually burns).

It’s better to be safe then sorry in this case. 22kW of power will flow through here, that is something not to be taken lightly, that is some serious amount of power!

 

 

 

If you want to order this socket yourself contact EV-Box (Netherlands), they can help you with that.

These are the parts I ordered (Mennekes part numbers):

  • 30012: Fixing ring with hinged lid IP44
  • 31016: Type 2 socket 32A with actuator
  • 30019: Actuator connector with 3 wires of 1m each

When I finish my EVSE I’ll post a full list of components I used.

Now it’s back to my shed for doing some more installation of other electronics like lighting.

Quassel IRC, never miss anything on IRC!

I was one of those guys who had irssi running inside a screen on a remote Linux box somewhere. It works just fine, but I always forgot to open the SSH session so I missed a lot of IRC conversations. Private messages were a problem as well, most of the times it was a couple of days later before I noticed somebody had actually sent me a PM…

It was time to change my IRC client, with the preference to always be online.

A short search lead me to the website of Quassel IRC, a distributed IRC server/client. Exactly what I was looking for! You just install the “core” on a remote Linux box and use the Linux, Windows, Mac OSX or Android client to participate on IRC.

The core has been running on a Ubuntu 10.04 machine for about one week now and it works like a charm. My IRC conversations are secured by SSL and I never miss a PM or when somebody tags me!

Integration of the client goes well on Ubuntu 12.04 with Unity, it integrates seamlessly with Unity and notifies me whenever I’m tagged or I receive a PM.

Looking for me on IRC? Find me on OFTC @ wido where I hang out in #ceph. Or find me on Freenode @ widodh in #cloudstack