Author: Wido den Hollander

CloudStack: Zone X is is not ready to launch console proxy yet

As you might know, I’m a committer in the Apache CloudStack project and I work on it on a daily basis.

I have a couple of development setups running and I upgraded one of them (where I do all my Ceph development) from 4.0 to 4.1 (isn’t out yet) and suddenly I got this message in my logs:

Zone 1 is not ready to launch console proxy yet

That log line didn’t tell me that much, so I started digging through the code as of WHY my Zone wasn’t ready, since it was working under 4.0.

It turns out that my global setting “secondary.storage.vm” wasn’t set to true and that caused my KVM zone not to work.

This setting can’t be changed through the Web UI (not sure why) and I had to change it in the database instead. After setting it to “true” my System VMs began to start again and all worked just fine.

It seems this was legacy on my end since the upgrade process doesn’t touch this setting at all. I’m adding some extra debugging to the code to make it a bit more clear as of WHY your zone isn’t ready.

Should you ever encounter this one, verify this setting.

Enhanced RBD support for CloudStack 4.2

About 1 hour ago the new storage subsystem got merged into the master branch of CloudStack. That is wonderful news for all you out there who want to use features like snapshotting with RBD in CloudStack.

In pre-4.2 CloudStack a snapshot was the same as a backup. As soon as you created a snapshot it would also copy that snapshot to the secondary storage. This could not only lead to high network utilization when talking about 1TB RBD volumes, but it also caused problems with the underlying ‘qemu-img’ tool. To make a long story short: Snapshots with RBD just wouldn’t work in CloudStack 4.0 or 4.1 without resorting to dirty hacking. Which we didn’t.

The new storage subsystem separates the backup and snapshot process. Snapshots are handled by the primary storage and they can be copied to the ‘backup storage’ on request. This allows is to use the full snapshot potential of RBD.

I was waiting for the storage subsystem to be merged into the master branch before I could start working on this. About two weeks ago I already wrote a small function spec in CloudStack’s wiki to describe what has to be done.

A couple of choices still have to be made. Traditionally we could do everything through libvirt and ‘qemu-img’, but from what I can see now we’ll run into some trouble. We might have to go through the process of wrapping librbd into a Java library to get it all done, but I’m not completely positive about that. Some patches for libvirt(-java) could probably also do the job, but it would take a lot of time and work to get those upstream and into the repositories. The goal is to have this new RBD code work natively on a Ubuntu 13.04 system.

The expectation is that CloudStack 4.2 will be released mid-July this year, but if you are a daredevil you can always track the master branch and play around with that.

I’ll post updates on the cloudstack-dev list on a regular base about the progress, but you can also watch the master branch and search for commits with ‘RBD’ in the message.

100% CPU utilization on a Cisco 887VA

Some time ago I wrote a blogpost about using a Cisco 887VA router on a XS4All (dutch ISP) connection. The original article is mostly in Dutch, but I’ll keep this one in English since it will probably help users all over the world.

A couple of days ago I got an e-mail from somebody who read my blogpost and asked me if the 887VA was able to handle more then 25Mbit. I never really tested it since I thought the copper-cable in our office wasn’t that good. During a download I logged into the router and saw that the CPU was 94% utilized!

The VDSL line was however online at 38Mbit, so how could this happen? Was the router underpowered?

I couldn’t wrap my head around it. A brand new VDSL router from Cisco couldn’t handle just 25Mbit? Something had to be wrong.

Some searching brought me to the Cisco Support Forums and one of the suggestions was to turn on CEF. A Cisco technology to improve Layer 3 performance.

Logging in to the router showed me indeed that CEF was disabled for both IPv4 and IPv6.

no ip cef
no ipv6 cef

Enabling CEF was simple:

conf t
ip cef
ipv6 cef

And voila! I suddenly was able to use the full 38Mbit with just ~50% CPU load.