Multipath iSCSI with Ubuntu 10.04 and a EqualLogic SAN

Recently we purchased a EqualLogic PS6000XVS for a KVM environment.

In most of our iSCSI systems we use Multipath I/O, we do this by giving the iSCSI Target two NIC’s and give each NIC a IP-Address in a different subnet over a physically different network. This way we have two seperate I/O path’s to the iSCSI Target.

The EqualLogic does not support this, it only supports one virtual IP in one network, so multipathing gets a bit difficult.

On the Dell Wiki there is configuration howto, so I read that carefully.

The examples are for RedHat, but we are using Ubuntu, but that should not make a big difference, but it did….

Our storage network is in the subnet where the virtual IP of the EqualLogic is You should know, this is a virtual IP, in total we have three PS6000 nodes, which do some magic by responding with a different MAC Address for towards each client.

One of our clients has the following configuration for the storage connectivity:

eth0      Link encap:Ethernet  HWaddr 14:FE:B5:C6:62:E0  
          inet addr:  Bcast:  Mask:
          RX packets:27263332 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25323692 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:24569609290 (22.8 GiB)  TX bytes:132201626154 (123.1 GiB)
          Interrupt:170 Memory:e6000000-e6012800 

eth1      Link encap:Ethernet  HWaddr 14:FE:B5:C6:62:E2  
          inet addr:  Bcast:  Mask:
          RX packets:27246580 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25335109 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:24549507448 (22.8 GiB)  TX bytes:132201622012 (123.1 GiB)
          Interrupt:178 Memory:e8000000-e8012800

It took some work to get this working. Bot NIC’s are connected to the same subnet, through different switches though.

The first problem you will run into is the ARP flux problem of Linux, I’m not going to write to much about this, on the internet there is more then enough information written about this topic.

I ended up with this configuration:

auto eth0
iface eth0 inet static
        post-up sysctl -w net.ipv4.conf.eth0.rp_filter=0
        post-up sysctl -w net.ipv4.conf.eth0.arp_ignore=1
        post-up sysctl -w net.ipv4.conf.eth0.arp_announce=2

auto eth2
iface eth2 inet static
        post-up sysctl -w net.ipv4.conf.eth2.rp_filter=0
        post-up sysctl -w net.ipv4.conf.eth2.arp_ignore=1
        post-up sysctl -w net.ipv4.conf.eth2.arp_announce=2

For Open-iSCSI I created two interfaces called ieth0 and ieth1 and routed my iSCSI traffic through them. How you can do this can be found at the Dell wiki.

But it did not work! I was able to ping the EqualLogic over eth0, but not over eth1. If I brought down eth0, it would work over eth1, but not vise versa. It took me a while to find it, but it’s due to a default setting in Ubuntu, done in /etc/sysctl.d/10-network-security.conf, this enables rp_filter (Reverse Path Filtering) by default, so I modified that file

# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks.

And voila! My iSCSI multipathing started to work! My multipath shows:

[size=1.0T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 13:0:0:0 sdk 8:160 [active][ready]
 \_ 14:0:0:0 sdj 8:144 [active][ready]
eql-0-8a0906-4f2b9e409-2b800184d024d9db_c () dm-4 EQLOGIC,100E-00
[size=2.0T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 6:0:0:0 sdg 8:96  [active][ready]
 \_ 11:0:0:0 sdf 8:80  [active][ready]

This should work under Ubuntu 10.04. Took me some time to figure it all out, but now it’s working like a charm. But still, I prefer multipathing over two different VLAN’s and subnets, really odd that the EqualLogic does not support this!

Printing over IPv6 to a Canon MP495

Yesterday I posted that my new Canon Pixma MP495 also supports IPv6.

I had to test if I could print over IPv6, so I switched from IPv4 to IPv6 in the printer configuration (Note: You have to select IPv4 or IPv6, there is no Dual-Stack!). Before doing so I wrote down the MAC Address of the printer, I would need that to find it on my network, since the printer would get a IP from the Router Announcements my Linux router send out.

After turning on IPv6 the printer got his address within a few seconds and I was able to browse through the webinterface with Firefox.

Now I wanted to print over IPv6, the first thing I checked was if CUPS under Ubuntu 10.04 supported IPv6. It seems that CUPS supports IPv6 since version 1.2 and Ubuntu 10.04 is shipped with CUPS 1.4, so that was OK.

Then I created a DNS record for my printer, I pointed a AAAA-record to my printer, just so I dind’t have to type the address all the time. And DNS has been developed for NOT typing IP-Addresses, isn’t it?

Now I had to configure CUPS to print over IPv6, my goal was to do this via the GUI and not use any command-line stuff, that was even easier that I thought.

Adding the printer can be done in a few simple steps:

  • Go to System -> Administration -> Printing
  • Add a printer
  • Choose “Network Printer”
  • Choose LPD/LPR Host or Printer
  • In the host field, put the DNS record to your printer (or add the printer in /etc/hosts)
  • Then choose “Probe”
  • At “Queue”, select “ps”
  • Click on “Forward”
  • Choose “Provide a PPD file”
  • Download this PPD file and choose it as the driver
  • Add the printer!

Your printer settings should then look like:

Your are all set, the printer should work over IPv6 after this steps. Happy printing over IPv6!

Bonding, VLAN and bridging under Ubuntu 10.04

The last few weeks I spend a lot of time upgrading Ubuntu 9.10 systems to 10.04, these systems are SuperMicro blade systems with 2 NIC’s per blade.

By using bonding (active-backup) we combine eth0 and eth1 to bond0. On top of the bond we use 8021q VLAN’s, so we have devices like bond0.100, bond0.303, etc, etc.

Those devices then are used to create bridges like vlanbr100 and vlanbr303 to give our KVM Virtual Machines access to our network.

This would result in a setup like:

eth0 -> |
        | -> bond0 -> bond0.100 -> vlanbr100
eth1 -> |          -> bond0.303 -> vlanbr303  

Under Ubuntu 9.10 and before this setup worked fine, but under Ubuntu 10.04 we noticed that the network inside the virtual machine wouldn’t work that well. The ARP reply (is-at) would be dropped at the bridge and didn’t get transferred to the Virtual Machine.

If I’d set the arp manually inside the VM, everything started to work, but ofcourse, that was not the way it was meant to be.

After hours of searching I found a Debian bugreport, that was exactly my problem!

It seems that Ubuntu’s ifenslave-2.6 package (1.10-14) under 10.04 has exactly the same bug. Backporting the ifenslave package from 10.10 (1.10-15) fixed everything for me, my virtual machines would start to work again.

I created a bug report for this at Ubuntu, hopefully they will fix it in 10.04 rather quickly.

For now, if you have the same problem, just backport the ifenslave package from 10.10 to 10.04

Canon MP495 supports IPv6!

While we are nearing the end of the IPv4 pool, a lot of consumer electronics (even Enterprise routers) do not support IPv6.

Today I bought a new printer to use at home. It had to be a printer which would work over WiFi, after some time at the local store I choose the Canon Pixma MP495, a simple printer, just what I needed.

After configuring it (which I had to do via Windows), I browsed to the IP of the printer and saw that it supported IPv6! (Even IPsec) Wow, that is something you don’t see often.

Haven’t tested it with my Ubuntu 10.04 laptop yet, but it is nice to see manufacturers start implementing IPv6 in ordinary products!

Make-kpkg fails due to a ‘+’/plus appended to the kernel version

This weekend i tried to compile a patched 2.6.35 kernel and wanted to install this on multiple machines. But it kept failing when the .deb files were created.

make-kpkg kept complaining that the kernel version, 2.6.35-rbd+ was not in the “control info”.

After some searching i found out this was due to CONFIG_LOCALVERSION_AUTO not being set.

Setting this to ‘y’ solved it for me, now my kernel compiled and got packed the way it should.

Nokia, shame on you..

Last november i bought a Noka N900, the reason for my purchase was because it ran Maemo which gave me the freedom to do with my phone whatever i wanted to (with some limitations ofcourse 😉 ).

I’ve been using my N900 intensive since then, using it the whole day to keep up to date with my work, manage some servers through SSH and keep in touch with Jabber Instant Messaging.

Really, the N900 is a great phone and even better when you like Linux as much as i do.

But there was a downside…. A few weeks ago (half April) my phone started to complain that my memory was full, so i couldn’t receive any more IM’s or SMS’s. When i opened a terminal and did a “df -h” it showed me that i was only using 23% of the internal memory. A reboot solved it, so i guessed it was a bug somewhere.

After a few days the phone started to reboot itself and kept complaining whenever i wanted to save something to the internal MMC memory, so i opened up a terminal and typed “dmesg”, well, then i knew what was wrong: The internal MMC memory was broken.

So i started to search the web and i found this thread:

It seemed i wasn’t the only one! I contacted Nokia (since i bought the phone from them directly) and they told me i could return it to “BelCompany” (A Dutch phone store) which would then return it for me to the Nokia Service Center.

And so they did.. After a week i got my phone back with the message: Software reprogrammed

Well, i didn’t think that would fix it and it didn’t. After 5 minutes of usage i opened a terminal again and i saw the kernel spitting out errors again about mmcblk0p2. So i went back to “BelCompany” and they returned it again to Nokia.

Guess what? I got my phone back again with the same message and it is still broken!

So i got a broken phone back from the repair center for the second time! I brought the phone back to “BelCompany” again and they urged Nokia to replace my phone with a new N900.

Nokia, shame on you! When i returned my phone to the Nokia service center i attached 3 A4’s with error message from the phone and ways you could reproduce the problem, but in stead you just reprogrammed the MMC and sent it back to me again. Really not the way to treat people when they pay EUR 599,00 for a phone!

To be continued..

How to turn off the Journal with EXT4

For a specific system i wanted maximum performance, so i tried of turning the journaling on my ext4 device.

It took me some time to find out how, so here is a small howto:

1. Unmount your EXT4 filesystem
2. tune2fs -O ^has_journal /dev/sdX
3. mount your filesystem again.

And voila! You have an ext4 filesystem without a journal.

Note: This works if you have a kernel newer then 2.6.28!

ActiveSync (Z-Push) sync with Zarafa and Nokia N900

Recently i bought a Nokia N900 and started right away with configuring my Exchange (Zarafa) account, but i didn’t work.

Syncing a mobile phone works via Microsoft’s ActiveSync, in Zarafa’s case this is implemented via Z-Push.

After making a post on the Z-Push forum i got in contact with Andreas (a Z-Push) and he found out what the problem was. Z-Push implements Exchange 2003 and the Nokia N900 requires Exchange 2007.

Andreas started working on some fixes and after a few days of mailing he made an alpha version which also support the Nokia N900.

At the moment i am running the Alpha in conjunction with my Zarafa envirioment and it works fine, so for all the other N900 users, help is on its way!

Protecting yourself against a DDoS with varnish

Today we received another DDoS attack on of our clusters.

99% of the DDoS attacks we receive are floods on port 80, not really Syn Flood attacks, but just a large stream of garbage on port 80 from thousands of hosts. This results in Apache just spawning processes and eventually locking up.

About two weeks ago i read about Varnish, this high performance HTTP proxy also seems to be a real life-saver when it comes to DDoS attacks.

Since we were really out of options i gave Varnish a go and installed it on our webservers. I configured Apache to listen on and Varnish to listen on the public IP.

After doing this on 10 webservers i sat back and watched everything getting back to life!

This is because Varnish only forwards a HTTP request to the backend (Apache in this case) when it is complete, so this protects Apache from getting al the garbage and spawning useless childs.

So if you ever get a DDoS (and i really hope not!), keep Varnish in mind for saving yourself!

At the moment Varnish really looks like a permanent solution in our hosting envirioment, with some special Apache modules you can make it a transparant proxy, see: mod_extract_forwarded2

rtc timer problems when running Asterisk in KVM

For our VOIP we use Asterisk ( ). Recently we migrated the server from a physical machine to a KVM virtual machine under Ubuntu 9.04

When searching around i found some problems around running Asterisk in a VM, but there were also some succes stories.

I kept getting the message:
rtc: lost some interrupts at 1024Hz

I was able to solve this by disabling APIC for this virtual machine, i kept ACPI enabled.