How to turn off the Journal with EXT4

For a specific system i wanted maximum performance, so i tried of turning the journaling on my ext4 device.

It took me some time to find out how, so here is a small howto:

1. Unmount your EXT4 filesystem
2. tune2fs -O ^has_journal /dev/sdX
3. mount your filesystem again.

And voila! You have an ext4 filesystem without a journal.

Note: This works if you have a kernel newer then 2.6.28!

ActiveSync (Z-Push) sync with Zarafa and Nokia N900

Recently i bought a Nokia N900 and started right away with configuring my Exchange (Zarafa) account, but i didn’t work.

Syncing a mobile phone works via Microsoft’s ActiveSync, in Zarafa’s case this is implemented via Z-Push.

After making a post on the Z-Push forum i got in contact with Andreas (a Z-Push) and he found out what the problem was. Z-Push implements Exchange 2003 and the Nokia N900 requires Exchange 2007.

Andreas started working on some fixes and after a few days of mailing he made an alpha version which also support the Nokia N900.

At the moment i am running the Alpha in conjunction with my Zarafa envirioment and it works fine, so for all the other N900 users, help is on its way!

Protecting yourself against a DDoS with varnish

Today we received another DDoS attack on of our clusters.

99% of the DDoS attacks we receive are floods on port 80, not really Syn Flood attacks, but just a large stream of garbage on port 80 from thousands of hosts. This results in Apache just spawning processes and eventually locking up.

About two weeks ago i read about Varnish, this high performance HTTP proxy also seems to be a real life-saver when it comes to DDoS attacks.

Since we were really out of options i gave Varnish a go and installed it on our webservers. I configured Apache to listen on 127.0.0.1:80 and Varnish to listen on the public IP.

After doing this on 10 webservers i sat back and watched everything getting back to life!

This is because Varnish only forwards a HTTP request to the backend (Apache in this case) when it is complete, so this protects Apache from getting al the garbage and spawning useless childs.

So if you ever get a DDoS (and i really hope not!), keep Varnish in mind for saving yourself!

At the moment Varnish really looks like a permanent solution in our hosting envirioment, with some special Apache modules you can make it a transparant proxy, see: mod_extract_forwarded2

rtc timer problems when running Asterisk in KVM

For our VOIP we use Asterisk ( www.sipcat.com ). Recently we migrated the server from a physical machine to a KVM virtual machine under Ubuntu 9.04

When searching around i found some problems around running Asterisk in a VM, but there were also some succes stories.

I kept getting the message:
rtc: lost some interrupts at 1024Hz

I was able to solve this by disabling APIC for this virtual machine, i kept ACPI enabled.

Multipath iSCSI under Linux

Building a iSCSI Target (Server) under Linux is fairly simple, just install Enterprise iSCSI Target (IET) and you are ready. The Initiator (Client) is simple to, just use Open-iSCSI and you are ready to go, but how do you make this redundant?

When i first started using iSCSI i heard about the term “multipath”, i read that you could make a redundant IP link to your iSCSI Target with multipath, but how?

Searching on the web didn’t give me real practical anwsers. After using multipath for about 2 years now, i thought, why don’t i make a blog post about it so other people can have redundant iSCSI to!

For this example i have a iSCSI Target with two IP’s:

  1. 172.16.0.1/255.255.255.0
  2. 172.16.1.1/255.255.255.0

These IP’s given to eth0 and eth1, via two switches the connectivity is given to my initiator with the IP’s:

  1. 172.16.0.101/255.255.255.0
  2. 172.16.1.101/255.255.255.0

So there is a redundant network connection to the target, now we just have to start using this.

My target has as IQN: “iqn.2009-11-11.widodh.storage:iscsi-001”

I suppose you know how to configure IET and Open-iSCSI, so i’ll just skip the regular configuration. In this example my Target exports one LUN of 10GB.

On the client (Ubuntu 9.04 (Jaunty)) you have to install:

  1. open-iscsi
  2. multipath-tools

And that’s it, there is no configuration needed for multipath, this is all done dynamically.

Now we are going to discover the Target on both IP’s and log on to it:

iscsiadm -m discovery -t sendtargets -p 172.16.1.1
iscsiadm -m discovery -t sendtargets -p 172.16.0.1
iscsiadm -m node -T iqn.2009-11-11.widodh.storage:iscsi-001 -p 172.16.0.1 --login
iscsiadm -m node -T iqn.2009-11-11.widodh.storage:iscsi-001 -p 172.16.1.1 --login

The nicest thing about this is, that Multipath itself discovers that there is a redundant connection to a SCSI device and everything is done for you.

In “/dev/mapper” you’ll find (for example) “14945540000000000000000000100000099b2f8000f000000″and that is your multipath device.

You can list your multipath devices with:

multipath -ll

In my example this looked like:

14945540000000000000000000100000099b2f8000f000000dm-0 IET     ,VIRTUAL-DISK  
[size=35G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
 \_ 4:0:0:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:0:0 sdc 8:32  [active][ready]

Multipath detected a redundant path for “sdc” and “sdd” and created a device which i could use.

If one of the connections goes down for what ever reason, you should see this in your dmesg:

[ 2070.285310] device-mapper: multipath: Failing path 8:32.

Multipath will then show:

sdc: checker msg is "directio checker reports path is down"
14945540000000000000000000100000099b2f8000f000000dm-0 IET     ,VIRTUAL-DISK  
[size=35G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
 \_ 4:0:0:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 3:0:0:0 sdc 8:32  [failed][faulty]

Yes, you will see a lot of SCSI errors in your dmesg, but since you have a redundant path that is nothing to be worried about.

Just keep in mind, use “/dev/mapper/14945540000000000000000000100000099b2f8000f000000” as your block device for whatever you intent to use it!

Multipath in combination with iSCSI is really great, a simple network hickup will never get your services down and you can keep your network like a basic Layer-2 network, no STP is needed, the redundant paths can go over fully seperated links which decreases the chance for downtime!

Have fun using multipath!

My first post

This is just another personal blog on the big, big, big internet! I work as CTO at PCextreme B.V. and i will be posting a lot about my daily work.

Just like any other system administrator i run into problems i have to fix or some really undocumented software. Just like all other sysadmins i get a lot of information from various blogposts, so i thought, why not help other people by posting my information?

I’ll try to post as much as i can in english, but sometimes when the post is for dutch people only, i’ll make a post in dutch.

But most of my posts will be about Ubuntu Linux, KVM (Kernel Virtual Machine), iSCSI, DRBD, Apache, etc, etc. I’ll not try to post any information that can be found in a howto or documentation.

I hope i’ll be able to help some sysadmins in trouble!