Recently we purchased a EqualLogic PS6000XVS for a KVM environment.
In most of our iSCSI systems we use Multipath I/O, we do this by giving the iSCSI Target two NIC’s and give each NIC a IP-Address in a different subnet over a physically different network. This way we have two seperate I/O path’s to the iSCSI Target.
The EqualLogic does not support this, it only supports one virtual IP in one network, so multipathing gets a bit difficult.
On the Dell Wiki there is configuration howto, so I read that carefully.
The examples are for RedHat, but we are using Ubuntu, but that should not make a big difference, but it did….
Our storage network is in the subnet 192.168.32.0/19 where the virtual IP of the EqualLogic is 192.168.32.1. You should know, this is a virtual IP, in total we have three PS6000 nodes, which do some magic by responding with a different MAC Address for 192.168.32.1 towards each client.
One of our clients has the following configuration for the storage connectivity:
eth0 Link encap:Ethernet HWaddr 14:FE:B5:C6:62:E0
inet addr:192.168.37.4 Bcast:192.168.63.255 Mask:255.255.224.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:27263332 errors:0 dropped:0 overruns:0 frame:0
TX packets:25323692 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:24569609290 (22.8 GiB) TX bytes:132201626154 (123.1 GiB)
Interrupt:170 Memory:e6000000-e6012800
eth1 Link encap:Ethernet HWaddr 14:FE:B5:C6:62:E2
inet addr:192.168.38.4 Bcast:192.168.63.255 Mask:255.255.224.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:27246580 errors:0 dropped:0 overruns:0 frame:0
TX packets:25335109 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:24549507448 (22.8 GiB) TX bytes:132201622012 (123.1 GiB)
Interrupt:178 Memory:e8000000-e8012800
It took some work to get this working. Bot NIC’s are connected to the same subnet, through different switches though.
The first problem you will run into is the ARP flux problem of Linux, I’m not going to write to much about this, on the internet there is more then enough information written about this topic.
I ended up with this configuration:
auto eth0
iface eth0 inet static
address 192.168.37.4
netmask 255.255.224.0
post-up sysctl -w net.ipv4.conf.eth0.rp_filter=0
post-up sysctl -w net.ipv4.conf.eth0.arp_ignore=1
post-up sysctl -w net.ipv4.conf.eth0.arp_announce=2
auto eth2
iface eth2 inet static
address 192.168.38.4
netmask 255.255.224.0
post-up sysctl -w net.ipv4.conf.eth2.rp_filter=0
post-up sysctl -w net.ipv4.conf.eth2.arp_ignore=1
post-up sysctl -w net.ipv4.conf.eth2.arp_announce=2
For Open-iSCSI I created two interfaces called ieth0 and ieth1 and routed my iSCSI traffic through them. How you can do this can be found at the Dell wiki.
But it did not work! I was able to ping the EqualLogic over eth0, but not over eth1. If I brought down eth0, it would work over eth1, but not vise versa. It took me a while to find it, but it’s due to a default setting in Ubuntu, done in /etc/sysctl.d/10-network-security.conf, this enables rp_filter (Reverse Path Filtering) by default, so I modified that file
# Turn on Source Address Verification in all interfaces to # prevent some spoofing attacks. #net.ipv4.conf.default.rp_filter=1 #net.ipv4.conf.all.rp_filter=1
And voila! My iSCSI multipathing started to work! My multipath shows:
[size=1.0T][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=2][active] \_ 13:0:0:0 sdk 8:160 [active][ready] \_ 14:0:0:0 sdj 8:144 [active][ready] eql-0-8a0906-4f2b9e409-2b800184d024d9db_c () dm-4 EQLOGIC,100E-00 [size=2.0T][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=2][active] \_ 6:0:0:0 sdg 8:96 [active][ready] \_ 11:0:0:0 sdf 8:80 [active][ready]
This should work under Ubuntu 10.04. Took me some time to figure it all out, but now it’s working like a charm. But still, I prefer multipathing over two different VLAN’s and subnets, really odd that the EqualLogic does not support this!