VirtualBox images to experiment with IPv6

Around me I noticed that a lot of people don’t have hands-on experience with IPv6. The networks they work in do not support IPv6 nor does their ISP provide them with native IPv6 connectivity at home.

On my local systems I often use Virtual Box to set up (IPv6) testing environments. I thought I’d create some Virtual Machine images to get some hands-on experience with IPv6.

The images and README can be found on Github and are aimed to be easy to install and work with.

Requirements

To run the images you need to have Virtual Box installed. You also should be able to use the Linux command line as the Virtual Machines are based on Ubuntu 16.04.

More information can be found in the repository on Github in the README file.

Download

You can download the images here.

How to use

Please take a look at the README on Github. It tells you how to use them.

Happy testing!

Hitch TLS Proxy performance with 15k certificates

While testing with the Hitch TLS proxy in front of Varnish I stumbled upon a slow startup with a large amount of certificates.

In this case we (at PCextreme) want to run Hitch with around 50.000 certificates configured.

The webpage of Hitch says:

Safe for large installations: performant up to 15 000 listening sockets and 500 000 certificates.

10 minutes

I started testing on my local desktop with 15.000 certificates. My desktop is a Intel NUC with Ubuntu 14.04.

wido@wido-desktop:~/repos/hitch/src$ time sudo ./hitch -n 4 -u nobody -g nogroup --config=/opt/hitch/hitch.conf

real    9m40.088s
user    9m38.482s
sys 0m0.829s
wido@wido-desktop:~/repos/hitch/src$

A 10 minute startup time for Hitch is rather long. We started searching for the root-cause.

OpenSSL

After some searching we discovered the OpenSSL version in Ubuntu 14.04 was the problem. Testing with Ubuntu 15.10 showed us different results.

root@VM-9d8e8cfd-e30f-4c40-8c4e-2e098b0f11a5:~# time hitch --daemon --pidfile=/run/hitch.pid --user hitch --group hitch --config=/etc/hitch/hitch.conf

real    0m18.673s
user    0m6.780s
sys    0m2.000s

18 seconds is a lot better than 10 minutes!

Ubuntu 14.04 comes with OpenSSL 1.0.1f and Ubuntu 15.10 with 1.0.2d and that is where the difference seems to be.

100.000 certificates

After this we started testing with 100k certificates. It took 48 seconds to start with that amount of certificates configured.

For production we will use Ubuntu 16.04 which has similar results as Ubuntu 15.10.

So if you find Hitch slow when starting, check your OpenSSL version.

PXE boot over IPv6 with iPXE

For a Ceph project I’m involved in we wanted to figure out if we could PXE-boot our servers over IPv6. In this case we were using SuperMicro 5018A-AR12L servers with a additional Intel X520 10Gbit NIC.

The Ceph cluster in this case will be IPv6 only and user Layer 3 routing between 6 racks and a 180 machines initially (7.2PB raw). No IPv4 in this network present. That’s the goal!

By default these NICs only boot over IPv4, so we had to figure out if we could reconfigure them in a way so that they would PXE-boot over IPv6.

My search brought me to the iPXE project. A PXE-boot project which you can flash into your NICs or chainload using TFTP.

Before I started flashing machines I created a test setup in VirtualBox to see if I could get it working over IPv6.

iPXE and IPv6

By default the ISO you can download from the iPXE website isn’t build with IPv6 support. You have to manually compile iPXE with v6 support.

git clone git://git.ipxe.org/ipxe.git
cd ipxe/src
nano config/general.h

Now change:

#undef NET_PROTO_IPV6

To:

#define NET_PROTO_IPV4          /* IPv4 protocol */
#define NET_PROTO_IPV6          /* IPv6 protocol */

Now we can compile iPXE:

make bin/ipxe.iso

VirtualBox

To test this all I set up VirtualBox on my laptop. I created a machine called IPv6Router and a VM called iPXE.

The IPv6Router Instance has two network connections:

  • eth0: NAT
  • eth1: Host-Only Network vboxnet0

The machine iPXE got just one connection:

  • eth0: Host-Only Network vboxnet0

Networking: DHCPv6, Router Advertisements, HTTP and DNS

Before I could use this setup I needed to install a few services and configure the network on this machine.

I choose Ubuntu 14.04 in this case, the Linux distribution I prefer most.

interfaces configuration

First I had to configure eth1

auto eth1
iface eth1 inet6 static
    address 2001:db8::1
    netmask 64

Install packages

Before I could continue I needed a couple of packages on the system. All I needed was available in the Ubuntu repositories. Apt could install them for me quickly.

apt-get install isc-dhcp-server radvd unbound apache2

After the network was configured and the right packages were available I could configure all the services.

DHCPv6

/etc/dhcp/dhcpd6.conf

option dhcp6.user-class code 15 = string;
option dhcp6.bootfile-url code 59 = string;
option dhcp6.client-arch-type code 61 = array of unsigned integer 16;

option dhcp6.name-servers 2001:db8::1;

if exists dhcp6.client-arch-type and
   option dhcp6.client-arch-type = 00:07 {
    option dhcp6.bootfile-url "http://[2001:db8::1]/ipxe.efi";
} else if exists dhcp6.user-class and
          substring(option dhcp6.user-class, 2, 4) = "iPXE" {
    option dhcp6.bootfile-url "http://[2001:db8::1]/ubuntu.cfg";
}

subnet6 2001:db8::/64 {}
service isc-dhcp-server6 restart

radvd

/etc/radvd.conf

interface eth1
{
        MinRtrAdvInterval 5;
        MaxRtrAdvInterval 60;
        AdvSendAdvert on;
        AdvOtherConfigFlag on;
        IgnoreIfMissing off;

        prefix ::/64 {
        };

        RDNSS 2001:db8::1 {
        };
};
service radvd restart

Unbound

/etc/unbound/unbound.conf.d/local.conf

server:
    interface: 0.0.0.0
    interface: ::0
    interface-automatic: yes
    access-control: 127.0.0.1 allow
    access-control: ::1 allow
    access-control: 2001:db8::/32 allow
service unbound restart

Apache webserver

iPXE and the Ubuntu installer I was trying to bootstrap needed a webserver to download files from. I used Apache for that purpose.

Since I also experimented with TFTP in the process I had all my files in /srv/tftp so that’s where I also pointed Apache.

The reason why I choose HTTP over TFTP is just speed. It’s a lot faster and more modern.

/etc/apache2/sites-available/001-preseed.conf

<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	DocumentRoot /srv/tftp

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory /srv/tftp/>
            Options Indexes FollowSymLinks
            AllowOverride None
            Require all granted
        </Directory>
</VirtualHost>

Now enable this VirtualHost and disable the default one.

a2dissite 000-default
a2ensite 001-preseed

Restart Apache afterwards.

service apache2 restart

So with this configuration I’ve set up the following:

  • DHCPv6
  • IPv6 Router Advertisements
  • DNS resolving for clients
  • Apache for serving files over HTTP

Ubuntu Netboot using iPXE and Preseed

Now that everything is configured we can configure the configuration for iPXE.

Some searching on the internet brought me to help.ubuntu.com which explained how Ubuntu netboot could be used.

It is quite simple, you have to download netboot.tar.gz and extract it.

cd /srv/tftp
wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/netboot.tar.gz
tar xvfz netboot.tar.gz

This will extract a directory ubuntu-installer. It contains all we need to start a network installation.

We can reference to these files in a iPXE configuration file.

/srv/tftp/ubuntu.cfg

#!ipxe

kernel /ubuntu-installer/amd64/linux noapic nolapic acpi=off irqpoll preseed/url=http://[2001:db8::1]/preseed/ubuntu1404.cfg debian-installer=en_US auto locale=en_US kbd-chooser/method=us hostname=alpha fb=false debconf/frontend=noninteractive keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false netcfg/disable_autoconfig boolean=true netcfg/use_autoconfig boolean=true netcfg/disable_dhcp boolean=true
initrd /ubuntu-installer/amd64/initrd.gz
boot

Here I refer to a preseed file which is used by the Debian/Ubuntu installer. This process is called preseeding.

I had to add some tweaks to make it work over IPv6-only:

d-i netcfg/disable_autoconfig boolean true
d-i netcfg/use_autoconfig boolean true
d-i netcfg/disable_dhcp boolean true
d-i netcfg/dhcpv6_timeout string 10

/srv/tftp/preseed/ubuntu1404.cfg

# Language
d-i debian-installer/language string en
d-i debian-installer/locale string en_US.UTF-8
d-i localechooser/preferred-locale string en_US.UTF-8
d-i localechooser/supported-locales en_US.UTF-8

# Keyboard
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layout select USA
d-i keyboard-configuration/variant select USA
d-i keyboard-configuration/modelcode string pc105

# Network
d-i netcfg/disable_autoconfig boolean true
d-i netcfg/use_autoconfig boolean true
d-i netcfg/disable_dhcp boolean true
d-i netcfg/dhcpv6_timeout string 10
d-i netcfg/get_hostname string this-host
d-i netcfg/get_domain string this-host

# Timezone
d-i time/zone string UTC
d-i clock-setup/utc-auto boolean true
d-i clock-setup/utc boolean true
d-i time/zone string Europe/Amsterdam


# Software
d-i debconf debconf/frontend select Noninteractive
d-i pkgsel/install-language-support boolean false
tasksel tasksel/first multiselect standard, ubuntu-server

# Storage
d-i partman-auto/method string regular
d-i partman-auto/disk string /dev/sda
d-i partman-auto/choose_recipe select atomic
d-i partman/confirm_write_new_label boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true

# Mirror
d-i mirror/country string manual
d-i mirror/http/hostname string ubuntu.apt-get.eu
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string

# Users
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password password ceph
d-i passwd/root-password-again password ceph
d-i user-setup/encrypt-home boolean false
d-i user-setup/allow-password-weak boolean true

# No language support packages.
d-i pkgsel/install-language-support boolean false

# Additional packages
d-i pkgsel/include string ssh acpid ntp resolvconf

# Security updates
d-i pkgsel/update-policy select unattended-upgrades

# Upgrade
d-i pkgsel/upgrade select full-upgrade

# Update sshd_config to ensure root user is able to login
d-i preseed/late_command string sed -i 's/PermitRootLogin without-password/PermitRootLogin Yes/g' /target/etc/ssh/sshd_config

# Bootloader
d-i grub-installer/only_debian boolean true
d-i finish-install/reboot_in_progress note

Installing Ubuntu

I now started the iPXE Virtual Machine with ipxe.iso attached and it got up and running!

iPXE will boot, obtain a IPv6 address and run the Ubuntu installer. All over IPv6!

iPXE over IPv6

Ubuntu and the changing MAC address with bonding

With the ‘new’ style for configuring bonding under Ubuntu your bond device will not always have the same MAC address across reboots.

For example, you configure your bond in the /etc/network/interfaces file:

auto p9p1
iface p9p1 inet manual
        bond-master bond0

auto p10p1
iface p10p1 inet manual
        bond-master bond0

auto bond0
iface bond0 inet manual
        bond-slaves none
        bond-mode 4
        bond-miimon 100
        bond-updelay 5
        bond-downdelay 5

During boot, both interface p9p1 and p10p1 will be hot-plugged under bond0. The first device to be plugged into the bonding device determines which MAC address the bonded device gets.

Due to hardware timing it might be p9p1 OR p10p1 which is the first. This behavior makes the MAC address selection inconsistent between reboots and that might cause problems with:

  • DHCP for IPv4
  • IPv6 with SLAAC (Stateless Auto Configuration)
  • DHCPv6

This has been filed as bug #1288196 with Ubuntu, but no fix from that side so far.

The solutions for now:

auto p9p1
iface p9p1 inet manual
        bond-master bond0

auto p10p1
iface p10p1 inet manual
        pre-up sleep 5
        bond-master bond0

This makes sure p10p1 always comes online 5 seconds after p9p1.

But you can also set a static MAC address for the bonding device:

auto bond0
iface bond0 inet manual
        hwaddress fe:80:12:04:6d:6f
        bond-slaves none
        bond-mode 4
        bond-miimon 100
        bond-updelay 5
        bond-downdelay 5

Choose what you prefer or works best in your situation.

SQL connection error after upgrade to CloudStack 4.3.0

I just upgraded a small cluster of mine from CloudStack 4.2.1 to 4.3.0 and after installing the packages on my Ubuntu system the management server wouldn’t start due to a SQL error:

2014-03-25 20:52:13,643 INFO  [c.c.u.d.T.Transaction] (main:null) Is Data Base High Availiability enabled? Ans : false
2014-03-25 20:52:13,736 ERROR [c.c.u.d.Merovingian2] (main:null) Unable to get a new db connection
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true
	at java.sql.DriverManager.getConnection(DriverManager.java:635)
	at java.sql.DriverManager.getConnection(DriverManager.java:195)

I quickly remembered a licensing issue around JDBC which delayed 4.3.0 and I was right. The management server was missing the right JAR/package for the SQL connection.

A quick apt-get install fixed it:

$ sudo apt-get install libmysql-java

This should have been a dependency of the ‘cloudstack-management’ package, but that somehow slipped through. I already applied a patch in the master branch and I’ll make sure it gets into 4.3.1 and 4.4.0.

So if you are running Ubuntu and are upgrading to CloudStack 4.3.0 and run into this issue, simply install the package and it’s fixed.