VirtualBox images to experiment with IPv6

Around me I noticed that a lot of people don’t have hands-on experience with IPv6. The networks they work in do not support IPv6 nor does their ISP provide them with native IPv6 connectivity at home.

On my local systems I often use Virtual Box to set up (IPv6) testing environments. I thought I’d create some Virtual Machine images to get some hands-on experience with IPv6.

The images and README can be found on Github and are aimed to be easy to install and work with.

Requirements

To run the images you need to have Virtual Box installed. You also should be able to use the Linux command line as the Virtual Machines are based on Ubuntu 16.04.

More information can be found in the repository on Github in the README file.

Download

You can download the images here.

How to use

Please take a look at the README on Github. It tells you how to use them.

Happy testing!

Hitch TLS Proxy performance with 15k certificates

While testing with the Hitch TLS proxy in front of Varnish I stumbled upon a slow startup with a large amount of certificates.

In this case we (at PCextreme) want to run Hitch with around 50.000 certificates configured.

The webpage of Hitch says:

Safe for large installations: performant up to 15 000 listening sockets and 500 000 certificates.

10 minutes

I started testing on my local desktop with 15.000 certificates. My desktop is a Intel NUC with Ubuntu 14.04.

wido@wido-desktop:~/repos/hitch/src$ time sudo ./hitch -n 4 -u nobody -g nogroup --config=/opt/hitch/hitch.conf

real    9m40.088s
user    9m38.482s
sys 0m0.829s
wido@wido-desktop:~/repos/hitch/src$

A 10 minute startup time for Hitch is rather long. We started searching for the root-cause.

OpenSSL

After some searching we discovered the OpenSSL version in Ubuntu 14.04 was the problem. Testing with Ubuntu 15.10 showed us different results.

root@VM-9d8e8cfd-e30f-4c40-8c4e-2e098b0f11a5:~# time hitch --daemon --pidfile=/run/hitch.pid --user hitch --group hitch --config=/etc/hitch/hitch.conf

real    0m18.673s
user    0m6.780s
sys    0m2.000s

18 seconds is a lot better than 10 minutes!

Ubuntu 14.04 comes with OpenSSL 1.0.1f and Ubuntu 15.10 with 1.0.2d and that is where the difference seems to be.

100.000 certificates

After this we started testing with 100k certificates. It took 48 seconds to start with that amount of certificates configured.

For production we will use Ubuntu 16.04 which has similar results as Ubuntu 15.10.

So if you find Hitch slow when starting, check your OpenSSL version.

AnyIP: Bind a whole subnet to your Linux machine

IPv6 Prefix Delegation

In my previous post I wrote how you can use Docker with IPv6 and Prefix Delegation.

A IPv6 subnet routed to a Linux machine can be used with other things than Docker. That’s where the AnyIP feature of the kernel comes in.

Linux Kernel AnyIP

The AnyIP feature of the Linux kernel allows you to bind a complete IPv4 or IPv6 subnet to your system.

Instead of adding all addresses manually to the kernel you can tell it to bind a complete subnet.

Configuring

IPv4

ip -4 route add local 192.168.0.0/24 dev lo

In this case the Linux kernel will now respond to ARP requests for any IPv4 address in the 192.168.0.0/24 subnet.

IPv6

ip -6 route add local 2001:db8:100::/64 dev lo

In this case the kernel will respond for Neigh Sollicitations on any IPv6 address in the 2001:db8:100::/64 subnet.

Example usage

Let’s assume that you have the IPv6 prefix 2001:db8:100::/60 routed to your Linux machine through IPv6 prefix delegation.

From that /60 subnet we take the first /64 subnet and attach it to lo.

ip -6 route add local 2001:db8:100::/64 dev lo

You can now ping any of the addresses in that subnet:

  • 2001:db8:100::1
  • 2001:db8:100::100
  • 2001:db8:100::200
  • 2001:db8:100::dead:b33f

If you would start a webserver which listens on port 80 you can use any of the IPv6 addresses in that subnet and the webserver will respond to it.

Use cases

It could be that you want to to mass-shared hosting on a system where you want to assign each hostname/domainname it’s own IPv6 address. Instead of attaching single IPs to a interface you can simply attach a complete subnet and point traffic to any of the IPs in that subnet.

Demo

On a virtual machine on PCextreme’s Aurora Compute I deployed a Instance with Prefix Delegation enabled.

After running ‘dhclient’ I got the subnet 2a00:f10:500:40::/60 assigned to my Instance.

It was then just one line to attach a /64 subnet:

ip -6 route add local 2a00:f10:500:40::/64 dev lo

Random address generator

I wrote a small piece of Python code to generate a random IPv6 address:

#!/usr/bin/env python3
"""
Generate a random IPv6 address for a specified subnet
"""

from random import seed, getrandbits
from ipaddress import IPv6Network, IPv6Address

subnet = '2a00:f10:500:40::/64'

seed()
network = IPv6Network(subnet)
address = IPv6Address(network.network_address + getrandbits(network.max_prefixlen - network.prefixlen))

print(address)

Using a small loop in Bash I could now ping random addresses in that subnet:

while [ true ]; do ping6 -c 2 `./random-ipv6.py`; done

Some example output:

--- 2a00:f10:500:40:d142:1092:ea84:74b4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 10.252/11.680/13.108/1.428 ms
PING 2a00:f10:500:40:4e50:f264:6ea9:d184(2a00:f10:500:40:4e50:f264:6ea9:d184) 56 data bytes
64 bytes from 2a00:f10:500:40:4e50:f264:6ea9:d184: icmp_seq=1 ttl=56 time=10.0 ms
64 bytes from 2a00:f10:500:40:4e50:f264:6ea9:d184: icmp_seq=2 ttl=56 time=10.0 ms

--- 2a00:f10:500:40:4e50:f264:6ea9:d184 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 10.085/10.087/10.089/0.002 ms
PING 2a00:f10:500:40:d831:1f89:b06d:fe12(2a00:f10:500:40:d831:1f89:b06d:fe12) 56 data bytes
64 bytes from 2a00:f10:500:40:d831:1f89:b06d:fe12: icmp_seq=1 ttl=56 time=9.77 ms
64 bytes from 2a00:f10:500:40:d831:1f89:b06d:fe12: icmp_seq=2 ttl=56 time=10.1 ms

--- 2a00:f10:500:40:d831:1f89:b06d:fe12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 9.777/9.958/10.140/0.207 ms
PING 2a00:f10:500:40:2c45:26ee:5b93:fa2(2a00:f10:500:40:2c45:26ee:5b93:fa2) 56 data bytes
64 bytes from 2a00:f10:500:40:2c45:26ee:5b93:fa2: icmp_seq=1 ttl=56 time=10.2 ms
64 bytes from 2a00:f10:500:40:2c45:26ee:5b93:fa2: icmp_seq=2 ttl=56 time=10.0 ms

Docker and IPv6 Prefix Delegation

As posted earlier I have IPv6 Prefix Delegation working at our office to test with Docker.

One of the missing links was to automatically configure Docker to use the prefix obtained through DHCPv6+PD. I manually configured the prefix in Docker, but I also had to run dhclient manually.

I figured this could be automated so I gave it a try.

Ubuntu Networking

At first I tried to figure out if Ubuntu’s networking was somehow able to request a prefix through DHCPv6. Long story short: Neither Ubuntu nor CentOS are able to do so. You have to script this manually.

dhclient

To obtain a prefix I had to run dhclient manually. That wasn’t to hard. Simply run:

dhclient -6 -P -d -v eth0

This resulted in obtaining a prefix:

Bound to *:546
Listening on Socket/eth0
Sending on   Socket/eth0
PRC: Confirming active lease (INIT-REBOOT).
XMT: Forming Rebind, 0 ms elapsed.
XMT:  X-- IA_PD d5:68:28:08
XMT:  | X-- Requested renew  +3600
XMT:  | X-- Requested rebind +5400
XMT:  | | X-- IAPREFIX 2001:980:XXXX:140::/60
XMT:  | | | X-- Preferred lifetime +7200
XMT:  | | | X-- Max lifetime +7500
XMT:  V IA_PD appended.
XMT: Rebind on eth0, interval 940ms.
RCV: Reply message on eth0 from fe80::da67:d9ff:fe81:bcec.
RCV:  X-- IA_PD d5:68:28:08
RCV:  | X-- starts 1457617054
RCV:  | X-- t1 - renew  +604800
RCV:  | X-- t2 - rebind +967680
RCV:  | X-- [Options]
RCV:  | | X-- IAPREFIX 2001:980:XXXX:140::/60
RCV:  | | | X-- Preferred lifetime 1209600.
RCV:  | | | X-- Max lifetime 2592000.
RCV:  X-- Server ID: 00:03:00:01:d8:67:d9:81:bc:f0
PRC: Bound to lease 00:03:00:01:d8:67:d9:81:bc:f0.
PRC: Renewal event scheduled in 604800 seconds, to run for 362880 seconds.
PRC: Depreference scheduled in 1209600 seconds.
PRC: Expiration scheduled in 2592000 seconds.

As you can see, I got a /60 prefix. Now I had to somehow get this automated and configure Docker to use it.

Upstart

Since I was testing with Docker 1.10 under Ubuntu 14.04 I had to use Upstart to run dhclient.

The /etc/init/dhclient6-pd.conf Upstart script I created was rather simple:

description     "DHCPv6 Prefix Delegation client"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 30 3
umask 022

console log

exec dhclient -6 -P -d eth0

DHCP hook

dhclient has hooks which it can execute when something happens. I wrote a hook which extracted the delegated IPv6 prefix and restarted Docker.

I placed the hook in the default location for DHCP hooks: /etc/dhcp/dhclient-enter-hooks.d/docker-ipv6:

#!/bin/bash

SUBNET_SIZE=80
DOCKER_ETC_DIR="/etc/docker"
DOCKER_PREFIX_FILE="${DOCKER_ETC_DIR}/ipv6.prefix"

if [ ! -z "$new_ip6_prefix" ]; then
    SUBNET=$(sipcalc -S $SUBNET_SIZE $new_ip6_prefix|grep Network|head -n 1|awk '{print $3}')
    echo "${SUBNET}/${SUBNET_SIZE}" > $DOCKER_PREFIX_FILE

    if [ "$old_ip6_prefix" != "$new_ip6_prefix" ]; then
        service docker restart
    fi
fi

For this to work you need to modify /etc/default/docker so that this line reads:

DOCKER_OPTS="--ipv6 --fixed-cidr-v6=`cat /etc/docker/ipv6.prefix`"

The result

Docker was now running properly with a IPv6 subnet configured and my containers have a IPv6 address as well.

wido@wido-desktop:~$ docker exec -ti 94c8f02 ip addr show dev eth0
13: eth0:  mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:980:XXXX:140:0:242:ac11:2/80 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever
wido@wido-desktop:~$

Native IPv6 in my Docker containers fully automated and dynamic!

All the scripts I used can be found on Github.

Back home from Norway

Ferry from Lofoten

As I wrote in my previous post we took the ferry from Moskenes to Bodø at 07:00 on Friday.

We left the cabin at 04:30 to make sure we got there on time. It was just 89km, but Google Maps told me it would take me 1 hour and 30 minutes. Due to the snow, wind and darkness it took us 1:48 to get there. In time for the ferry!

After 3 hours and 15 minutes we arrived in Bodø to head towards the SuperCharger in Grong. A 512km drive.

Model S on Ferry to Bodø

To Grong

The trip to Grong was long. Nothing really special to mention. Ice and snow on the roads, that is mainly it. A exhausting and long drive mainly.

Fiskevägen all over again!

We took the Fiskevägen route. This time from Grong to Krokom where I took it the other way around last year.

The 240km trip took us 4 hours and 30 minutes. We took it slowly since the view is just awesome!

Halfway we stopped in Rötviken to take a break and charge. Just as last year it was just a 3,6kW charger. It added only 4km of range while we took a break. We did it mainly for the show.

Free 50kW CHAdeMO charger

Getting from the Krokom (Sweden) SuperCharger to the one in Mora (Sweden) is over 300km and I don’t like such long stretches. At home I already found a free CHAdeMO charger in Sveg which is a small town along the E45 from Krokom to Mora.

I should be just a matter of plug in and hit the start button. It was!

Green Highway charger in Sveg

This charger is also part of the Green Highway. Much better than the 3,6kW charger in Rötviken!

We stayed in a Hotel in Sveg. So during our dinner in Sveg the car could fully charge.

‘Almost’ Home

From Sveg we followed the E45 towards Göteborg and down to Malmö and into Denmark. Slept in Bremen and drove the last 600km home.

This part of the trip was not that special. We just drove for 2 days 🙂

Energy Consumption

When I left home I hit the reset button for both trip counters. The end result is 5.571,3km with a total energy consumption of 1,179kWh. That averages to 212Wh/km.

Energy Consumption

Last day before we head back

Aurora Borealis

One of the things we can for was the Aurora Borealis, also called the ‘Northern Lights’.

For two nights we had clear skies and saw a beautiful display. You can find enough pictures on the internet, so I won’t post all of them!

aurora-borealias-1

aurora-borealias-2

aurora-borealias-3

Route home

The route home from the Lofoten Islands is going to be 3.000km. Our guess is that it will take us 4 to 5 days. Not due to the charging, but because you simply can not drive very fast through Norway and Sweden.

Ferry from the Lofoten

We will be taking the ferry from Moskenes to Bodø. The Hurtigruten from Stamsun to Bodø was not an option this time as it leaves late at night.

Route Norway Back 2016

This part of the trip probably won’t be very special. I hope that Fiskevägen will be as nice as it was last year. Besides from that we are not expecting any highlights anymore.

Snow performance of a RWD Model S

Waking up with snow

When we went to best last night the forecast said there would be 1 to 2cm of snowfall. Well, this morning it showed differently. It was 20 to 30cm!

At first it all looked good, the sun was shining and we wanted to go out for a hike through the mountains in the fresh snow.

Driveway to Cabin

More snow

The weather turned however and more and more snow came down on us. No hiking yet, so we started to play some cards.

After a few hours the father of the house owner showed up with his tractor to clear the driveway. We could go out!

Tractor with snow plow

Eventually we had about 30cm of snow over a period of 14 hours.

Model S under snow

Car under snow from back

RWD in the snow

Most people (including Tesla) talk about the snow performance of the Dual Motor Model S. Mine however is a RWD 85kWh from September 2013.

With the Hankook i-cept Evo 2 (W320) tyres it however performs just fine. Sure, it sometimes slips and traction control has to kick in quite often, but overall it works just fine.

Eventually we went out hiking and to get there we had to drive through fresh snow. Not a problem at all.

Obviously the Dual-Motor is a no-brainer when you live in these conditions, but a RWD Model S probably outperforms most other RWD vehicles, if not all.

Frozen charge port

After we went hiking we drove back to the cabin. At the cabin I wanted to plug in again, but the charge port would not open. It was frozen.

It are just a few drops of water which can cause the port to freeze. The solution is simple. Take any credit card format plastic card and use it to force the port open. Simple as that!

Trondheim to the Lofoten Islands

Hurtigruten

We love driving the Model S, but after driving for over 72 hours it is also nice to be ‘driven’.

That’s why we took the Hurtigruten ferry from Trondheim to Stamsund. Stamsund is a port on the Lofoten Islands just 21km from the house we rented on the Lofoten through Airbnb.

On Tuesday we boarded the MS Nordlys at 11:00 and arrived in Stamsund the next day around 19:00.

Hurtigruten ferry dock

Selfie at Hurtigruten

Getting of the ship was tight. With only centimeters to spare and guiduance of the crew we were able to manouvre the Model S off the ship. Yes, it is a wide car!

‘Our’ house

After leaving the ship it took us roughly 45 minutes to drive to the house in Valberg. A beautiful house at the coast looking over the fjords. What an amazing place!

We are going to stay here for a few days to see the Aurora Borealis before we continue more North on the Lofoten.

Model S at house Lofoten

Non-studded tyres

Just as last year I’m driving non studded tyres. Why? We have to pass through Germany and Denmark and studded tyres are not allowed there.Last year I used Nokian Hakka R2 tyres which were great! This year I’m driving Hankook i-cept Evo 2 (W320) tyres and they work very good as well. The last 500m to the house is pure ice and you notice that the tyres have a hard time keeping traction. The traction control in the Model S works exceptionally good however and it works just fine.

Keep in mind: I drive a RWD Model S from September 2013. It is not a new Dual-Motor AWD Model S!

230V network

Most of the part of Norway have a 230V instead of the 400V we have in the Netherlands and other parts of Europe. This means that my UMC (Universal Mobile Connector) does not work here. This is a safety measure of the UMC.

In Norway you can recognize this by the Blue 230V label on electrical installations.

Norwegian 230V label

The UMC performs a check if there is 0V between Ground and Neutral, but here that’s not the case. There is 120V between GND and N which makes my UMC show a red light. It thinks there is a ground leak, which is a bad thing.

UMC red light

There is a special Norwegian version of the UMC, but I built my own using Smart EVSE. It does NOT perform a Ground check, but it allows me to charge.

SmartEVSE homebrew UMC

My Model S is happily charging at 13A right now.

Model S next to house Lofoten

This means I have a new charging POI on my Model S’s screen!

Charging POI on Lofoten

Time to relax!

After being on the road for 5 days it is time to relax. Watch the Aurora, go out hiking and do nothing.

From Middelburg to Trondheim

To Hirtshals

Last Saturday we left at 08:00 from Middelburg for the 1.100km drive to Hirtshals, Denmark. From there we would take the ferry to Larvik, Norway on Sunday morning.

It took us 14 hours to reach Hirtshals. Traffic was bad, very bad starting at Hamburg towards the border. Roadworks and border controls made it stop and go over almost 100km!

A short night followed since our ferry left at 08:00.

Lier South SuperCharger

After arriving in Larvik our first SuperCharger in Norway was Lier South, 100km from Larvik.

It was busy! After we parked all 8 stalls are occupied. Other Model S had to wait in the queue.

Lier South SuperCharger

A queue is bad, but it also shows that the infrastructure is used! It’s not a charger which is rarely used. From what I understood it was also a vacation period, so that might have caused the spike in traffic.

Lillehammer

After charging in Lier we headed to Lillehammer. We would stay the night there and charge again.

Fortum CHAdeMO

While heading to Lillehammer I stopped at a CHAdeMO from Fortum to see if I could charge there. The people from Fortum told me that I could use my Dutch phone and send a SMS to active it.

Well, that didn’t work. I borrowed a RFID tag from somebody else as a backup. On the Lofoten Islands I will need to use a Fortum charger, so I wanted to know if it worked. Lesson learned. It doesn’t.

Fortum CHAdeMO charger

Busy times at Lillehammer

On the E6 to Lillehammer we already spotted a lot of Model S coming from Lillehammer, so I expected the SuperCharger to be crowded.

It was! 9 of the 10 stalls we busy, so we parked at the last stall available.

As we were charging we saw more Model S arrive. We still had 100km left in the battery and we would leave the next morning. We vacated the stall and to decided to charge the next morning for the 155km drive to Dombas and Trondheim.

We checked in at the hotel and went for a dinner in Lillehammer.

SuperCharging with a cold battery

The next morning the car had been in -8C for the night. When I switched to ‘Drive’ a warning indicated that regenerative braking had been disabled. This was due to the battery being cold.

SuperCharging didn’t go very fast. When I just started it would charge with 17kW and slowly climbed to roughly 30kW before we had enough to leave for Dombas.

This was a similar experience as last year at the Krokom SuperCharger in -22C.

The picture below shows that we were charging with 24kW where under normal conditions it should have been about 80kW.

Slow Lillehammer SuperCharger

To Trondheim

From Lillehammer we drove to the Dombas SuperCharger. After a charge and lunch there we headed down to Klett (near Trondheim).

Nothing really special on this part of the trip. The temperature was about -5C and the (road) conditions were good.

To the Lofoten

Our destination is a house we rented through Airbnb on the Lofoten Islands.

From Trondheim we are taking the Hurtigruten ferry to Stamsund on the Lofoten. This will take 2 days.

From Stamsund to the house it is just 21km. Time to relax!

Energy Consumption

The tripmeter shows 1861km and a total usage of 391kWh. That’s 210Wh/km. Not bad at all!

MariaDB Galera cluster on IPv6

MariaDB Galera

I try to set as much IPv6-only infrastructure as possible and the same goes for a new MariaDB Galera cluster I had to build.

According to the release notes MariaDB 10.1 should have IPv6 support, but it didn’t work out for me. I wouldn’t get my Galera cluster to work over IPv6-only.

Galera

I tracked the root-cause down to Galera not parsing the addresses properly and it had to be tweaked a bit.

Configuration

With the configuration posted below I was able to get a MariaDB 10.1 setup working on IPv6-only.

[mysqld]
query_cache_size=0
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_type=0

bind-address = ::

wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

wsrep_cluster_name="ns01"
wsrep_cluster_address="gcomm://ns011.XXX.eu,ns012.XXX.nl,ns013.XXX.info"

wsrep_sst_method=rsync

wsrep_node_name="ns011"

wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567; ist.recv_addr=[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4568"
wsrep_node_address = "[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4567"
wsrep_sst_receive_address = "[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4444"

This resulted in the Galera cluster functioning properly on a IPv6-only network. It’s indeed a bit more configuration then with IPv4, but that will probably be resolved in a future release.

The MariaDB status properly shows being connected over IPv6:

MariaDB [(none)]> show status like 'wsrep_incoming_addresses';
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
| Variable_name            | Value                                                                                                              |
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
| wsrep_incoming_addresses | [2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:3306,[2a00:f10:400:XX:XX:d8ff:fe00:2ef]:3306,[2a00:1d20:3:XX:XX:c01:3:53]:3306 |
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]>