Back home from Norway

Ferry from Lofoten

As I wrote in my previous post we took the ferry from Moskenes to Bodø at 07:00 on Friday.

We left the cabin at 04:30 to make sure we got there on time. It was just 89km, but Google Maps told me it would take me 1 hour and 30 minutes. Due to the snow, wind and darkness it took us 1:48 to get there. In time for the ferry!

After 3 hours and 15 minutes we arrived in Bodø to head towards the SuperCharger in Grong. A 512km drive.

Model S on Ferry to Bodø

To Grong

The trip to Grong was long. Nothing really special to mention. Ice and snow on the roads, that is mainly it. A exhausting and long drive mainly.

Fiskevägen all over again!

We took the Fiskevägen route. This time from Grong to Krokom where I took it the other way around last year.

The 240km trip took us 4 hours and 30 minutes. We took it slowly since the view is just awesome!

Halfway we stopped in Rötviken to take a break and charge. Just as last year it was just a 3,6kW charger. It added only 4km of range while we took a break. We did it mainly for the show.

Free 50kW CHAdeMO charger

Getting from the Krokom (Sweden) SuperCharger to the one in Mora (Sweden) is over 300km and I don’t like such long stretches. At home I already found a free CHAdeMO charger in Sveg which is a small town along the E45 from Krokom to Mora.

I should be just a matter of plug in and hit the start button. It was!

Green Highway charger in Sveg

This charger is also part of the Green Highway. Much better than the 3,6kW charger in Rötviken!

We stayed in a Hotel in Sveg. So during our dinner in Sveg the car could fully charge.

‘Almost’ Home

From Sveg we followed the E45 towards Göteborg and down to Malmö and into Denmark. Slept in Bremen and drove the last 600km home.

This part of the trip was not that special. We just drove for 2 days 🙂

Energy Consumption

When I left home I hit the reset button for both trip counters. The end result is 5.571,3km with a total energy consumption of 1,179kWh. That averages to 212Wh/km.

Energy Consumption

Last day before we head back

Aurora Borealis

One of the things we can for was the Aurora Borealis, also called the ‘Northern Lights’.

For two nights we had clear skies and saw a beautiful display. You can find enough pictures on the internet, so I won’t post all of them!

aurora-borealias-1

aurora-borealias-2

aurora-borealias-3

Route home

The route home from the Lofoten Islands is going to be 3.000km. Our guess is that it will take us 4 to 5 days. Not due to the charging, but because you simply can not drive very fast through Norway and Sweden.

Ferry from the Lofoten

We will be taking the ferry from Moskenes to Bodø. The Hurtigruten from Stamsun to Bodø was not an option this time as it leaves late at night.

Route Norway Back 2016

This part of the trip probably won’t be very special. I hope that Fiskevägen will be as nice as it was last year. Besides from that we are not expecting any highlights anymore.

Snow performance of a RWD Model S

Waking up with snow

When we went to best last night the forecast said there would be 1 to 2cm of snowfall. Well, this morning it showed differently. It was 20 to 30cm!

At first it all looked good, the sun was shining and we wanted to go out for a hike through the mountains in the fresh snow.

Driveway to Cabin

More snow

The weather turned however and more and more snow came down on us. No hiking yet, so we started to play some cards.

After a few hours the father of the house owner showed up with his tractor to clear the driveway. We could go out!

Tractor with snow plow

Eventually we had about 30cm of snow over a period of 14 hours.

Model S under snow

Car under snow from back

RWD in the snow

Most people (including Tesla) talk about the snow performance of the Dual Motor Model S. Mine however is a RWD 85kWh from September 2013.

With the Hankook i-cept Evo 2 (W320) tyres it however performs just fine. Sure, it sometimes slips and traction control has to kick in quite often, but overall it works just fine.

Eventually we went out hiking and to get there we had to drive through fresh snow. Not a problem at all.

Obviously the Dual-Motor is a no-brainer when you live in these conditions, but a RWD Model S probably outperforms most other RWD vehicles, if not all.

Frozen charge port

After we went hiking we drove back to the cabin. At the cabin I wanted to plug in again, but the charge port would not open. It was frozen.

It are just a few drops of water which can cause the port to freeze. The solution is simple. Take any credit card format plastic card and use it to force the port open. Simple as that!

Trondheim to the Lofoten Islands

Hurtigruten

We love driving the Model S, but after driving for over 72 hours it is also nice to be ‘driven’.

That’s why we took the Hurtigruten ferry from Trondheim to Stamsund. Stamsund is a port on the Lofoten Islands just 21km from the house we rented on the Lofoten through Airbnb.

On Tuesday we boarded the MS Nordlys at 11:00 and arrived in Stamsund the next day around 19:00.

Hurtigruten ferry dock

Selfie at Hurtigruten

Getting of the ship was tight. With only centimeters to spare and guiduance of the crew we were able to manouvre the Model S off the ship. Yes, it is a wide car!

‘Our’ house

After leaving the ship it took us roughly 45 minutes to drive to the house in Valberg. A beautiful house at the coast looking over the fjords. What an amazing place!

We are going to stay here for a few days to see the Aurora Borealis before we continue more North on the Lofoten.

Model S at house Lofoten

Non-studded tyres

Just as last year I’m driving non studded tyres. Why? We have to pass through Germany and Denmark and studded tyres are not allowed there.Last year I used Nokian Hakka R2 tyres which were great! This year I’m driving Hankook i-cept Evo 2 (W320) tyres and they work very good as well. The last 500m to the house is pure ice and you notice that the tyres have a hard time keeping traction. The traction control in the Model S works exceptionally good however and it works just fine.

Keep in mind: I drive a RWD Model S from September 2013. It is not a new Dual-Motor AWD Model S!

230V network

Most of the part of Norway have a 230V instead of the 400V we have in the Netherlands and other parts of Europe. This means that my UMC (Universal Mobile Connector) does not work here. This is a safety measure of the UMC.

In Norway you can recognize this by the Blue 230V label on electrical installations.

Norwegian 230V label

The UMC performs a check if there is 0V between Ground and Neutral, but here that’s not the case. There is 120V between GND and N which makes my UMC show a red light. It thinks there is a ground leak, which is a bad thing.

UMC red light

There is a special Norwegian version of the UMC, but I built my own using Smart EVSE. It does NOT perform a Ground check, but it allows me to charge.

SmartEVSE homebrew UMC

My Model S is happily charging at 13A right now.

Model S next to house Lofoten

This means I have a new charging POI on my Model S’s screen!

Charging POI on Lofoten

Time to relax!

After being on the road for 5 days it is time to relax. Watch the Aurora, go out hiking and do nothing.

From Middelburg to Trondheim

To Hirtshals

Last Saturday we left at 08:00 from Middelburg for the 1.100km drive to Hirtshals, Denmark. From there we would take the ferry to Larvik, Norway on Sunday morning.

It took us 14 hours to reach Hirtshals. Traffic was bad, very bad starting at Hamburg towards the border. Roadworks and border controls made it stop and go over almost 100km!

A short night followed since our ferry left at 08:00.

Lier South SuperCharger

After arriving in Larvik our first SuperCharger in Norway was Lier South, 100km from Larvik.

It was busy! After we parked all 8 stalls are occupied. Other Model S had to wait in the queue.

Lier South SuperCharger

A queue is bad, but it also shows that the infrastructure is used! It’s not a charger which is rarely used. From what I understood it was also a vacation period, so that might have caused the spike in traffic.

Lillehammer

After charging in Lier we headed to Lillehammer. We would stay the night there and charge again.

Fortum CHAdeMO

While heading to Lillehammer I stopped at a CHAdeMO from Fortum to see if I could charge there. The people from Fortum told me that I could use my Dutch phone and send a SMS to active it.

Well, that didn’t work. I borrowed a RFID tag from somebody else as a backup. On the Lofoten Islands I will need to use a Fortum charger, so I wanted to know if it worked. Lesson learned. It doesn’t.

Fortum CHAdeMO charger

Busy times at Lillehammer

On the E6 to Lillehammer we already spotted a lot of Model S coming from Lillehammer, so I expected the SuperCharger to be crowded.

It was! 9 of the 10 stalls we busy, so we parked at the last stall available.

As we were charging we saw more Model S arrive. We still had 100km left in the battery and we would leave the next morning. We vacated the stall and to decided to charge the next morning for the 155km drive to Dombas and Trondheim.

We checked in at the hotel and went for a dinner in Lillehammer.

SuperCharging with a cold battery

The next morning the car had been in -8C for the night. When I switched to ‘Drive’ a warning indicated that regenerative braking had been disabled. This was due to the battery being cold.

SuperCharging didn’t go very fast. When I just started it would charge with 17kW and slowly climbed to roughly 30kW before we had enough to leave for Dombas.

This was a similar experience as last year at the Krokom SuperCharger in -22C.

The picture below shows that we were charging with 24kW where under normal conditions it should have been about 80kW.

Slow Lillehammer SuperCharger

To Trondheim

From Lillehammer we drove to the Dombas SuperCharger. After a charge and lunch there we headed down to Klett (near Trondheim).

Nothing really special on this part of the trip. The temperature was about -5C and the (road) conditions were good.

To the Lofoten

Our destination is a house we rented through Airbnb on the Lofoten Islands.

From Trondheim we are taking the Hurtigruten ferry to Stamsund on the Lofoten. This will take 2 days.

From Stamsund to the house it is just 21km. Time to relax!

Energy Consumption

The tripmeter shows 1861km and a total usage of 391kWh. That’s 210Wh/km. Not bad at all!

MariaDB Galera cluster on IPv6

MariaDB Galera

I try to set as much IPv6-only infrastructure as possible and the same goes for a new MariaDB Galera cluster I had to build.

According to the release notes MariaDB 10.1 should have IPv6 support, but it didn’t work out for me. I wouldn’t get my Galera cluster to work over IPv6-only.

Galera

I tracked the root-cause down to Galera not parsing the addresses properly and it had to be tweaked a bit.

Configuration

With the configuration posted below I was able to get a MariaDB 10.1 setup working on IPv6-only.

[mysqld]
query_cache_size=0
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_type=0

bind-address = ::

wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

wsrep_cluster_name="ns01"
wsrep_cluster_address="gcomm://ns011.XXX.eu,ns012.XXX.nl,ns013.XXX.info"

wsrep_sst_method=rsync

wsrep_node_name="ns011"

wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567; ist.recv_addr=[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4568"
wsrep_node_address = "[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4567"
wsrep_sst_receive_address = "[2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:4444"

This resulted in the Galera cluster functioning properly on a IPv6-only network. It’s indeed a bit more configuration then with IPv4, but that will probably be resolved in a future release.

The MariaDB status properly shows being connected over IPv6:

MariaDB [(none)]> show status like 'wsrep_incoming_addresses';
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
| Variable_name            | Value                                                                                                              |
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
| wsrep_incoming_addresses | [2a00:f10:121:XX:XX:a0ff:fe00:1bc7]:3306,[2a00:f10:400:XX:XX:d8ff:fe00:2ef]:3306,[2a00:1d20:3:XX:XX:c01:3:53]:3306 |
+--------------------------+--------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]>

IPv6 Prefix Delegation on a Cisco 887VA behind a XS4All VDSL2 connection

XS4All connection

At the PCextreme office we have a XS4All VDSL2 connection which has native IPv6. We get a /48 from XS4All.

I wrote two earlier blogposts about getting the Cisco 887VA router setup which might be of interest before you continue reading:

IPv6 Prefix Delegation

From XS4All we get a /48 routed to our office using DHCPv6 Prefix Delegation. We are experimenting and testing with Docker at the office where we also want to test the IPv6 capabilities of Docker.

The goal was to sub-delegate /60 subnets out of a /56 towards clients internally. I had to figure out how to get this configured on Cisco IOS.

  • We get a /48 delegated from XS4All
  • The first /56 is used for our local networks (LAN, Guest and Servers)
  • The second /56 is used as a pool to delegate /60 subnets from

Sipcalc

To calculate the IPv6 subnets used the tool ‘sipcalc’. I needed to find the second /56 in our /48:

sipcalc -S 56 2001:980:XX::/48

The output is rather long, so I trimmed it a bit:

-[ipv6 : 2001:980:XX::/48] - 0

[Split network]
Network			- 2001:0980:XX:0000:0000:0000:0000:0000 -
			  2001:0980:XX:00ff:ffff:ffff:ffff:ffff
Network			- 2001:0980:XX:0100:0000:0000:0000:0000 -
			  2001:0980:XX:01ff:ffff:ffff:ffff:ffff
Network			- 2001:0980:XX:0200:0000:0000:0000:0000 -
			  2001:0980:XX:02ff:ffff:ffff:ffff:ffff
...
...
Network			- 2001:0980:XX:ff00:0000:0000:0000:0000 -
			  2001:0980:XX:ffff:ffff:ffff:ffff:ffff

-

In this case 2001:0980:XX:0100:0000:0000:0000:0000:/56 is the second /56 in our /48.

Cisco IOS

Some searching brought me to cisco.com which had some examples.

Eventually it was actually quite easy to get it working.

Configuration

You need a DHCPv6 pool inside the Cisco and tell it to start a DHCPv6 server on the proper interface.

ipv6 dhcp pool local-ipv6
 prefix-delegation pool local-ipv6-pd-pool lifetime 3600 1800
 dns-server 2001:888:0:6::66
 dns-server 2001:888:0:9::99
 domain-name pcextreme.nl
interface Vlan1
 ip address 192.168.5.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ipv6 address xs4all-prefix ::1/64
 ipv6 enable
 ipv6 nd other-config-flag
 ipv6 nd ra interval 30
 ipv6 nd ra dns server 2001:888:0:6::66
 ipv6 nd ra dns server 2001:888:0:9::99
 ipv6 dhcp server local-ipv6 rapid-commit
 ipv6 mld query-interval 60
ipv6 local pool local-ipv6-pd-pool 2001:980:XX:100::/56 60

That’s all!

Asking for a Prefix

On my Ubuntu desktop I could now request a subnet:

wido@wido-desktop:~$ sudo dhclient -6 -P -v eth0
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Bound to *:546
Listening on Socket/eth0
Sending on   Socket/eth0
PRC: Soliciting for leases (INIT).
XMT: Forming Solicit, 0 ms elapsed.
XMT:  X-- IA_PD d5:68:28:08
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT: Solicit on eth0, interval 1060ms.
RCV: Advertise message on eth0 from fe80::da67:d9ff:fe81:bcec.
RCV:  X-- IA_PD d5:68:28:08
RCV:  | X-- starts 1455279332
RCV:  | X-- t1 - renew  +900
RCV:  | X-- t2 - rebind +1440
RCV:  | X-- [Options]
RCV:  | | X-- IAPREFIX 2001:980:XX:100::/60
RCV:  | | | X-- Preferred lifetime 1800.
RCV:  | | | X-- Max lifetime 3600.
RCV:  X-- Server ID: 00:03:00:01:d8:67:d9:81:bc:f0
RCV:  Advertisement recorded.
PRC: Selecting best advertised lease.

As you can see I got 2001:980:XX:100::/60 delegated to my desktop.

IPv6 routes

After I asked for a subnet on my desktop this is how the routes look like. You can see a /60 being routed to my Link-Local Address.

firewall-vdsl-veldzigt#show ipv6 route
IPv6 Routing Table - default - 8 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       H - NHRP, D - EIGRP, EX - EIGRP external, ND - ND Default
       NDp - ND Prefix, DCE - Destination, NDr - Redirect, O - OSPF Intra
       OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2, ON1 - OSPF NSSA ext 1
       ON2 - OSPF NSSA ext 2, la - LISP alt, lr - LISP site-registrations
       ld - LISP dyn-eid, a - Application
S   ::/0 [1/0]
     via Dialer0, directly connected
S   2001:980:XX::/48 [1/0]
     via Null0, directly connected
C   2001:980:XX::/64 [0/0]
     via Vlan1, directly connected
L   2001:980:XX::1/128 [0/0]
     via Vlan1, receive
C   2001:980:XX:1::/64 [0/0]
     via Vlan300, directly connected
L   2001:980:XX:1::1/128 [0/0]
     via Vlan300, receive
S   2001:980:XX:100::/60 [1/0]
     via FE80::C23F:D5FF:FE68:XX, Vlan1
L   FF00::/8 [0/0]
     via Null0, receive
firewall-vdsl-veldzigt#

The subnet is working now and I can use it to hand it out to my Docker containers.

Fully electric to Norway, again!

Last Year

In the beginning of 2015 I drove a 5.500km trip with my father to the most Northern Tesla Motors SuperCharger. For a few reasons:

  • To see the Aurora Borealis
  • To prove it can be done with an electric car
  • Because I like roadtrips and travelling

This winter I’m going it again!

My Model S

I have a ‘classic’ Model S from September 2013. No Auto-Pilot features or All-Wheel drive. It’s a 85kWh RWD model.

The ODO currently displays 110.000km and when we get back it won’t be long before I hit the 120.000km.

Still enjoying this car every time I drive it.

Lofoten

My girlfriend also wants to go to Norway to see the Aurora Borealis. She heard me telling her all the stories for over a year about how great it was and how much I like Norway.

So I said: “Why don’t we go there?”

This year the destination will be the Lofoten Islands. From what I’ve seen and heard it is about the best place to watch the Aurora Borealis!

Route to Lofoten

Our route will take us from Middelburg (Netherlands) to Hirtshals (Denmark) where we take the ferry towards Larvik (Norway). Following the Tesla SuperChargers we will drive to Bodø from where we take the ferry to the Lofoten Islands.

Route Norway 2016

On the Lofoten there are no Tesla SuperChargers, so I’ll be using the CHAdeMO chargers using the CHAdeMO adapter to charge my Model S there.

Tesla CHAdeMO adapter

I found these CHAdeMO chargers on elbil.no’s Hurtigladekartet and on uppladdning.nu.

Route back home

We’ll drive back through Sweden where I want to take Fiskevägen again. What a beautiful route!

Route Norway Back 2016

The total route should be about 6.000km

Preparations

Since I did almost all the preparing last year already I still have about everything I need.

Making sure we have enough water, food, heat and proper winter gear with us. We should be fine!

ISC Kea DHCPv6 server

DHCPv6

In most situations StateLess Address AutoConfiguration (SLAAC) works just fine when you work with simple clients in a IPv6 network. But in other cases you want to assign pre-defined addresses or prefixes to clients and there DHCPv6 comes in to play.

While working on the IPv6 implementation for Apache CloudStack I found Kea, a DHCPv6 server from ISC.

DHCPv6 DUID

With IPv4 you could easily identify a client based on the MAC-address it send the DHCP request from. With IPv6 there is a DUID. The “DHCP Unique Identifier”. This is generated by the client and then used by the DHCPv6 server. A few possibilities the clients can choose from:

  • DUID-LL: DUID Based on Link-layer Address
  • DUID-LLT: Link-layer Address Plus Time
  • DUID-EN: Assigned by Vendor Based on Enterprise Number

While DUID seems nice, it can’t be dictated by the DHCPv6 server. The client generates the DUID itself and sends it towards the server. Not something you prefer if your are not in control of the clients.

In a cloud you are in control over the MAC-address, so that is what you want to use where possible. It can’t be spoofed by the client.

ISC Kea

Kea is a DHCPv4/DHCPv6 server being developed by the Internet Systems Consortium. It is a extensible and flexible DHCP server. Facebook uses it in their datacenters.

My goal was very simple. Set up Kea and see if I can use it to hand out an address to a client.

Configuration

I download the tarball and tested it with this configuration between two simple KVM VMs on my desktop.

{
    "Dhcp6": {
        "renew-timer": 1000,
        "rebind-timer": 2000,
        "preferred-lifetime": 3000,
        "valid-lifetime": 4000,
        "lease-database": {
            "type": "memfile",
            "persist": true,
            "name": "/tmp/kea-leases6.csv",
            "lfc-interval": 1800
        },
        "interfaces-config": {
            "interfaces": [ "eth1/2001:db8::1" ]
        },
        "mac-sources": ["duid"],
        "subnet6": [
            {
                "subnet": "2001:db8::/64",
                "id": 1024,
                "interface": "eth1",
                "pools": [
                    { "pool": "2001:db8::100-2001:db8::ffff" }
                ],
                "pd-pools": [
                    {
                        "prefix": "2001:db8:fff::",
                        "prefix-len": 48,
                        "delegated-len": 60
                    }
                ],
                "reservations": [
                    {
                        "hw-address": "52:54:00:d6:c2:a9",
                        "ip-addresses": [ "2001:db8::5054:ff:fed6:c2a9" ]
                    }
                ]
            }
        ]
    }
}

Starting Kea with this configuration was rather simple:

Starting Kea

$ kea-dhcp6 -c /etc/kea.json -d

Logs

When it starts you see some interesting bits in the log:

DHCP6_CONFIG_NEW_SUBNET a new subnet has been added to configuration: 2001:db8::/64 with params t1=1000, t2=2000, preferred-lifetime=3000, valid-lifetime=4000, rapid-commit is disabled
DHCPSRV_CFGMGR_ADD_SUBNET6 adding subnet 2001:db8::/64
HOSTS_CFG_ADD_HOST add the host for reservations: hwaddr=52:54:00:d6:c2:a9 ipv6_subnet_id=1024 hostname=(empty) ipv4_reservation=(no) ipv6_reservation0=2001:db8::5054:ff:fed6:c2a9
HOSTS_CFG_GET_ONE_SUBNET_ID_HWADDR_DUID get one host with IPv6 reservation for subnet id 1024, HWADDR hwtype=1 52:54:00:d6:c2:a9, DUID (no-duid)
HOSTS_CFG_GET_ALL_HWADDR_DUID get all hosts with reservations for HWADDR hwtype=1 52:54:00:d6:c2:a9 and DUID (no-duid)
HOSTS_CFG_GET_ALL_IDENTIFIER get all hosts with reservations using identifier: hwaddr=52:54:00:d6:c2:a9
HOSTS_CFG_GET_ALL_IDENTIFIER_COUNT using identifier hwaddr=52:54:00:d6:c2:a9, found 0 host(s)
HOSTS_CFG_GET_ONE_SUBNET_ID_HWADDR_DUID_NULL host not found using subnet id 1024, HW address hwtype=1 52:54:00:d6:c2:a9 and DUID (no-duid)
HOSTS_CFG_GET_ONE_SUBNET_ID_ADDRESS6 get one host with reservation for subnet id 1024 and including IPv6 address 2001:db8::5054:ff:fed6:c2a9
HOSTS_CFG_GET_ALL_SUBNET_ID_ADDRESS6 get all hosts with reservations for subnet id 1024 and IPv6 address 2001:db8::5054:ff:fed6:c2a9
HOSTS_CFG_GET_ALL_SUBNET_ID_ADDRESS6_COUNT using subnet id 1024 and address 2001:db8::5054:ff:fed6:c2a9, found 0 host(s)
HOSTS_CFG_GET_ONE_SUBNET_ID_ADDRESS6_NULL host not found using subnet id 1024 and address 2001:db8::5054:ff:fed6:c2a9
DHCPSRV_MEMFILE_DB opening memory file lease database: lfc-interval=1800 name=/tmp/kea-leases6.csv persist=true type=memfile universe=6
DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /tmp/kea-leases6.csv

You can see it has one reservation based on the MAC-address of the client which it handed out after it booted:

ALLOC_ENGINE_V6_HR_ADDR_GRANTED reserved address 2001:db8::5054:ff:fed6:c2a9 was assigned to client duid=[00:01:00:01:1e:47:7e:66:52:54:00:d6:c2:a9], tid=0xe7899a

Ubuntu client

The client was a simple Ubuntu 14.04 client with this network configuration:

auto eth0
iface eth0 inet dhcp
iface eth0 inet6 dhcp

And indeed, it obtained the correct address:

root@ubuntu1404:~# ip addr show dev eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:d6:c2:a9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.100/24 brd 192.168.100.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:db8::5054:ff:fed6:c2a9/64 scope global deprecated dynamic 
       valid_lft 62sec preferred_lft 0sec
    inet6 fe80::5054:ff:fed6:c2a9/64 scope link 
       valid_lft forever preferred_lft forever
root@ubuntu1404:~#

Lease database

Kea can store the leases in a CSV file or MySQL database if you want. In this test I used /tmp/kea-leases6.csv as a CSV file to store the leases in.

In production a MySQL database is probably easier to use, but for the test CSV worked just fine.

Slow requests with Ceph: ‘waiting for rw locks’

Slow requests in Ceph

When a I/O operating inside Ceph is taking more than X seconds, which is 30 by default, it will be logged as a slow request.

This is to show you as a admin that something is wrong inside the cluster and you have to take action.

Origin of slow requests

Slow requests can happen for multiple reasons. It can be slow disks, network connections or high load on machines.

If a OSD has slow requests you can log on to the machine and see what Ops are blocking:

ceph daemon osd.X dump_ops_in_flight

waiting for rw locks

Yesterday I got my hands on a Ceph cluster which had a very high number, over 2k, of slow requests.

On all OSDs they showed ‘waiting for rw locks’

This is hard to diagnose and it was. Usually this is where OSDs are busy connecting to other OSDs or performing any other network actions.

Usually when you see ‘waiting for rw locks’ there is something wrong with the network.

The network

In this case the Ceph cluster is connecting over Layer 2 and that network didn’t change. A few hours earlier there was a change to the Layer 3 network, but since Ceph was running over Layer 2 we didn’t connect the two dots.

After some more searching we noticed that the hosts couldn’t perform DNS lookups properly.

DNS

Ceph doesn’t use DNS internally, but it could still be that it was a problem.

After some searching we found that DNS wasn’t the problem, but there were two default routes on the system where one was down.

Layer 3

This Ceph cluster is communicating over Layer 3 and the problem was caused by the fact that the cluster had a hard time talking back to various clients.

This caused various network buffers to fill up and that caused communication problems between OSDs.

So always make sure you double-check the network since that is usually the root-cause.