Maximum amount of Docker containers on a single host

While playing with Docker I wanted to know how many containers I could spawn on a single system.

A quick for-loop told me that the maximum is 1023 containers on a single host:

Error response from daemon: Cannot start container 09c8f46b59ccc311e8d0352789db6debd0fa1df98186c5cda98583d762d48601: adding interface vetha5d205e to bridge docker0 failed: exchange full

The limitation here is the Linux bridging which can’t have more then 1023 interfaces attached. Specifically net/bridge/br_private.h BR_PORT_BITS cannot be extended because of spanning tree requirements.

wido@wido-desktop:~$ docker ps|wc -l
1024
wido@wido-desktop:~$

Although that says 1024 there is a header line, so we have to subtract one. That brings it to 1023.

wido@wido-desktop:~$ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64
wido@wido-desktop:~$

Ubuntu and the changing MAC address with bonding

With the ‘new’ style for configuring bonding under Ubuntu your bond device will not always have the same MAC address across reboots.

For example, you configure your bond in the /etc/network/interfaces file:

auto p9p1
iface p9p1 inet manual
        bond-master bond0

auto p10p1
iface p10p1 inet manual
        bond-master bond0

auto bond0
iface bond0 inet manual
        bond-slaves none
        bond-mode 4
        bond-miimon 100
        bond-updelay 5
        bond-downdelay 5

During boot, both interface p9p1 and p10p1 will be hot-plugged under bond0. The first device to be plugged into the bonding device determines which MAC address the bonded device gets.

Due to hardware timing it might be p9p1 OR p10p1 which is the first. This behavior makes the MAC address selection inconsistent between reboots and that might cause problems with:

DHCP for IPv4
IPv6 with SLAAC (Stateless Auto Configuration)
DHCPv6

This has been filed as bug #1288196 with Ubuntu, but no fix from that side so far.

The solutions for now:

auto p9p1
iface p9p1 inet manual
        bond-master bond0

auto p10p1
iface p10p1 inet manual
        pre-up sleep 5
        bond-master bond0

This makes sure p10p1 always comes online 5 seconds after p9p1.

But you can also set a static MAC address for the bonding device:

auto bond0
iface bond0 inet manual
        hwaddress fe:80:12:04:6d:6f
        bond-slaves none
        bond-mode 4
        bond-miimon 100
        bond-updelay 5
        bond-downdelay 5

Choose what you prefer or works best in your situation.

Using the Link-Local Address of IPv6

Link Local

One of the things not know to people is the functionality a Link-Local Address with IPv6 provides.

You might have seen them on your Linux (or any other) system. For example, on my Linux system:

wido@desktop:~$ ip addr show dev eth1
3: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:8f:9f:af:62 brd ff:ff:ff:ff:ff:ff
    inet 10.0.199.15/16 brd 10.0.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:8fff:fe9f:af62/64 scope link 
       valid_lft forever preferred_lft forever
wido@desktop:~$

As you can see, my Link-Local Address in this case is fe80::5054:8fff:fe9f:af62. What can I do with it?

What is it used for?

With IPv6 the Link-Local Address is used for multiple purposes:

Finding Routers using a Router Solicitation
Performing Duplicate Address Detection
Finding Neighbors

The Link-Local is however a fully functional address which you can use for multiple things.

Using Link-Local

Here at the office my colleague has a desktop and his Link-Local Address is fe80::821f:2ff:fed6:5f08.

So can I ping the address?

wido@wido-desktop:~$ ping6 fe80::821f:2ff:fed6:5f08
connect: Invalid argument
wido@wido-desktop:~$

No, that doesn’t seem to work. Or does it?

wido@wido-desktop:~$ ping6 -I eth0 -c 2 fe80::821f:2ff:fed6:5f08
PING fe80::821f:2ff:fed6:5f08(fe80::821f:2ff:fed6:5f08) from fe80::c23f:d5ff:fe68:2808 eth0: 56 data bytes
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=1 ttl=64 time=0.566 ms
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=2 ttl=64 time=0.612 ms

--- fe80::821f:2ff:fed6:5f08 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.566/0.589/0.612/0.023 ms
wido@wido-desktop:~$

So when I specify the interface I can ping his desktop!

You can also specify the interface this way: fe80::821f:2ff:fed6:5f08%eth0

wido@wido-desktop:~$ ping6 -c 2 fe80::821f:2ff:fed6:5f08%eth0
PING fe80::821f:2ff:fed6:5f08%eth0(fe80::821f:2ff:fed6:5f08) 56 data bytes
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=1 ttl=64 time=0.539 ms
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=2 ttl=64 time=0.481 ms

--- fe80::821f:2ff:fed6:5f08%eth0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.481/0.510/0.539/0.029 ms
wido@wido-desktop:~$

So can I SSH to it or do anything else with it?

wido@wido-desktop:~$ ssh fe80::821f:2ff:fed6:5f08%eth0
The authenticity of host 'fe80::821f:2ff:fed6:5f08%eth0 (fe80::821f:2ff:fed6:5f08%eth0)' can't be established.
ECDSA key fingerprint is d8:d7:d0:bd:3c:6a:18:31:e5:26:b1:13:96:a8:e1:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'fe80::821f:2ff:fed6:5f08%eth0' (ECDSA) to the list of known hosts.
wido@fe80::821f:2ff:fed6:5f08%eth0's password: 

wido@wido-desktop:~$

Indeed, I can! I can also telnet to the address:

wido@wido-desktop:~$ telnet fe80::821f:2ff:fed6:5f08%eth0 22
Trying fe80::821f:2ff:fed6:5f08%eth0...
Connected to fe80::821f:2ff:fed6:5f08%eth0.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.9
^]quit

telnet> quit
Connection closed.
wido@wido-desktop:~$

It is a functional address which you can use on your local network.

Security

Even if you think IPv6 is disabled on your system because you haven’t configured it, it isn’t.

Should you disable IPv6 then? No! Learn to work with it. IPv4 space is running out very quickly, so disabling it is not a wise thing to do.

Just make sure your firewall policies for both IPv4 and IPv6 are up to date. I’ve seen many systems where IPv6 isn’t firewalled at all, which makes them open to anybody on the local network.

Link-Local Addresses are not routed over the internet, so somebody has to gain access to the local Layer 2 LAN before it can connect via Link-Local, but still, keep it in mind.