VXLAN
Virtual Extensible LAN uses encapsulation technique to encapsulate OSI layer 2 Ethernet frames within layer 4 UDP datagrams. More on this can be found on the link provided.
For a Ceph and CloudStack environment I needed to set up a Proof-of-Concept using VXLAN and some refurbished hardware. The main purpose of this PoC is to verify that VXLAN works with CloudStack, Ceph and Ubuntu 18.04
VyOS
VyOS is an open source network operating system based on Debian Linux. It supports VXLAN, so using this we were able to test VXLAN in this setup.
In production a other VXLAN capable router would be used, but for a PoC VyOS works just fine running on a regular server.
Configuration
The VyOS router is connected to ‘the internet’ with one NIC and the other NIC is connected to a switch.
Using static routes a IPv4 subnet (/24) and a IPv6 subnet (/48) are routed towards the VyOS router. These are then splitted and send to multiple VLANs.
As it took me a while to configure VXLAN under VyOS
I’m only posting that configuration.
interfaces { ethernet eth0 { address 31.25.96.130/30 address 2a00:f10:100:1d::2/64 duplex auto hw-id 00:25:90:80:ed:fe smp-affinity auto speed auto } ethernet eth5 { duplex auto hw-id a0:36:9f:0d:ab:be mtu 9000 smp-affinity auto speed auto vif 300 { address 192.168.0.1/24 description VXLAN mtu 9000 } vxlan vxlan1000 { address 10.0.0.1/23 address 2a00:f10:114:1000::1/64 group 239.0.3.232 ip { enable-arp-accept enable-arp-announce } ipv6 { dup-addr-detect-transmits 1 router-advert { cur-hop-limit 64 link-mtu 1500 managed-flag false max-interval 600 name-server 2a00:f10:ff04:153::53 name-server 2a00:f10:ff04:253::53 other-config-flag false prefix 2a00:f10:114:1000::/64 { autonomous-flag true on-link-flag true valid-lifetime 2592000 } reachable-time 0 retrans-timer 0 send-advert true } } link eth5.300 mtu 1500 vni 1000 } vxlan vxlan2000 { address 109.72.91.1/26 address 2a00:f10:114:2000::1/64 group 239.0.7.208 ipv6 { dup-addr-detect-transmits 1 router-advert { cur-hop-limit 64 link-mtu 1500 managed-flag false max-interval 600 name-server 2a00:f10:ff04:153::53 name-server 2a00:f10:ff04:253::53 other-config-flag false prefix 2a00:f10:114:2000::/64 { autonomous-flag true on-link-flag true valid-lifetime 2592000 } reachable-time 0 retrans-timer 0 send-advert true } } link eth5.300 mtu 1500 vni 2000 } }
VLAN 300 on eth5 is used to route VNI 1000 and 2000 in their own multicast groups.
The MTU of eth5 is set to 9000 so that the encapsulated traffic of VXLAN can still be 1500 bytes.
Ubuntu 18.04
To test if VXLAN was actually working on the Ubuntu 18.04 host I made a very simple script:
ip link add vxlan1000 type vxlan id 1000 dstport 4789 group 239.0.3.232 dev vlan300 ttl 5 ip link set up dev vxlan1000 ip addr add 10.0.0.11/23 dev vxlan1000 ip addr add 2a00:f10:114:1000::101/64 dev vxlan1000
That works! I can ping 10.0.0.11 and 2a00:f10:114:1000::1 from my Ubuntu 18.04 machine!