Using fail2ban to block unauthorized calls to CloudStack API

Apache CloudStack does not have a build-in mechanism to rate-limit failed authentication attemps on the API. This potentially allows an attacker to brute-force credentials and gain access.

The api.allowed.source.cidr.list configuration option in CloudStack can be used to globally or on an account level limit the source IPs where the API allows requests from. This is always good to do (if possible), but it does not cover every use-case.

Sometimes you just want to keep malicious traffic outside the door and fail2ban can help there.

Nginx proxy in front of CloudStack

A common use-case is that the Management server of Apache CloudStack is not directly connected to the network, but placed behind a reverse proxy like Nginx or something similar.

This proxy can then also handle SSL termination.

In this example we’re using Nginx as a proxy.

fail2ban

Using fail2ban we can scan the access logs of Nginx and block IP addresses who are abusing our API. In this case we filter on two HTTP status codes:

  • 401
  • 531

This results in that we create the following files:

  • /etc/fail2ban/jail.d/nginx-401.conf
  • /etc/fail2ban/jail.d/nginx-531.conf
  • /etc/fail2ban/filter.d/nginx-401.conf
  • /etc/fail2ban/filter.d/nginx-531.conf

jail.d/nginx-401.conf

[nginx-401]
enabled = true
port = http,https
filter = nginx-401
action = iptables-allports
logpath = %(nginx_access_log)s
bantime = 3600
findtime = 600
maxretry = 25
ignoreip = 127.0.0.1/8

filter.d/nginx-401.conf

[Definition]
failregex = ^ -."(GET|POST|HEAD).HTTP.*" 401
ignoreregex =

Change 401 to 531 where needed to also block HTTP codes 531.

iptables

The action taken by fail2bain is iptables-allports which causes iptables to block all traffic from the particular source IP when it is being banned.

Using the Link-Local Address of IPv6

Link Local

One of the things not know to people is the functionality a Link-Local Address with IPv6 provides.

You might have seen them on your Linux (or any other) system. For example, on my Linux system:

wido@desktop:~$ ip addr show dev eth1
3: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:8f:9f:af:62 brd ff:ff:ff:ff:ff:ff
    inet 10.0.199.15/16 brd 10.0.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:8fff:fe9f:af62/64 scope link 
       valid_lft forever preferred_lft forever
wido@desktop:~$

As you can see, my Link-Local Address in this case is fe80::5054:8fff:fe9f:af62. What can I do with it?

What is it used for?

With IPv6 the Link-Local Address is used for multiple purposes:

  • Finding Routers using a Router Solicitation
  • Performing Duplicate Address Detection
  • Finding Neighbors

The Link-Local is however a fully functional address which you can use for multiple things.

Using Link-Local

Here at the office my colleague has a desktop and his Link-Local Address is fe80::821f:2ff:fed6:5f08.

So can I ping the address?

wido@wido-desktop:~$ ping6 fe80::821f:2ff:fed6:5f08
connect: Invalid argument
wido@wido-desktop:~$

No, that doesn’t seem to work. Or does it?

wido@wido-desktop:~$ ping6 -I eth0 -c 2 fe80::821f:2ff:fed6:5f08
PING fe80::821f:2ff:fed6:5f08(fe80::821f:2ff:fed6:5f08) from fe80::c23f:d5ff:fe68:2808 eth0: 56 data bytes
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=1 ttl=64 time=0.566 ms
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=2 ttl=64 time=0.612 ms

--- fe80::821f:2ff:fed6:5f08 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.566/0.589/0.612/0.023 ms
wido@wido-desktop:~$

So when I specify the interface I can ping his desktop!

You can also specify the interface this way: fe80::821f:2ff:fed6:5f08%eth0

wido@wido-desktop:~$ ping6 -c 2 fe80::821f:2ff:fed6:5f08%eth0
PING fe80::821f:2ff:fed6:5f08%eth0(fe80::821f:2ff:fed6:5f08) 56 data bytes
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=1 ttl=64 time=0.539 ms
64 bytes from fe80::821f:2ff:fed6:5f08: icmp_seq=2 ttl=64 time=0.481 ms

--- fe80::821f:2ff:fed6:5f08%eth0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.481/0.510/0.539/0.029 ms
wido@wido-desktop:~$

So can I SSH to it or do anything else with it?

wido@wido-desktop:~$ ssh fe80::821f:2ff:fed6:5f08%eth0
The authenticity of host 'fe80::821f:2ff:fed6:5f08%eth0 (fe80::821f:2ff:fed6:5f08%eth0)' can't be established.
ECDSA key fingerprint is d8:d7:d0:bd:3c:6a:18:31:e5:26:b1:13:96:a8:e1:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'fe80::821f:2ff:fed6:5f08%eth0' (ECDSA) to the list of known hosts.
wido@fe80::821f:2ff:fed6:5f08%eth0's password: 

wido@wido-desktop:~$

Indeed, I can! I can also telnet to the address:

wido@wido-desktop:~$ telnet fe80::821f:2ff:fed6:5f08%eth0 22
Trying fe80::821f:2ff:fed6:5f08%eth0...
Connected to fe80::821f:2ff:fed6:5f08%eth0.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.9
^]quit

telnet> quit
Connection closed.
wido@wido-desktop:~$

It is a functional address which you can use on your local network.

Security

Even if you think IPv6 is disabled on your system because you haven’t configured it, it isn’t.

Should you disable IPv6 then? No! Learn to work with it. IPv4 space is running out very quickly, so disabling it is not a wise thing to do.

Just make sure your firewall policies for both IPv4 and IPv6 are up to date. I’ve seen many systems where IPv6 isn’t firewalled at all, which makes them open to anybody on the local network.

Link-Local Addresses are not routed over the internet, so somebody has to gain access to the local Layer 2 LAN before it can connect via Link-Local, but still, keep it in mind.