While testing with the Hitch TLS proxy in front of Varnish I stumbled upon a slow startup with a large amount of certificates.
In this case we (at PCextreme) want to run Hitch with around 50.000 certificates configured.
The webpage of Hitch says:
Safe for large installations: performant up to 15 000 listening sockets and 500 000 certificates.
I started testing on my local desktop with 15.000 certificates. My desktop is a Intel NUC with Ubuntu 14.04.
wido@wido-desktop:~/repos/hitch/src$ time sudo ./hitch -n 4 -u nobody -g nogroup --config=/opt/hitch/hitch.conf real 9m40.088s user 9m38.482s sys 0m0.829s wido@wido-desktop:~/repos/hitch/src$
A 10 minute startup time for Hitch is rather long. We started searching for the root-cause.
After some searching we discovered the OpenSSL version in Ubuntu 14.04 was the problem. Testing with Ubuntu 15.10 showed us different results.
root@VM-9d8e8cfd-e30f-4c40-8c4e-2e098b0f11a5:~# time hitch --daemon --pidfile=/run/hitch.pid --user hitch --group hitch --config=/etc/hitch/hitch.conf real 0m18.673s user 0m6.780s sys 0m2.000s
18 seconds is a lot better than 10 minutes!
Ubuntu 14.04 comes with OpenSSL 1.0.1f and Ubuntu 15.10 with 1.0.2d and that is where the difference seems to be.
After this we started testing with 100k certificates. It took 48 seconds to start with that amount of certificates configured.
For production we will use Ubuntu 16.04 which has similar results as Ubuntu 15.10.
So if you find Hitch slow when starting, check your OpenSSL version.