Over the last week, I've been building a new server for some friends and I to host our own NextCloud instance. Part of this is to keep our technical eyes up-to-date and relevant, with the other being to reduce some of our reliance on Google and to own our own data.
After using Ansible to set up the server and configure its security and firewalls, I created our Let's Encrypt SSL certificates (as this step is more easily completed outside Ansible). I then decided to go a step further and create client certificates which would prohibit anyone but the holders of those certificates from accessing the NextCloud instance. This meant that even if users had poor passwords or if there was a zero-day on NextCloud, we had a further line of defence to prevent access.
Every single forum post I found online said that we couldn't create client certificates with Let's Encrypt as we don't have the root certificate, meaning no signing ability. The only solution that seemed viable from here was to create our own Certificate Authority (CA) and combine origin SSL termination from Let's Encrypt with certificates generated from our own CA.
I knew the basics, from setting up a CA on my home server, however this time would be a little difference since we were using Certbot to provision our certificates and Cloudflare to manage our DNS/protect our edge.
Taking a huge number of examples from other Ansible code, and a few trips into StackOverflow, I put together an Ansible role that would manage our server CA and create client certificates for us. Our server
vars/main.yml was then extended to customise variables from this role to create client certificates:
After running Ansible, we had a number of client certificates that I distributed to everyone who used the server. A certificate was created for each user so everyone had their own separate access keys. During the previous step, I also configured Ansible to alter Nginx configuration and make it require client certificates. The resulting Nginx configuration thus included the following lines:
Before anyone says anything, I know it's better to use an intermediate certificate rather than the root, but we wanted something quick to start with.
Unfortunately, regardless of any of the above, we received the following error.
This was definitely an error I'd seen before when I set up another CA for myself and spent hours banging my head against the wall to get the right combination of OpenSSL commands and certificates created. I initially assumed that I'd done something wrong when formulating the Ansible role, however the answer turned out to be more simple than that.
Because of our set up, any internet traffic that reaches our server origin has to pass through Cloudflare which acts as both a CDN and WAF for us. My hypothesis was that something was happening on the wire between their edge and our origin that meant client certificates weren't getting transmitted with the request.
Bypassing Cloudflare by hard-coding our server IP in
/etc/hosts confirmed this, so it looked like we'd have to can the whole idea of client certificates and instead restrict our access to when we were on our Wireguard VPN.
A brief look through Cloudflare options however gave me a bit of hope as there was reference to client certificates. I was presented with two options when I clicked 'Create client certificate':
- Generate private key and CSR with Cloudflare
- Use my private key and CSR
Seeing as we'd gone through the effort of creating our own CA, I decided that we'd allow Cloudflare to take our CSRs (helpfully already on the server from our Ansible role), and create and sign some certificates so we could take advantage of restricting based on valid certificate at the edge.
Once I had the signed certificates from Cloudflare, I needed to go back to our server and create some combined PKCS #12 files that could be loaded into each of our browsers in order to authenticate with the Cloudflare edge. The following command uses the signed certificate from Cloudflare and the private key on the server.
The benefit of this method is that while Cloudflare is able to authenticate our client certificates, they only hold what is publicly available so our private keys are never leaked to a third party. The benefit of this is that even if someone breaks into my Cloudflare account, they would not be able to make or retrieve valid client certificates.
The next step to this approach is to add a firewall rule that ensures all requests coming in to specific hostnames have a valid certificate. The below rule blocks requests that traverse Cloudflare that are not accompanied by a valid client certificate.
Ah but what about people who are sneaky and use your /etc/hosts method
The final step to this approach is to deny access to Nginx from anyone outside Cloudflare's IP ranges. As is well documented elsewhere on the internet, I took the Cloudflare IP list and configured Ansible to add the following to my Nginx configuration.
This then allows us to get the best of all worlds:
- SSL certificates managed with Certbot and Let's Encrypt
- Our own CA to create client certificates without third party involvement
- Protection at the edge with client certificates
- Locking Nginx requests and responses to those coming from the edge so people can't bypass it
I'll have another blog post up in the coming weeks about how I then integrated NextCloud, Wiki.js, as well as a number of other custom services with Keycloak acting as an identity provider (IdP) to reduce the number of usernames and password in use.
1: We're not stupid enough to try to host our own mail.