Client certificates, Let's Encrypt, custom CAs and Cloudflare

Over the last week, I've been building a new server for some friends and I to host our own NextCloud instance. Part of this is to keep our technical eyes up-to-date and relevant, with the other being to reduce some of our reliance on Google and to own our own data^[1].

After using Ansible to set up the server and configure its security and firewalls, I created our Let's Encrypt SSL certificates (as this step is more easily completed outside Ansible). I then decided to go a step further and create client certificates which would prohibit anyone but the holders of those certificates from accessing the NextCloud instance. This meant that even if users had poor passwords or if there was a zero-day on NextCloud, we had a further line of defence to prevent access.

Every single forum post I found online said that we couldn't create client certificates with Let's Encrypt as we don't have the root certificate, meaning no signing ability. The only solution that seemed viable from here was to create our own Certificate Authority (CA) and combine origin SSL termination from Let's Encrypt with certificates generated from our own CA.

I knew the basics, from setting up a CA on my home server, however this time would be a little difference since we were using Certbot to provision our certificates and Cloudflare to manage our DNS/protect our edge.

Taking a huge number of examples from other Ansible code, and a few trips into StackOverflow, I put together an Ansible role that would manage our server CA and create client certificates for us. Our server vars/main.yml was then extended to customise variables from this role to create client certificates:

ca_passphrase: somethingsecrethere
ca_country_name: AU
ca_organization_name: Oursite
ca_organizational_unit_name: ca.oursite.com
ca_state_or_province_name: NSW
ca_email_address: [email protected]
ca_common_name: Oursite Root CA
ca_requests:
  - name: adam
    email_address: [email protected]
    common_name: oursite.com
    subject_alt_name:
      - DNS:www.oursite.com
      - DNS:wiki.oursite.com
      - DNS:cloud.oursite.com
    country_name: AU
    organization_name: Oursite
    organizational_unit_name: Client Certificate
    passphrase: somethingsecrethere
    cipher: aes256
    
ca_privatekey_path: "{{ ca_private_path }}/OurSite_Root_CA.pem"
ca_csr_path: ca/OurSite_Root_CA.csr
ca_certificate_path: "{{ ca_certs_path }}/OurSite_Root_CA.crt"
ca_root_name: "Oursite_Root"

After running Ansible, we had a number of client certificates that I distributed to everyone who used the server. A certificate was created for each user so everyone had their own separate access keys. During the previous step, I also configured Ansible to alter Nginx configuration and make it require client certificates. The resulting Nginx configuration thus included the following lines:

server {
    listen 443 ssl http2;
    server_name oursite.com;
    index index.html index.htm;

    ssl_certificate     /etc/letsencrypt/live/oursite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/oursite.com/privkey.pem;
    ssl_verify_client on;
    ssl_client_certificate /etc/ssl/ca/certs/OurRoot_CA.crt;
    location / {
        proxy_pass http://localhost:3434/;
    }
}

Before anyone says anything, I know it's better to use an intermediate certificate rather than the root, but we wanted something quick to start with.

Unfortunately, regardless of any of the above, we received the following error.

This was definitely an error I'd seen before when I set up another CA for myself and spent hours banging my head against the wall to get the right combination of OpenSSL commands and certificates created. I initially assumed that I'd done something wrong when formulating the Ansible role, however the answer turned out to be more simple than that.

Because of our set up, any internet traffic that reaches our server origin has to pass through Cloudflare which acts as both a CDN and WAF for us. My hypothesis was that something was happening on the wire between their edge and our origin that meant client certificates weren't getting transmitted with the request.

Bypassing Cloudflare by hard-coding our server IP in /etc/hosts confirmed this, so it looked like we'd have to can the whole idea of client certificates and instead restrict our access to when we were on our Wireguard VPN.

A brief look through Cloudflare options however gave me a bit of hope as there was reference to client certificates. I was presented with two options when I clicked 'Create client certificate':

• Generate private key and CSR with Cloudflare
• Use my private key and CSR

Seeing as we'd gone through the effort of creating our own CA, I decided that we'd allow Cloudflare to take our CSRs (helpfully already on the server from our Ansible role), and create and sign some certificates so we could take advantage of restricting based on valid certificate at the edge.

Once I had the signed certificates from Cloudflare, I needed to go back to our server and create some combined PKCS #12 files that could be loaded into each of our browsers in order to authenticate with the Cloudflare edge. The following command uses the signed certificate from Cloudflare and the private key on the server.

openssl pkcs12 -export -out adam.p12 -in adam.cert.pem -inkey adam.key.pem

The benefit of this method is that while Cloudflare is able to authenticate our client certificates, they only hold what is publicly available so our private keys are never leaked to a third party. The benefit of this is that even if someone breaks into my Cloudflare account, they would not be able to make or retrieve valid client certificates.

The next step to this approach is to add a firewall rule that ensures all requests coming in to specific hostnames have a valid certificate. The below rule blocks requests that traverse Cloudflare that are not accompanied by a valid client certificate.

Ah but what about people who are sneaky and use your /etc/hosts method

The final step to this approach is to deny access to Nginx from anyone outside Cloudflare's IP ranges. As is well documented elsewhere on the internet, I took the Cloudflare IP list and configured Ansible to add the following to my Nginx configuration.

server {
    listen 443 ssl http2;
    server_name oursite.com;
    index index.html index.htm;

    include /etc/nginx/cloudflare-allow.conf;
    deny all;
    ssl_certificate     /etc/letsencrypt/live/oursite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/oursite.com/privkey.pem;
    ssl_verify_client on;
    ssl_client_certificate /etc/ssl/ca/certs/OurRoot_CA.crt;
    location / {
        proxy_pass http://localhost:3434/;
    }
}

# https://www.cloudflare.com/ips
# IPv4
allow 173.245.48.0/20;
allow 103.21.244.0/22;
allow 103.22.200.0/22;
allow 103.31.4.0/22;
allow 141.101.64.0/18;
allow 108.162.192.0/18;
allow 190.93.240.0/20;
allow 188.114.96.0/20;
allow 197.234.240.0/22;
allow 198.41.128.0/17;
allow 162.158.0.0/15;
allow 104.16.0.0/12;
allow 172.64.0.0/13;
allow 131.0.72.0/22;

# IPv6
allow 2400:cb00::/32;
allow 2606:4700::/32;
allow 2803:f800::/32;
allow 2405:b500::/32;
allow 2405:8100::/32;
allow 2a06:98c0::/29;
allow 2c0f:f248::/32;

This then allows us to get the best of all worlds:

• SSL certificates managed with Certbot and Let's Encrypt
• Our own CA to create client certificates without third party involvement
• Protection at the edge with client certificates
• Locking Nginx requests and responses to those coming from the edge so people can't bypass it

I'll have another blog post up in the coming weeks about how I then integrated NextCloud, Wiki.js, as well as a number of other custom services with Keycloak acting as an identity provider (IdP) to reduce the number of usernames and password in use.

Footnotes

1: We're not stupid enough to try to host our own mail.