Nginx Configuration w/ SSL Certs

Or "How I messed up hosting two ssl certs on one server"

So this story comes out of my making a few mistakes when setting up my personal site. Long story short: I'd had my personal site running on the same server for cost-cutting, and my morals finally caught up with me so I needed to split my projects out and reconfigure.

The Setup

I bought a second Digital Ocean droplet (512 MB Memory / 20 GB Disk / NYC3 - Ubuntu NodeJS 6.11.0 on 16.04 for those curious). The goal was to move over my personal site as well as this blog (blog.codelemur.com). The first thing I set up was astoellis.me which was 90% hassle-free! Simple as setting up a new user, adding my repository & SSH to github privileges, and then pulling & running the server to host my simple HTML & CSS site. Almost no JS, very simple, but I'm planning on revamping later.

I used certbot to install an SSL certificate and make sure that all my stuff was securely served. I used their basic configuration, resulting in something like the following:

server {
    listen 443 ssl;
    server_name astoellis.me www.astoellis.me;
    ssl_certificate path/to/cert
    ssl_certificate_key /path/to/key



    location / {
        gzip on;
        proxy_pass <host:port>;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    location ~/.well-known {
        allow all;
    }


}

 server {
   gzip on;
   listen 80;

   server_name astoellis.me www.astoellis.me;
   return 301 https://$host$request_uri;

}

Pretty straight-forward, made sense to me, so I kinda ignored it.

Pt 2: The Downfall

The second project of the day was to move over blog.codelemur.com. As you can now tell, it's actually running and serving, so something must have gone right. But let's not focus on that, let's focus on how wrong it went, since that's more educational.

Ghost

I followed the instructions found on the CLI docs almost to a T (I did a few things wrong by accident, such as creating the mysql user myself and not letting ghost set that up, but whatever, it worked out).

The docs are super easy to follow, and besides a few hiccups here and there (trying to reinstall in a non-empty directory), it went swimmingly and I was able to at least verify that it was running on the port I expected, and with all the security I wanted.

One of the really nifty things that the Ghost CLI allows you to do is to have them set up the SSL & nginx configuration for your site. Check it out here if you're interested. Essentially they'll add a <your-blog>.conf to the /etc/nginx/sites-available/ directory, run the certbot stuff, and install the keys in ghost's directory.

The problem

Everything seemed to be good to go. Ghost was running and I was feeling proud of myself.

In my hubris, I decided to go ahead and check out my site.

Oh no. that's not at all what I want.

To Google I went, searching for Ghost-specific problems. There weren't a whole ton, and I was getting mixed messages. Maybe I needed to check out my base nginx config? Maybe I needed to re-check my cert or uninstall and recertify.

I ended up running certbot for blog.codelemur.com and getting a new certificate, changing the path to the key/pem, and still was getting no improvement. The most infurtiating part of it was that even when I was checking the site's status from multiple sources, they all stated that it looked like a working configuration, and that in theory, the page should be loading.

SSL Labs
Running the SSL Labs check to see if it was a bad SSL configuration, I was getting nothing useful. I was getting A+ certification, and none of the things looked shifty, except for a red herring I encountered. Somehow (and I'm still working on this, I'm quite novice), there was a second certificate being sent to SSL labs from astoellis.me This was pretty perplexing, but now I'm guessing it comes back to my multiple SSL configurations.
OpenSSL s_client
One of the forums/blogs also suggested trying to diagnose via the openssl s_client. I ran openssl s_client -connect blog.codelemur.com:443 to see what was going on. The client connected, the chain looked like this:

Certificate chain
 0 s:/CN=astoellis.me
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

I felt pretty dumb, as we were going on about 4 hours of me having no clue what was going on.

The Fix

So finally I ended up here.

The problems I had encountered was with having multiple SSL blocks being used in my nginx configuration. astoellis.me was running just fine, but the issue came about with some of the enormously useful SSL params that were being loaded into sites-enabled/blog.conf. Here's the configuration:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;

ssl_dhparam /var/www/ghost/system/files/dhparam.pem;

Now most of these hints pointed towards ssl_stapling being enabled. I checked my nginx version, I checked the configuration up and down, but what I didn't realize is that I now had a conflict.

If you have two server blocks that handle SSL, and one of them has ssl_stapling on; and the other has it enabled (or no definition), Nginx doesn't know how to treat it. I'm still a little fuzzy on this, but that's what I gathered from reading a few posts.

Essentially, by having a mismatch in the ssl_session_tickets parameter, it ended up creating garbled handshakes, resulting in the browser considering it SSL_PROTOCOL_ERROR. Voila.

To fix this, I moved it from the bottom-level specificity and up into the main configuration file under the http{} block. This allowed it to be set for both sites, and like magic, my site was back up and running after a quick nginx test & reload.

tl;dr

If you encounter SSL_PROTOCOL_ERROR and are running multiple https-enabled server blocks check your SSL conmfiguration for ssl_session_tickets and make sure they are either enabled in the same way, or simply move it to the top level.

I hope this helps someone at some point
¯\(ツ)/¯

-A