If you’re like me, you probably woke up on Monday, June 1st, and found some of your services suddenly failing, whether it’s your mobile app failing to connect to your API or your images suddenly failing to load.
In my case, some of our mobile apps (Android, specifically) were reporting errors trying to retrieve data from our APIs. After debugging for a bit I found that it was an SSL issue.
Opening the URL in question in Chrome or making requests on iOS, however, showed no issues, and I was greeted to the normal green lock icon representing a normal certificate validation.
Modern browsers and HTTP clients like the iOS one have a better TLS trust verification strategy than older HTTP clients like cURL and the Android built-in one, they will build a chain of trust to the root instead of stopping and failing at the first expired certificate in the chain.
Chrome happily validates the certificate chain, recognizing the newer USERTrust Root Certificate
cURL, however, fails validation saying the certificate has expired
So what’s the problem, and how do we debug further?
Well to investigate SSL issues we can rely on good old OpenSSL. You can use OpenSSL to fetch all certificate data for a given server by running:
# Hostname without protocol (no http/s) openssl s_client -connect <hostname>:<port>
The output you will receive should contain a section containing the full list of certificates in the chain:
Certificate chain 0 s: ### No.0 is your own certificate, followed by all intermediate and root certificates 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root 3 s:/C=US/ST=DE/L=Wilmington/O=Corporation Service Company/CN=Trusted Secure Certificate Authority 5 i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
Notice the AddTrust in the 1st and 2nd positions in the certificate chain. These certificates are now expired and are causing the certificate has expired errors in cURL and the Android HTTP client.
How do we mitigate this?
The proper way to mitigate this is on the server-side, by updating the SSL configuration of the server, or the reverse-proxy if using TLS-termination.
You can check your certificate chain using https://whatsmychaincert.com. Just put the URL of your site there and it will spit out useful information about your certificate chain issues. You can even put in your hostname (domain) and it will auto-remove the expired AddTrust certificates and produce a fixed
.crt file you can put into your existing configuration as a drop-in replacement.
If you’re not a server administrator or you’re having issues with another platform, and just want to fix your client until the server admin picks up the slack, you have a few options:
On Linux-based client environments (Ubuntu and others), you can edit the ca-certificates.conf file:
In there you will find 2 references to AddTrust, a quick fix is to just prepend
! before each line and run
update-ca-certificates to apply the changes afterward.
On macOS, a simple way to fix this for me was to edit
/etc/ssl/cert.pem, find the AddTrust certificate, and completely remove it.
Make sure you only delete lines between and including:
### AddTrust AB -----END CERTIFICATE-----
This should fix your issues and cURL and other HTTP clients should now be happy to resolve the new USERTrust Root Certificate properly.
Hope you found this helpful and I saved you some headache, cheers!