# Fixing the AddTrust External CA Root Expiration

If you’re like me, you probably woke up on **Monday**, June 1st, and found some of your services suddenly failing, whether it’s your mobile app failing to connect to your API or your images suddenly failing to load.

In my case, some of our mobile apps (Android, specifically) were reporting errors trying to retrieve data from our APIs. After debugging for a bit I found that it was an SSL issue.

Opening the URL in question in Chrome or making requests on iOS, however, showed no issues, and I was greeted to the normal green lock icon representing a normal certificate validation.

**The issue:**

Modern browsers and HTTP clients like the iOS one have a better TLS trust verification strategy than older HTTP clients like cURL and the Android built-in one, they will build a chain of trust to the root instead of stopping and failing at the first expired certificate in the chain.

![Chrome happily validates the certificate chain, recognizing the newer USERTrust Root Certificate](https://cdn.hashnode.com/res/hashnode/image/upload/v1603180902460/yMinOfIB7.png)
Chrome happily validates the certificate chain, recognizing the newer USERTrust Root Certificate

![cURL, however, fails validation saying the certificate has expired](https://cdn.hashnode.com/res/hashnode/image/upload/v1603180922292/zcuqjewpn.png)
cURL, however, fails validation saying the certificate has expired

### So what’s the problem, and how do we debug further?

Well to investigate SSL issues we can rely on good old OpenSSL. You can use OpenSSL to fetch all certificate data for a given server by running:

```
# Hostname without protocol (no http/s)
openssl s_client -connect <hostname>:<port>
```

The output you will receive should contain a section containing the full list of certificates in the chain:

```
Certificate chain
 0 s: ### No.0 is your own certificate, followed by all intermediate and root certificates
 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=US/ST=DE/L=Wilmington/O=Corporation Service Company/CN=Trusted Secure Certificate Authority 5
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
```

Notice the **AddTrust** in the 1st and 2nd positions in the certificate chain. These certificates are now expired and are causing the **certificate has expired** errors in **cURL** and the Android HTTP client.

### How do we mitigate this?

The proper way to mitigate this is on the **server-side**, by updating the SSL configuration of the server, or the reverse-proxy if using TLS-termination.

You can check your certificate chain using [https://whatsmychaincert.com](https://whatsmychaincert.com). Just put the URL of your site there and it will spit out useful information about your certificate chain issues. You can even put in your [hostname](https://gis-qa.digital-farming.com/) (domain) and it will auto-remove the expired AddTrust certificates and produce a fixed `.crt` file you can put into your existing configuration as a drop-in replacement.

**If you’re not a server administrator** or you’re having issues with another platform, and just want to fix your client until the server admin picks up the slack, you have a few options:

On Linux-based client environments (Ubuntu and others), you can edit the **ca-certificates.conf** file:

```
/etc/ca-certificates.conf
```

In there you will find 2 references to **AddTrust**, a quick fix is to just prepend `!` before each line and run `update-ca-certificates` to apply the changes afterward.

On **macOS**, a simple way to fix this for me was to edit `/etc/ssl/cert.pem`, find the **AddTrust** certificate, and completely remove it.

Make sure you only delete lines between and including:

```
### AddTrust AB
-----END CERTIFICATE-----
```

This should fix your issues and cURL and other HTTP clients should now be happy to resolve the new **USERTrust Root Certificate** properly.

Hope you found this helpful and I saved you some headache, cheers!