Fixing HTTPSConnectionPool “certificate verify failed” error in FreeNAS 11.3

FreeNAS 11.x is no longer receiving package updates due to the major version update to TrueNAS 12.x. A side effect is that the CA certificates database from systems on 11.x is now outdated and contains expired CA certificates. This causes errors such as the following when attempting to create jails or update the OS:

Update server could not be reached

HTTPSConnectionPool(host='update-master.ixsystems.com', port=443): Max retries exceeded with url: /FreeNAS/trains.txt (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

Traceback

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
cnx.do_handshake()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1806, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1546, in _raise_ssl_error
_raise_current_error()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

[ ... ]

On almost any other OS you could just update the package, but since FreeNAS doesn’t enable pkg for the base OS this puts you in a catch 22 situation: you need to talk to a remote server over HTTPS to get the updates, but you need the certificate database update to talk to the remote server.

There are solutions posted online, but I found that they were mostly incomplete and the complete fix was spread over a number of posts.

To avoid any doubt: this solution should be applied to the FreeNAS base OS, rather than within a jail. If you’re getting SSL errors like the above from a Python application in a jail, you may need to repeat the steps for both the base OS and jail. However, I have not validated these steps for use within jails and therefore cannot guarantee that they will work or be safe in such a context.

(side-note: I strongly recommend applying the fixes described here over SSH or at the local console, rather than through the shell in the web UI. in the event that you make a mistake and break something, it’s far easier to recover if you’ve got an active SSH session or direct access to the local console)

The root certificate database you need is ca-root-nss.crt. Unfortunately the source repository for this database file is non-trivial, and as far as I can tell they don’t ship first-party built versions of the database file. As such, the path of least friction is also somewhat fraught: get the built version from someone else. I downloaded it onto my FreeNAS system as follows:

wget https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src

At the time of writing, this extracted copy is from nss-3.71, whereas the one on my system was from nss-3.46.

Downloading a root CA database from some random website is a terrible idea from a security perspective, because you’re trusting that they didn’t inject their own CA or modify something. However, I searched around and found a few different copies of this file from various sources, and they all matched this copy exactly. Not a perfect guarantee, but good enough for this quick hack so that updates work again. You can use the sha256 command on FreeNAS/BSD or the sha256sum command on Linux to generate hashes of the files for comparison if you want to check for yourself.

If, for some reason, you can’t get the file to download with wget on your FreeNAS system, you can download the file on another computer and transfer it to your FreeNAS box via SCP/SFTP, a shared drive, or USB stick.

There are several locations that this certificate database needs to be placed. Most posts online only mention one or two, but there are actually several locations that need to be inspected and updated.

The main one, used by the system, is as follows:

/usr/local/share/certs/ca-root-nss.crt

Rename this to ca-root-nss.crt.old and copy the downloaded ca-root-nss.crt.src in there as ca-root-nss.crt. For example:

cd ~
wget https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src
cd /usr/local/share/certs/
mv ca-root-nss.crt ca-root-nss.crt.old
cp ~/ca-root-nss.crt.src ./ca-root-nss.crt

There are some other locations that should contain symlinks to this file, but in practice they may not be symlinked and instead contain a separate copy of the database file. The file paths are as follows:

/etc/ssl/cert.pem
/usr/local/etc/ssl/cert.pem

Both of these should be symlinks to /usr/local/share/certs/ca-root-nss.crt, but this is not a guarantee. Double check:

root@goliath[~]# ls -l /etc/ssl/
lrwxr-xr-x  root  wheel     38  cert.pem -> /usr/local/share/certs/ca-root-nss.crt
-rw-r--r--  root  wheel  10847  openssl.cnf
-rw-r--r--  root  wheel   2156  truenas_cacerts.pem

root@goliath[~]# ls -l /usr/local/etc/ssl
-rw-r--r--  root  wheel  789910  cert.pem
lrwxr-xr-x  root  wheel      38  cert.pem.sample -> /usr/local/share/certs/ca-root-nss.crt

In my case, /etc/ssl/cert.pem was correctly symlinked to the copy in /usr/local/share/certs, but /usr/local/etc/ssl/cert.pem was not for some reason. If you also run into this, the fix is easy - just remove the copy and recreate it as a symlink to the correct file:

mv /usr/local/etc/ssl/cert.pem /usr/local/etc/ssl/cert.pem.old
ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/etc/ssl/cert.pem

Checking again, we can see that the file is now symlinked to the correct location:

root@goliath[~]# ls -l /usr/local/etc/ssl
-rw-r--r--  root  wheel  38  cert.pem -> /usr/local/share/certs/ca-root-nss.crt
lrwxr-xr-x  root  wheel  38  cert.pem.sample -> /usr/local/share/certs/ca-root-nss.crt

This is where most of the solutions I found online ended. So far we’ve fixed the CA database for programs and OS components that link against OpenSSL, but we haven’t actually fixed the issue for the FreeNAS middleware. This is because the middleware is written in Python, and Python has its own certificate database as part of one of its own packages:

root@goliath[~]# ls -l /lib/python3.7/site-packages/certifi/

-rw-r--r--  root  wheel      52  __init__.py
-rw-r--r--  root  wheel      41  __main__.py
drwxr-xr-x  root  wheel       8  __pycache__
-rw-r--r--  root  wheel  282085  cacert.pem
-rw-r--r--  root  wheel     218  core.py

While you could use a symlink here, I recommend not doing so, just in case a future Python package update modifies the file, propagating changes back to the OS and potentially breaking things. Instead, just rename the existing file and copy the updated version in:

cd /lib/python3.7/site-packages/certifi/
mv cacert.pem cacert.pem.old
cp /usr/local/share/certs/ca-root-nss.crt ./cacert.pem

This should ensure that everything has been updated. All that’s left to do is restart the middleware service:

service middlewared restart

This will terminate any sessions you have open in the web UI, so you’ll have to log in again. But when you do, the error should go away and you should be able to get jails and upgrades working again!

I’ve written a short and hacky all-in-one shell script to do all of these steps. I’ll offer this warning: it is far better to do these steps one by one rather than just running this blindly, so you can check whether the symlinks already exist. However, if you really do just want to run a thing to fix it, this should work:

#!/usr/bin/env zsh

# halt on error
set -e

# get the database
wget https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src

# fix the main database file
mv /usr/local/share/certs/ca-root-nss.crt /usr/local/share/certs/ca-root-nss.crt.old
cp ./ca-root-nss.crt.src /usr/local/share/certs/ca-root-nss.crt

# fix /etc/ssl/
mv /etc/ssl/cert.pem /etc/ssl/cert.pem.old
ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem

# fix /usr/local/etc/ssl/
mv /usr/local/etc/ssl/cert.pem /usr/local/etc/ssl/cert.pem.old
ln -s /usr/local/share/certs/ca-root-nss.crt /usr/local/etc/ssl/cert.pem

# fix python
mv /lib/python3.7/site-packages/certifi/cacert.pem /lib/python3.7/site-packages/certifi/cacert.pem.old
cp /usr/local/share/certs/ca-root-nss.crt /lib/python3.7/site-packages/certifi/cacert.pem

# restart the middleware
service middlewared restart