Python unable to see CA certificates in Fedora/Rocky/Alma/RHEL systems

[nonell-viavi]

Hi,

I think there is an issue when attempting to use this precompiled binaries on a rhel-based system.
I've read the quirks section, and I haven't seen a reference to this problem, that is why I'm raising an issue.

The reason we care about this funcionality is because we're trying to use this python + poetry to get "portable" dev environments accross a bunch of developers (using mise).

Issue description

The following code reproduces the problem on a rhel-based system:

docker run -it --rm rockylinux:8 bash -c \
  "dnf update -y && dnf install -y wget ca-certificates && \
  wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  ./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""

Logs

Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'

cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>]  26.14M  8.59MB/s    in 3.0s

2024-05-06 08:57:30 (8.59 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]

Traceback (most recent call last):
  File "/python/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/python/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/python/lib/python3.10/http/client.py", line 1455, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/python/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/python/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/python/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/python/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/python/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/python/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/python/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/python/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/python/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>

But the same python program, urlopen works just fine on a debian-based system.

docker run -it --rm debian:12 bash -c \
  "apt update -y && apt install -y wget ca-certificates && \
  wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  ./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""

Logs

...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'

cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>]  26.14M  8.63MB/s    in 3.0s

2024-05-06 08:53:54 (8.63 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]

b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xed}\xebr\x1bG\xb2\xe6\x7fE\xcc;\x94\xa1X\x91\x1a\xab\x01\x82\xe0M\x14\t\x8f$S2=\xba\xd0\xa6d\x9d\xb3>\x0eE\xa3\xd1\x00\x9a\x04\xba\xa1\xbe\x90\x84g\xe6\x01\xce\x9f}\x85}\xc5}\x84\xfd2\xab\xaa\xbb\xfa\n@\xe4\xe8\xd8\x11\x9a\t\x8b@\xa3\xba.Yy\xaf\xcc\xac\xa3o\x86\x81\x13/\xe6'

Other things I've tried

I've tried installing pip-system-certs and certifi, using the provided pip, but neither of them fixed the issue.

Thank you for reading this far, and let me know if there is something else I can do to help. I'm stuck and unable to fix this by myself (I think).

[dirkpetersen]

I ran into the exact same thing on Python 3.10 (3.8 worked fine) , and used monkey patching in the sitecustomize.py to enable urllib ssl globally per ChatGPT recommendation:

vi $(python -c "import site; print(site.getsitepackages()[0])")/sitecustomize.py

paste this

import ssl
try:
    import certifi
    # Define a function that returns a default SSL context with certifi's CA bundle
    def create_certifi_context(purpose=ssl.Purpose.SERVER_AUTH, *, cafile=None, capath=None, cadata=None):
        return ssl.create_default_context(purpose, cafile=certifi.where(), capath=capath, cadata=cadata)
    # Set the default SSL context creation function to use certifi's CA bundle
    ssl._create_default_https_context = create_certifi_context
except:
    print('certifi package not installed')

[keichi]

I also ran into this. Explicitly specifying the SSL certificate file seems to fix the issue:
export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt

[zanieb]

Is this a problem of the standalone builds? i.e., is this reproducible with a local build with pyenv? I wonder if this is just a difference in how the certificates are configured on the systems?

[keichi]

Just built CPython 3.11.9 with pyenv and tested but this issue didn't happen.

After some digging, I've found that RHEL-based distros store their root CA certificate bundle as /etc/ssl/certs/ca-bundle.crt while Debian-based distros store it as/etc/ssl/certs/ca-certificates.crt. Also these paths seem to be baked in when OpenSSL is built. So yes, this comes from the different CA bundle paths between distros.

[paveldikov]

This is a pretty serious issue for us too -- heavy RHEL users. For RHEL9 it sort-of-works due to RedHat adding a compatibility symlink, but not so for 7 and 8.

Asking users to export env variables as a workaround is not viable.

Would it be possible to avoid statically linking libssl and friends for the dynamic builds? I think it is understandable for the static builds, but for the dynamic ones it is quite misleading. There is however the complication of having to support differing OpenSSL major versions...

Alternatively, patch OpenSSL's path detection to also support known trust store paths from a wide sample of Linux distributions?