ProxyError with https proxy url

[Kludex]

Originally opened by @ElliottMoos on 2020-12-08 22:04:30 in encode/httpx

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

I get a ProxyError when setting TUNNEL_ONLY to true in httpx.Proxy() for an https proxy url.

To reproduce

import httpx

proxy = httpx.Proxy(
       f"https://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_HTTPS_PORT}",
       mode="TUNNEL_ONLY",
)

proxies = {
    "http://": proxy,
    "https://": proxy,
}

def main():
    with httpx.Client(proxies=proxies) as client:
        response = client.get('https://ifconfig.me/ip')
        print('httpx', response.text)

Expected behavior

httpx <ip>

Actual behavior

received exception of type <class 'httpx.ProxyError'>, url=https://ifconfig.me/ip,
msg=[SSL: PRE_MAC_LENGTH_TOO_LONG] invalid alert (_ssl.c:1091)

Environment

• OS: macOS
• Python version: 3.7.7
• HTTPX version: 0.16.1
• Async environment: No
• HTTP proxy: Yes
• Custom certificates: No

Additional context

Issue #897 is a similar issue but it is considered resolved.

[Kludex]

Originally posted by @florimondmanca on 2020-12-09T08:49:10Z

@ElliottMoos Hello!

First question Id have here is: does this reproduce if not setting a particular proxy mode?

The default behavior is to tunnel HTTPS requests, so given that you're requesting an HTTPS resource (ifconfig.me) I would expect the issue to remain if you just dropped the mode option and just passed:

proxies="https://..."

If that's the case then we can look at the dance happening when using an HTTPS proxy URL.

Also, are you able to make requests through this HTTPS proxy using another client, such as cURL or Requests?

[Kludex]

Originally posted by @ElliottMoos on 2020-12-10T20:32:01Z

@florimondmanca Hi! Thanks for checking in.

I can reproduce this without the TUNNEL_ONLY setting, which makes sense considering the functionality you described.

cURL and requests both work. Here is the output of a little test:

requests version: 2.25.0

Code:

proxies = {
    "http://": f"https://{PROXY_USER}:{PROXY_PASS}@<proxy>",
    "https://": f"https://{PROXY_USER}:{PROXY_PASS}@<proxy>",
}


def main():
    res = requests.get("https://ifconfig.me/ip", proxies=proxies)
    print("requests", res.text)

    with httpx.Client(proxies=proxies, timeout=60) as client:
        response = client.get("https://ifconfig.me/ip")
        print("httpx", response.text)


if __name__ == "__main__":
    main()

Output:

(.venv) bash-3.2$ python httpx_test.py 
requests [redacted]
Traceback (most recent call last):
(Abbreviated)
httpx.ProxyError: [SSL: PRE_MAC_LENGTH_TOO_LONG] invalid alert (_ssl.c:1076)

Full Traceback

Traceback (most recent call last):
  File "/Users/elliott.moos/Development/Personal/httpx_test/.venv/lib/python3.7/site-packages/httpcore/_sync/http_proxy.py", line 239, in _tunnel_request
    proxy_connection.start_tls(host, timeout)
  File "/Users/elliott.moos/Development/Personal/httpx_test/.venv/lib/python3.7/site-packages/httpcore/_sync/connection.py", line 177, in start_tls
    self.socket = self.connection.start_tls(hostname, timeout)
  File "/Users/elliott.moos/Development/Personal/httpx_test/.venv/lib/python3.7/site-packages/httpcore/_sync/http11.py", line 87, in start_tls
    self.socket = self.socket.start_tls(hostname, self.ssl_context, timeout)
  File "/Users/elliott.moos/Development/Personal/httpx_test/.venv/lib/python3.7/site-packages/httpcore/_backends/sync.py", line 50, in start_tls
    self.sock, server_hostname=hostname.decode("ascii")
  File "/Users/elliott.moos/.pyenv/versions/3.7.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/elliott.moos/Development/Personal/httpx_test/.venv/lib/python3.7/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.ConnectError: [SSL: PRE_MAC_LENGTH_TOO_LONG] invalid alert (_ssl.c:1076)

[Kludex]

Originally posted by @florimondmanca on 2020-12-10T21:33:58Z

Okay, so from the traceback you should, it fails during .start_tls(), which is the point where we wrap the TCP socket that was open by the proxy during the initial CONNECT request, and do the TLS handshake on it.

In your case, the proxy URL is an HTTPS one, so we do the CONNECT request via HTTPS, which means the TCP socket we get is already wrapped around TLS. So, it shouldn't be much of a surprise that doing a TLS handshake with the remote server on a connection that's encrypted is bound to lead to… problems. :-) At least our .start_tls()-based approach to tunneling is obviously too naive to handle this case.

Incidently, we don't actually have a proper test for + in HTTPCore (only HTTP/HTTP and HTTP/HTTPS).

We use pproxy there, so one way to possibly reproducing this would be to setup pproxy locally with TLS certs, and then issuing a request to an HTTPS resource through that.

[Kludex]

Originally posted by @florimondmanca on 2020-12-10T22:29:00Z

@ElliottMoos Are you sure the Requests example gets any traffic routed to the HTTPS proxy? I tried your snippet and I can actually put anything as the https proxy and the request will work, so…?

As for curl, I can confirm it works well for ifconfig.me. I forced --http1.1 so we're not bothered with HTTP/2 dances, and passed -v so we can see the full list of events happening:

# Generate proxy TLS certs
$ pip install trustme-cli
$ trustme-cli

# Start proxy
pproxy -d -l http+ssl://localhost:8483 --ssl server.pem,server.key

# Issue a request
curl -v --http1.1  --proxy-cacert client.pem -x https://localhost:8483 https://www.ifconfig.me/ip
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8483 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:`@STRENGTH`
* successfully set certificate verify locations:
*   CAfile: client.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: O=trustme v0.6.0; OU=Testing cert #D8C2Ox3CWaqKQLk-
*  start date: Jan  1 00:00:00 2000 GMT
*  expire date: Jan  1 00:00:00 2038 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: O=trustme v0.6.0; OU=Testing CA #6bn9BQ2f6wS7A88p
*  SSL certificate verify ok.
* Establish HTTP proxy tunnel to www.ifconfig.me:443
> CONNECT www.ifconfig.me:443 HTTP/1.1
> Host: www.ifconfig.me:443
> User-Agent: curl/7.54.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Connection: close
< 
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:`@STRENGTH`
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=www.ifconfig.me
*  start date: Dec  1 02:56:34 2020 GMT
*  expire date: Mar  1 02:56:34 2021 GMT
*  subjectAltName: host "www.ifconfig.me" matched cert's "www.ifconfig.me"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1D2
*  SSL certificate verify ok.
> GET /ip HTTP/1.1
> Host: www.ifconfig.me
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Thu, 10 Dec 2020 22:20:07 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 13
< Access-Control-Allow-Origin: *
< Via: 1.1 google
< 
* Connection #0 to host localhost left intact
37.120.158.20

So, as we can see:

1. curl connects to the proxy via TLS, doing the handshake and exchanging certs.
2. curl issues a CONNECT request for example.org:443, which succeeds.
3. curl does a new handshake with example.org over the tunnel.

[Kludex]

Originally posted by @florimondmanca on 2020-12-10T22:36:58Z

So, I think right now HTTPCore simply doesn't support HTTPS proxies, or wrongly so. Here's why.

Right now the ssl_context we pass to the proxy transport is used for both:

• Establishing the HTTP tunnel via a CONNECT request:

https://github.com/encode/httpcore/blob/62b47316983c5309c8f7b703139d45af1cd402d8/httpcore%2F_async%2Fhttp_proxy.py#L196-L200

• Doing the HTTPS handshake with the remote server via .start_tls():

https://github.com/encode/httpcore/blob/12a25fdb311cc3f0feb2b7ca7f0afab46989fe19/httpcore%2F_async%2Fhttp11.py#L83-L88

But as curl shows us, those two should be decoupled. In my setup above, I have custom certs for my proxy, but I'd expect the HTTP client to use my default TLS system certs to communicate with ifconfig.me.

So, along with the standard ssl_context=... option, we should have a proxy_ssl_context=... option that would be used for the initial CONNECT request.

[Kludex]

Originally posted by @florimondmanca on 2020-12-10T23:40:35Z

Well, actually, the strategy for curl and Requests for HTTPS requests to HTTPS proxies seems to be this:

• Requests: ignore the proxy altogether, just issue the HTTPS request through to the remote directly.
• curl: establish the tunnel through CONNECT, but issue a normal HTTPS request afterwards.

Put simply, neither really "tunnels" HTTPS requests through the proxy — they just bypass it in practice. Or am I missing something?

Neither strategy consists in "unwrapping" the TCP/TLS connection. Also, we don't need a new proxy_ssl_context option. If the HTTPS proxy uses custom certs, users can do httpx.create_ssl_context() and .add_verify_location(...), so that the ssl_context works for both the CONNECT request as well as any other requests to regular hosts.

I'm not sure which of curl's or Requests' behavior we should adopt; basically, do we want to issue a CONNECT even though we won't be using the connection eventually? For simplicity I'd say "no, just do HTTPS directly", so, Requests' approach. But… (?) 🤷‍♂️

[Kludex]

Originally posted by @florimondmanca on 2020-12-15T21:23:50Z

I'm realizing that urllib3 has explicit support for HTTPS + HTTPS, and it does so thanks to the stdlib's http.client implementation of HTTP Connect tunnels:

https://github.com/urllib3/urllib3/blob/cad4850614d86b4db56a785bc9f984eed342f8be/src/urllib3/connectionpool.py#L910

https://docs.python.org/3/library/http.client.html#http.client.HTTPConnection.set_tunnel

@ElliottMoos Curious if you could confirm that the following script works on your setup…?

import http.client
from base64 import b64encode

PROXY_HOST = ...
PROXY_HTTPS_PORT = ...
PROXY_USER = ...
PROXY_PASS = ...

userpass = (PROXY_USER.encode(), PROXY_PASS.encode()
auth = b64encode(b":".join(userpass)).decode()
PROXY_HEADERS = [("Proxy-Authorization", auth)]

conn = http.client.HTTPSConnection(PROXY_HOST, PROXY_HTTPS_PORT, headers=PROXY_HEADERS)
conn.set_tunnel("ifconfig.me")
response = conn.request("GET", "/ip")

Planning to take a look at what the stdlib's .set_tunnel() implementation does.

[Kludex]

Originally posted by @florimondmanca on 2020-12-15T23:25:15Z

Okay so, looking at urllib3, it turns out that HTTPS proxy support there is pretty new too — and this is the PR we probably want to take inspiration from: urllib3/urllib3#1923

[Kludex]

Originally posted by @florimondmanca on 2020-12-15T23:28:35Z

At this point I'm marking this as enhancement because it's quite clear that what we need is to add support for HTTPS proxies — that support is not there yet. :)

[Kludex]

Originally posted by @ElliottMoos on 2020-12-15T23:31:57Z

@florimondmanca I was just going to follow up with you on this.

It looks like the HTTPSConnection constructor doesn't have a headers kwarg available, but the set_tunnel method does (not that we'd throw the headers in there).

After reading this SO post, the ProxyManager resulted in a "connection reset" error which leads me to believe something in the auth mechanism isn't working, either on the proxy's end or in ProxyManager.

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(54, 'Connection reset by peer')))

I was getting similar errors with a modified version of your suggestion.

I followed the SO post's syntax exactly, fyi. What do you think?

[Kludex]

Originally posted by @florimondmanca on 2020-12-16T00:06:22Z

Did you try with the latest urllib3 version? I believe HTTPS proxy support was only added in 1.26 there. To me, ConnectionResetError sounds like the proxy hard-closing the connection because of a protocol error... such as a failure to complete the TLS handshake bc the client can't do it, or something else after the CONNECT request. 🤔 So yeah, I'd check you're on 1.26, and otherwise maybe ask urllib3 folks about that?

[Kludex]

Originally posted by @florimondmanca on 2020-12-21T16:04:11Z

@ElliottMoos Hey! I'm closing this off for now, as I've logged #1434 to track HTTPS proxies support properly. We can continue discussion about your setup and use case here, but that way we have a single issue to track HTTPS proxies support (which we obviously don't actually have proper support for right now). Thanks!