page.goto: net::ERR_TUNNEL_CONNECTION_FAILED When using browser-pool and https proxy

[GuiKlesseMiranda]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/browser-pool

Issue description

I am facing an issue when using BrightData HTTPS proxies together with BrowserPool.

When I use plain Playwright and configure the BrightData HTTPS proxy directly in the launch options, everything works as expected.

However, when using the same proxy through BrowserPool, even when the proxy is configured in the launch, all HTTPS navigations fail with the following error:

page.goto: net::ERR_TUNNEL_CONNECTION_FAILED

I have also tried setting ignoreHTTPSErrors: true, but the problem still occurs.

In other words, the HTTPS CONNECT tunnel is not being established correctly when the browser is created via BrowserPool, while the same proxy works fine with Playwright without the pool.

Just informing you that when using the same proxy as HTTP, it works, but there is a scenario in which I need to use HTTPS.

Code sample

const page = await browserPool.newPage({
  proxyUrl: 'https://USER:PASSWORD@HOST:PORT',
});

await page.goto('https://example.com');

Package version

3.15.3

Node.js version

v22.14.0

Operating system

MACOS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[AndreiMaelstrom]

Hi

I am seeing the same behaviour when trying to use an https rotating proxy. Standalone playwright works correctly on the same machine. Looks like a crawlee-linked issue somehow.

L.E.: I am not using the browser pool, but rather static chromium.

Node version: v24.12.0
Crawlee: 3.15.3
Playwright: 1.58.1
Package manager: pnpm

[barjin]

Hello, and thank you both for your interest in this project.

Unfortunately, I wasn't able to reproduce this issue locally. It's possible that the proxy module in Crawlee (implemented with Node's native HTTP(S) module) has different TLS requirements than, e.g., Chromium. This could lead to errors like the one you're mentioning. See that BrightData proxies can try to establish TLS connections with, e.g., self-signed certificates (see docs).

Without a better repro, I'm afraid I cannot provide more help right now. Could you please try capturing the traffic with Wireshark and sending a screenshot of the failed request?

[GuiKlesseMiranda]

I don't think it will be easy solvable

[barjin]

Thank you for the details @GuiKlesseMiranda . The connection to the HTTPS proxy server couldn't be established because of a TLS error.

Note that Crawlee-managed browsers will not connect to the proxy directly, but rather through a local proxy-chain instance (this is because of an old issue with authenticating to proxies in Puppeteer / Playwright, see e.g. here). This local proxy-chain instance will connect to the proxy server using Node's native https module, which is likely throwing this error.

Note that in proxy-chain, you can choose to ignoreUpstreamProxyCertificate (see here), but this option is not exposed in Crawlee. We can add this option to Crawlee's interfaces, but until then, your only option is likely patch-package or similar.