Python build standalone fails to download free-threaded builds

[sureshjoshi]

Multiple items at play here:

1. version.json might be a few months stale (not a big deal, just needs a regen)
2. Documentation for how to add URL-based PBS seems to be incorrect (order of URL vs SHA/length)
3. Some PBSes are zst, which don't seem to be "officially" captured in our code, but it may work anyways as a result of what we shell out to (this was hard to tell, as it's mixed in with other errors and I couldn't even make it to a sandbox) -

   pants/src/python/pants/core/util_rules/archive.py

   Line 168 in 3782956

   is_tar = archive_suffix.endswith(

4. The free-threaded builds (and some other flavours) provide extra directories, not just the install dir - so our copying might be held up here too as it appears to expect the install_only or install_only_stripped gz files.

This can all be fixed pretty easily, but it begs the question of whether we should just use uv as an external tool and ask it to download the python versions to our named_caches (since astral owns the PBS repo).

Alternatively, science also does this better - both as a CLI tool, and here is the associated code which handles matching against different flavours: https://github.com/a-scie/lift/blob/24af473e116eb780350719f61bcdaa01b40ccf19/science/providers/python_build_standalone.py#L535

In any case, lots of solutions...

[gayanMatch]

Thanks for the detailed write-up @sureshjoshi. I took a look at the current PBS implementation and wanted to share some thoughts on the proposed solutions.

I think direct fixes to the existing extraction logic is the best path forward. Here's my reasoning:

The required changes are fairly contained:

1. Zstandard support - Add .tar.zst to the archive format detection in src/python/pants/core/util_rules/archive.py. The extraction logic already delegates to shell commands, so it mainly needs to recognize the format and invoke tar --zstd or zstd -d piped to tar.
2. Directory structure handling - Update the extraction/installation logic in src/python/pants/backend/python/providers/python_build_standalone/rules.py to handle the different directory layout of free-threaded builds rather than assuming the python/ root from install_only variants.
3. Version data - Regenerate versions_info.json to include recent releases and free-threaded builds.
4. Documentation fix - Straightforward correction to the URL parameter ordering.

I think using uv/science could introduce some additional issues.
While both tools handle PBS variants well, adding them as external dependencies would:

• Introduce version compatibility concerns
• Create coupling to tools that may evolve independently
• Potentially conflict with Pants' hermetic caching model
• Add complexity for what are ultimately surgical fixes to existing code

The external tool approach would make more sense if PBS variants were proliferating rapidly or the extraction logic became significantly more complex, but for now direct fixes seem simpler and more aligned with Pants' architecture.
I'm happy to help with implementation if this approach sounds reasonable.

[cburroughs]

I rely on the Python Build Standalone backend at $DAYJOB and would be happy to be tagged on PRs.

[sureshjoshi]

I've wanted to basically solely rely on Pants to provide everything in my setups, but I'd never been able to get this running fully in the past. It'd been a while since I tried this, though.

Gonna mock up some options later this month after Benjy's NG PRs.