Make apt fail fast, so the retry has something to retry

[danielhanchen]

The answer to "is installing packages broken?"

No. Nothing about this repo's dependencies. GitHub's Azure Ubuntu mirror was degraded, and apt has no timeout that stops it waiting on a dead connection.

Every one of the four stalled jobs ends the same way:

04:47:02  Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
05:16:29  ##[error]The operation was canceled.

Twenty-nine minutes and twenty-seven seconds of total silence, mid-fetch of a 126 kB index file. Two of the four hung on that exact file. What comes immediately before it is the other half of the story:

04:46:54  Ign:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
04:46:55  Ign:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
04:46:57  Ign:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
04:47:01  Ign:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
04:47:01  Hit:2 https://archive.ubuntu.com/ubuntu noble InRelease

The Azure mirror was already gone, so apt failed over through /etc/apt/apt-mirrors.txt to the public archive, which is not provisioned for this fleet.

apt did not consider any of that an error. Acquire::http::Timeout defaults to 120s and is an idle timeout, so a socket that is open and trickling never trips it. This is a long-running, well-documented condition on GitHub's runners, tracked upstream in actions/runner-images#675, #6486, #6894 and #7048.

What this changes

The helper now writes /etc/apt/apt.conf.d/99-unsloth-ci-fail-fast before its first attempt:

Acquire::Retries "3";
Acquire::http::Timeout "20";
Acquire::https::Timeout "20";
Acquire::ftp::Timeout "20";

A 29-minute hang becomes a ~1 minute failure. That matters for a reason beyond speed: the wall-clock kill is what orphans the dpkg lock, so an apt that gives up on its own is an apt we never have to kill, and the retry is retrying a failed command rather than resuming a half-killed one.

It is written by the helper rather than assumed of the runner image so that it also covers playwright install --with-deps, which shells out to apt without ever naming it.

Not pinning a mirror, and the evidence is why: in one stall Azure was dead and the public archive hung; in another Azure was serving fine and the transfer stalled partway through a 13.6 MB package. Neither is reliably better. The mirrorlist failover is already the right mechanism, it just needed to be allowed to give up.

Verification

The generated config is validated against apt itself, not by eye. apt-config dump round-trips all four options:

$ APT_CONFIG=apt-fail-fast.conf apt-config dump | grep -i acquire
Acquire::Retries "3";
Acquire::http::Timeout "20";
Acquire::https::Timeout "20";
Acquire::ftp::Timeout "20";

The helper's behaviour is exercised end to end with shims for sudo and fuser, so the best-effort fallbacks are covered rather than assumed: success on the first attempt, a command that always fails (3 attempts, all reported as "exited with status 1" rather than as a stall), a command that hangs (killed and reported as "did not finish within 2s"), recovery on attempt 2, the no-command case, and an unwritable config warning and continuing rather than aborting. All as expected.

Guard: test_the_helper_makes_apt_fail_fast asserts the four options are in the config the helper actually writes, that the timeout stays well under apt's useless 120s default, that it is applied before the retry loop (configured inside or after it, the first attempt is the one that stalls and it would run unconfigured), and that an unwritable config does not abort. Four mutants reintroduced and confirmed red.

One of those mutants initially survived, which is worth recording: the first version of the test searched the whole file, so it matched the option names in the header comment that explains them, and deleting the line that actually set one went unnoticed. It now parses the generated config.

tests/studio: 4457 passed, 4 skipped.