Skip to content

[Bug]: SSH environment bootstrap cannot finish on slow links (240 s cap on a 183 MB archive, no resume) #36469

Closed as not planned

Description

@seankoji

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/desktop

Steps to reproduce

  1. On macOS, add a Linux x86_64 host as an SSH environment. The host has no ~/.t3/runtime/versions/<pinned-version>.
  2. Give the host a healthy but slow path to release-assets.githubusercontent.com (~100–160 KB/s measured).
  3. Connect. The desktop writes ~/.t3/ssh-launch/<id>/run-t3.next.<pid>, which downloads the pinned t3-<version>-linux-x64.tar.gz.
  4. Wait. The launch ends without a server; the staging directory is removed and the next connection starts the same download from zero.

Expected behavior

A slow but healthy download should be able to finish, or resume from the bytes already fetched. The archive is 183,016,791 bytes. A fixed 240 s total cap requires at least ~763 KB/s; below that it can never succeed. An interrupted attempt should not discard the partial archive.

Actual behavior

REMOTE_ARCHIVE_DOWNLOAD_SECONDS = 240 in packages/ssh/src/tunnel.ts is passed to curl --max-time 240 for the archive. The runner creates a fresh .staging-* directory per attempt and removes it on exit, so there is no resume. The attempt is also bounded by REMOTE_ARCHIVE_LAUNCH_TIMEOUT_MS = 900_000 (15 min), which is shorter than the ~19–28 minutes the download needs at the measured throughput.

After failed connections, ~/.t3/runtime/versions/ held .staging-* directories of 132,722,764 / 82,327,672 / 32,733,027 / 29,425,664 bytes and a held .0.1.10-nightly.20261113.4656.install.lock. The desktop reports the environment as unreachable while an existing T3 server on the host is healthy.

Impact

Blocks work completely

Version or commit

0.1.10-nightly.20261113.4656 (desktop and pinned remote runtime). main still contains the same constants.

Environment

  • Client: macOS 26.7.1 arm64, T3 Code (Nightly) 0.1.10-nightly.20261113.4656
  • Remote: Linux Mint 22.3, kernel 7.0.0-38-generic, x86_64
  • Throughput to release assets: ~100–160 KB/s (best measured ~164,800 B/s)
  • Remote curl: system 8.5.0; the remote PATH also has Homebrew curl 8.23.0

Logs or stack traces

Archive: 183,016,791 bytes
Required at 240 s: 762,570 B/s
Measured: 108,468–164,800 B/s
Partial staging dirs: 132,722,764; 82,327,672; 32,733,027; 29,425,664 bytes
Outer launch timeout: 900,000 ms

Relevant generated-script path:

T3_STAGING="$(mktemp -d "$HOME/.t3/runtime/versions/.staging-XXXXXX")"
trap 'rm -rf "$T3_STAGING" "$T3_LOCK"' EXIT
t3_fetch() {
  ... curl -fsSL --connect-timeout 30 --max-time "$3" "$1" -o "$2" ...
}
t3_fetch "$T3_RELEASE_BASE_URL/v$T3_ARCHIVE_VERSION/$T3_ARCHIVE" "$T3_STAGING/$T3_ARCHIVE" 240

Workaround

Preinstall the pinned runtime outside the capped path by resuming an abandoned staging download, verifying SHA-256 against SHA256SUMS, extracting with --strip-components=1, writing .install-complete, and moving the directory into ~/.t3/runtime/versions/<version>. After that the same launcher returns t3 --version in ~0.8 s.

Related

Activity

  1. juliusmarminge commented on Nov 14, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    On origin/main cbf0f0a, I rendered the real SSH runner with SshTunnel.buildRemoteT3RunnerScript (packages/ssh/src/tunnel.ts) and ran it in a real sh three times, as three connection attempts. Each run used a fake uname (Linux x86_64) and a fake curl that simulates a healthy but slow link: the full archive needs 1000 simulated seconds, curl honours --max-time by writing only max-time × rate bytes and exiting 28, and it supports -C/--continue-at. All three attempts failed with 'curl: (28) Operation timed out after 240 seconds' and then 'Could not download t3 …'. Each started from zero, because the runner makes a fresh mktemp .staging-* directory and its EXIT trap removes it. t3 --version never ran. This is the failing test on factory/repro-36469 (commit 34c4fc8). As a check, I temporarily replaced --max-time with --speed-limit/--speed-time in tunnel.ts; the new test and the existing download tests then passed, and I reverted the edit. I also tried an end-to-end run of the generated runner with real curl against a local server throttled to 100 KB/s, but my driver gave it the wrong release base URL (an extra /v), so it got a 404 at once and showed nothing. I did not drive the full desktop app through an SSH connection, so this is not a real-app reproduction. The code path and the failing test match the report exactly.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

  2. seankoji commented on Nov 16, 2026

    @seankoji
    Author

    Still reproduces, now with a real app + real host (today, 16 Nov 2026 AEDT), so this is the exact-app data your repro run was missing.

    Versions

    • Desktop: 0.1.10-nightly.20261113.4643, macOS 26.7.3 arm64
    • Remote: Ubuntu 24.04.5 on WSL2 (6.18.33.2-microsoft-standard-WSL2), x86_64, background service on 0.1.10-preview.638.b6b87b41
    • Transport: SSH environment (seank@<host>:2222)

    Steps

    1. Add the host as an SSH environment; its ~/.t3/runtime/versions/ holds only the preview build.
    2. Connect. The generated run-t3.next.<pid> fetches t3-0.1.10-nightly.20261113.4643-linux-x64.tar.gz (182,976,304 bytes) with curl -fsSL --connect-timeout 30 --max-time 240.
    3. Throughput to release assets measured ~110–190 KB/s from three separate hosts on the same LAN (Mac, the WSL host, and a wired NAS); a Cloudflare reference download on the same link did 9.5 MB/s. The archive needs ~763 KB/s to fit in 240 s.

    Actual

    • Every attempt ends in curl: (28); the desktop retries silently. The first failed .staging-* directory is dated 13 Nov 23:00 AEDT — three days of failed attempts across three pinned builds (...4630, ...4643, ...4656), leaving 48 abandoned .staging-* directories totalling 1.8 GB.
    • No progress or error reaches the UI. When the SSH agent also failed mid-attempt, the environment hung for the full REMOTE_ARCHIVE_LAUNCH_TIMEOUT_MS = 900_000 before reporting.
    • Meanwhile the host's own T3 server was healthy and answering on its port the whole time.

    Workaround, same as the report
    Resumable un-capped fetch, SHA-256 verified against SHA256SUMS (27445bd87cc908c42d16525de9b615a491292286f440c54a9b9f5b9352b968f9), extracted with --strip-components=1, .install-complete written, moved atomically into ~/.t3/runtime/versions/<version>/. The same launcher then returned t3 --version in 0.8 s, ensure-ssh-environment succeeded, and the -L 57767:127.0.0.1:3773 forward answered 200.

    Why this recurs
    The remote service cannot be pre-aligned with the desktop, so each desktop update mandates a new 183 MB fetch on connect. t3 update <that version> refuses it as a downgrade (nightly.N sorts below preview.N) without --allow-downgrade, and passing that would also switch the host's background service — neither is a reasonable ask for satisfying a version probe.

    Asks, in preference order

    1. Replace --max-time 240 with --speed-limit/--speed-time, or resume with -C - into a stable path so partial bytes survive (as [Bug]: Remote server update shows no download progress, caps the whole archive download at 10 minutes, and discards it if the attempt is interrupted #18113 did for the self-update path).
    2. Reuse an already-running service when its build differs instead of failing the environment.
    3. Land Desktop SSH environments ignore T3CODE_RELEASE_BASE_URL for the remote runtime download #31382 so T3CODE_RELEASE_BASE_URL reaches the SSH runner and a LAN mirror can serve it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions