Skip to content

Use 2 parallel frontend threads by default on the nightly and dev channel - #162848

Merged
rust-bors[bot] merged 5 commits into
rust-lang:mainfrom
Kobzol:parallel-frontend-default-on-nightly
Nov 13, 2026
Merged

rust-bors[bot] merged 5 commits into
rust-lang:mainfrom
Kobzol:parallel-frontend-default-on-nightly

Conversation

@Kobzol

@Kobzol Kobzol commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 16, 2026
@petrochenkov petrochenkov self-assigned this Sep 16, 2026
@petrochenkov petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 16, 2026
@petrochenkov

Copy link
Copy Markdown
Contributor

CI fails due to UI tests, they should be compiled with --jobs-frontend=1 because they require reproducible diagnostics.

Comment thread compiler/rustc_session/src/config.rs Resolved
@Kobzol

Kobzol commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Wasn't there some flag that reordered the diagnostics, or made compiletest to compare them in an order agnostic way? If at all possible, I think that it would be better if we actually ran (at least most) UI tests under the parallel frontend.

@petrochenkov

petrochenkov commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

I think that it would be better if we actually ran (at least most) UI tests under the parallel frontend.

We do run them, on the test-x86_64-gnu-parallel-frontend job, in special mode.

Wasn't there some flag that reordered the diagnostics, or made compiletest to compare them in an order agnostic way?

Yeah, but that's not enough for some tests, and those tests are disabled with //@ ignore-parallel-frontend on that job (but run in the default mode).
Output blessing is also not supported in the parallel mode, but we want to keep it in the UI tests by default.

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 16, 2026
@Kobzol

Kobzol commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Can't we solve blessing by forcing 1 frontend thread when --bless is passed?

@petrochenkov

Copy link
Copy Markdown
Contributor

I'd rather keep the things as is, it's just good to have precise snapshot testing running on something reproducible.
Besides ui tests, --jobs-frontend=1 will need to be used on mir dump tests, and maybe some other test suites.

@Kobzol

Kobzol commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

I don't want to block this PR on that, but I find it a bit concerning that we aren't testing a bunch of nightly stuff on our CI. We already don't test the next trait solver, and now we also wouldn't be testing the parallel frontend, even though both would be enabled on nightly.

I'm happy to do the compiletest/bootstrap work to enable running the parallel frontend everywhere. Do you want that to be the default eventually (and perhaps have a separate one-off job that runs things sequentially, like we have the parallel job today)?

@petrochenkov

Copy link
Copy Markdown
Contributor

Do you want that to be the default eventually

For ui tests (.stderr snapshot testing) - no, I don't expect diagnostics to become reproducible anytime soon, if at all.
Technically we could make -Zui-testing reset the frontend threads to 1 (or otherwise slow down the compiler to serialize the diagnostics) instead of passing --jobs-frontend=1, but that doesn't seem to make much difference.

@petrochenkov

petrochenkov commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

If there are enough CI resources we could also run rustc twice on every ui test - once in the default mode (if it's parallel) and once in single-threaded mode for precise .stderr comparisons.
Most of //~ ERROR annotations in .rs files should match in parallel mode too, with exception of some tests involving query cycles.

and perhaps have a separate one-off job that runs things sequentially

So this ^^^ would mean limiting the precise .stderr comparison testing to some x86_64-linux-gnu.
I'm not sure, maybe it won't be that terrible, eventually, but it will skip some parts of the test suite.

@Kobzol

Kobzol commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Maybe I misunderstood how the current tooling works. I thought that:

  • There is some flag or logic that can compare the emitted diagnostics in an order-independent way. If that exists, then I think that we don't need precise comparisons? As long as the order is the only thing that changes, we can bless with 1 thread and then compare with multiple threads. Or do you consider the actual precise diagnostics ordering to be load-bearing across the whole test suite, rather than e.g. in a subset of tests which we would test with 1 thread (or both with 1 thread and with the parallel frontend)?
  • If there are tests that are truly broken under the parallel frontend, we can just mark them as such, and always run them with 1 thread (rather than skipping them entirely when the parallel frontend is enabled by default).

@petrochenkov

Copy link
Copy Markdown
Contributor

There is some flag or logic that can compare the emitted diagnostics in an order-independent way.

This is correct.

If that exists, then I think that we don't need precise comparisons?

I think that we do still need them, at least while the single-threaded compiler is the default.
Losing them is bad in the same sense as e.g. anonymizing line numbers in .stderr files (which we do), and similarly to the line numbers we can sacrifice them if there are strong reasons (like diff sizes for line numbers). So they are not exactly load bearing, but good to have, and should be kept if we can afford it.

@petrochenkov

Copy link
Copy Markdown
Contributor

There is some flag or logic that can compare the emitted diagnostics in an order-independent way.

Although that infra currently compares the output on per-line basis, not on per-diagnostic, so technically the diagnostics can be interspersed and messed up entirely, and the check will still pass.

@Kobzol
Kobzol force-pushed the parallel-frontend-default-on-nightly branch from dbe31dc to dc90826 Compare September 18, 2026 08:10
@rustbot rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc labels Sep 18, 2026
@rust-log-analyzer

This comment has been minimized.

@Kobzol
Kobzol force-pushed the parallel-frontend-default-on-nightly branch 2 times, most recently from 3003fd9 to 77bff57 Compare September 18, 2026 08:35
@rust-log-analyzer

This comment has been minimized.

@Kobzol

Kobzol commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Interesting. The obtain-borrowck.rs ui-fulldeps test now fails, likely because it override borrow check and stores some data in a thread local. But the interesting thing is that it only seems to happen when -Zthreads=1 is used. I presume that rustc behaves slightly differently when -Zthreads is unset, vs when it is set to 1?

rust-bors Bot pushed a commit that referenced this pull request Nov 9, 2026
Rollup merge of #165676 - aerooneqq:make-proxy-weak-reference-in-helper-thread, r=petrochenkov

Make `Proxy` weak reference as it seems that it is not dropped at all

Split from #165506 which makes `Proxy` droppable, which locally fixes the issue:
```rust
thread 'jobserver::jobserver_and_j' (126990) panicked at src/tools/cargo/tests/testsuite/jobserver.rs:411:10:

---- expected: tests/testsuite/jobserver.rs:405:27
++++ actual:   stderr
   1    1 | [WARNING] a `-j` argument was passed to Cargo but Cargo is also configured with an external jobserver in its environment, ignoring the `-j` parameter
   2    2 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
   3    3 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
        4 + make[1]: INTERNAL: Exiting with 1 jobserver tokens available; should be 2!
```

(see #162848 (comment)).

r? @petrochenkov
@aerooneqq

Copy link
Copy Markdown
Contributor

#165854 has merged, we can try once again, it should fix jobserver tests.

@Kobzol

Kobzol commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@bors try jobs=* nolimit

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Nov 12, 2026
…r=<try>

Use 2 parallel frontend threads by default on the nightly and dev channel


try-job: *
try-nolimit
@rust-bors

rust-bors Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: e0ea3ff (e0ea3ff45e588a0bf1c256d1370b3ff09a050b60)
Base parent: 8eb778d (8eb778d04094d3af4b26580cab327e0413eb1e02)

@Kobzol

Kobzol commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 13, 2026
@petrochenkov

Copy link
Copy Markdown
Contributor

Assuming it's ok to land this now from the infra point of view.
@bors r+ rollup=never

@rust-bors

rust-bors Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

📌 Commit da0f0ea has been approved by petrochenkov

It is now in the queue for this repository.

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 13, 2026
@Kobzol

Kobzol commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

It's not ideal w.r.t. the testing story, but also not very different from what we are doing with the next trait solver and Polonius. So let's see how it goes.

rust-bors Bot pushed a commit that referenced this pull request Nov 13, 2026
…r=petrochenkov

Use 2 parallel frontend threads by default on the nightly and dev channel



Part of https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Enabling.20the.20parallel.20frontend.20by.20default.20on.20nightly/with/624597688.

UI tests will still use one frontend thread by default.

CC @petrochenkov
@rust-bors

This comment has been minimized.

@JonathanBrouwer

Copy link
Copy Markdown
Member

@bors yield
Yielding to the rollup

@rust-bors

rust-bors Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Auto build was cancelled. Cancelled workflows:

The next pull request likely to be tested is #165918.

@rust-bors

This comment has been minimized.

@rust-bors

rust-bors Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

☀️ Test successful - CI
Approved by: petrochenkov
Duration: 3h 4m 25s
Pushing 6ca2b34 to main...

@github-actions

Copy link
Copy Markdown
Contributor
What is this? This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 80196be (parent) -> 6ca2b34 (this PR)

Test differences

Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 6ca2b34602bf2f3591ffa7359c166b6409a44b0a --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

  1. test-x86_64-gnu-miri: 54m 18s -> 1h 28m (+63.4%)
  2. test-x86_64-rust-for-linux: 37m 10s -> 1h (+62.6%)
  3. test-arm-android: 1h 16m -> 1h 53m (+48.8%)
  4. dist-ohos-armv7: 52m 47s -> 1h 15m (+43.3%)
  5. dist-x86_64-msvc-alt: 2h 52m -> 1h 40m (-41.9%)
  6. dist-x86_64-illumos: 1h 16m -> 1h 47m (+40.2%)
  7. test-x86_64-gnu-llvm-22-3: 1h 25m -> 1h 59m (+39.3%)
  8. dist-loongarch64-musl: 2h 6m -> 1h 19m (-36.8%)
  9. dist-x86_64-mingw: 2h 50m -> 1h 47m (-36.8%)
  10. dist-i686-linux: 1h 56m -> 1h 15m (-35.1%)
How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

@rust-timer

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (6ca2b34): comparison URL.

Overall result: no relevant changes - BENCHMARK(S) FAILED

@rustbot label: -perf-regression

❗ ❗ ❗ ❗ ❗
Warning ⚠️: The following benchmark(s) failed to build:

  • tokio-1.53.1
  • runtime:nbody
  • Job failure
  • serde_derive-1.0.219
  • projection-caching
  • syn-2.0.101-new-solver
  • typenum-1.18.0
  • regex-automata-0.4.8
  • wg-grammar-new-solver
  • stm32f4-0.15.1
  • unify-linearly
  • large-workspace
  • deeply-nested-multi
  • html5ever-0.31.0-new-solver
  • diesel-2.2.10-new-solver
  • token-stream-stress
  • html5ever-0.31.0
  • ripgrep-14.1.1-nll
  • unicode-normalization-0.1.24
  • runtime:fmt
  • runtime:hashmap
  • serde-1.0.219
  • deep-vector
  • clap_derive-4.5.32-nll
  • libc-0.2.172
  • cranelift-codegen-0.119.0
  • enum-stress
  • clap_derive-4.5.32
  • hyper-1.6.0
  • nalgebra-0.33.0-new-solver
  • bitmaps-3.2.1-new-solver
  • helloworld
  • runtime:text-search
  • serde-1.0.219-nll
  • include-blob
  • ucd
  • nalgebra-0.33.0
  • issue-46449
  • runtime:raytracer
  • runtime:svg
  • typenum-1.18.0-new-solver
  • eza-0.21.2
  • bitmaps-3.2.1
  • syn-2.0.101
  • runtime:nes
  • deref-sidebar-stress
  • cargo-0.87.1
  • runtime:arc-rc
  • wf-projection-stress-65510
  • externs
  • await-call-tree
  • serde-1.0.219-new-solver
  • issue-58319
  • derive
  • runtime:compression
  • unused-warnings
  • deeply-nested-multi-new-solver
  • ripgrep-14.1.1
  • tt-muncher
  • helloworld-tiny
  • image-0.25.6
  • ripgrep-14.1.1-tiny
  • wg-grammar
  • diesel-2.2.10
  • runtime:css
  • many-assoc-items
  • runtime:parsing
  • regression-31157
  • tuple-stress
  • issue-88862
  • coercions
  • runtime:bufreader
  • ctfe-stress-5

❗ ❗ ❗ ❗ ❗

cc @rust-lang/wg-compiler-performance

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

Results (secondary -2.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
3.2% [3.2%, 3.2%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-3.9% [-5.9%, -2.6%] 3
All ❌✅ (primary) - - 0

Cycles

This perf run didn't have relevant results for this metric.

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.761s -> 468.104s (-4.42%)
Artifact size: 406.16 MiB -> 406.16 MiB (0.00%)

@Kobzol

Kobzol commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Funnily enough, this looks like a coincidence. Looks like the ARM machine was restarted just before this PR, and its perf event config got lost (I thought I resolved this already, grr).

@Zalathar

Copy link
Copy Markdown
Member

Since this landed, a bunch of stuff has been encountering weird assertion failures in LLVM.

I don't have any specific evidence that this change is responsible, but given the timing it feels like a plausible suspect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-clippy Relevant to the Clippy team. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants