Conversation
I think this might help startup time a little bit, since it looks like on linux at least the std implementation uses syscalls to determine the parallelism, whereas num_cpus was processing the /proc/cpuinfo file. It also removes another dependency.
|
this builds on #1410 |
sharkdp
left a comment
sharkdp
left a comment
There was a problem hiding this comment.
Thank you. Can you please add a ChangeLog with a reference to the bug ticket?
Set a limit of how many threads fd will use by default. On hosts that have a large number of cores, using additional threads has diminishing returns, and having large numbers of threads increases the setup cost. Thus we don't necessarily want to use the same number of threads as we have cores. Fixes: sharkdp#1203
tmccombs
force-pushed
the
limit-default-threads
branch
from
114bc76 to
5ee6365
Compare
| ## Changes | ||
|
|
||
| - The default number of threads is now constrained to be at most 16. This should improve startup time on | ||
| systems with many CPU cores. (#1203) |
There was a problem hiding this comment.
Maybe I'm missing something, but isn't MAX_DEFAULT_THREADS actually 20?
I think it would also be good to note that maximum in the CLI help.
There was a problem hiding this comment.
Ugh. I had a local change to set it to 16 instead of 20, but that didn't get included in the commit.
There was a problem hiding this comment.
We may want to tweak the limit again post #1422 anyway. I get better perf all the way up to 48 threads on my 24 core CPU and the startup overhead is minimal now
There was a problem hiding this comment.
Actually I just checked more carefully and the ideal thread count for me is 24 on my 24 core/48 thread CPU. I'm curious how this scales on different machines.
./fd-batch -u -j24 --search-path ~/code/bfs/bench/corpus/chromium ran
1.02 ± 0.04 times faster than ./fd-batch -u -j28 --search-path ~/code/bfs/bench/corpus/chromium
1.02 ± 0.03 times faster than ./fd-batch -u -j32 --search-path ~/code/bfs/bench/corpus/chromium
1.07 ± 0.03 times faster than ./fd-batch -u -j20 --search-path ~/code/bfs/bench/corpus/chromium
1.07 ± 0.03 times faster than ./fd-batch -u -j36 --search-path ~/code/bfs/bench/corpus/chromium
1.08 ± 0.03 times faster than ./fd-batch -u -j40 --search-path ~/code/bfs/bench/corpus/chromium
1.09 ± 0.02 times faster than ./fd-batch -u -j48 --search-path ~/code/bfs/bench/corpus/chromium
1.09 ± 0.02 times faster than ./fd-batch -u -j44 --search-path ~/code/bfs/bench/corpus/chromium
1.21 ± 0.02 times faster than ./fd-batch -u -j16 --search-path ~/code/bfs/bench/corpus/chromium
1.52 ± 0.03 times faster than ./fd-batch -u -j12 --search-path ~/code/bfs/bench/corpus/chromium
2.15 ± 0.04 times faster than ./fd-batch -u -j8 --search-path ~/code/bfs/bench/corpus/chromium
4 participants
Set maximum default threads
Set a limit of how many threads fd will use by default. On hosts that
have a large number of cores, using additional threads has diminishing
returns, and having large numbers of threads increases the setup cost.
Thus we don't necessarily want to use the same number of threads as we
have cores.
Fixes: #1203