Skip to content

ii.filesMerge: did O(n^2) re-merge of sequences (chunks of same key)#19680

Merged
AskAlexSharov merged 7 commits into
release/3.4from
alex/ef_reset4_s_34
Mar 6, 2026
Merged

ii.filesMerge: did O(n^2) re-merge of sequences (chunks of same key)#19680
AskAlexSharov merged 7 commits into
release/3.4from
alex/ef_reset4_s_34

Conversation

@AskAlexSharov

Copy link
Copy Markdown
Collaborator

Story: i noticed on profiler that during InvertedIndex files merge deriveFields() fun did many re-allocs. Initially i had intent to 2x over-alloc to amortize it - but realized that we doing multiple merges of same key (multiple chunks from multiple small files) - each such merge produced "bigger sequence". So, clearly higher-level logic of merge files is wrong and need merge all chunks at once.


Before: incremental pairwise merge — for a key in N files, reads C + 2C + 3C + ... + NC = O(N²·C) elements and calls EliasFano.ResetForWrite(→ deriveFields → make([]uint64, ...)) N times, allocating a new backing array each time since the merged size always exceeds the previous capacity.

After: single-pass accumulation — collects all N items for the current key, computes totalCount and maxOff in one scan, initialises the builder once with the correct size (so deriveFields only allocates once per key at the right capacity), then iterates the N files in ascending txNum order adding all values in a single pass. O(N·C) reads, O(1) allocations per key.

BenchmarkInvertedIndexMergeFiles:

  ┌───────┬─────────┬─────────┬─────────┐
  │ files │ before  │  after  │ speedup │
  ├───────┼─────────┼─────────┼─────────┤
  │ 4     │ 840 µs  │ 763 µs  │ 1.1x    │
  ├───────┼─────────┼─────────┼─────────┤
  │ 8     │ 1306 µs │ 819 µs  │ 1.6x    │
  ├───────┼─────────┼─────────┼─────────┤
  │ 16    │ 3116 µs │ 970 µs  │ 3.2x    │
  ├───────┼─────────┼─────────┼─────────┤
  │ 32    │ 9718 µs │ 1305 µs │ 7.4x    │
  └───────┴─────────┴─────────┴─────────┘

@AskAlexSharov AskAlexSharov requested a review from JkLondon March 6, 2026 10:03
@AskAlexSharov AskAlexSharov changed the title [wip] ii.filesMerge: did O(n^2) re-merge of sequences (chunks of same key) ii.filesMerge: did O(n^2) re-merge of sequences (chunks of same key) Mar 6, 2026
@AskAlexSharov AskAlexSharov merged commit 3d231db into release/3.4 Mar 6, 2026
22 checks passed
@AskAlexSharov AskAlexSharov deleted the alex/ef_reset4_s_34 branch March 6, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants