Skip to content

l3d: emit the close phase so dense rows record queries, idle and JVM working set after GC - #556

Merged
tae898 merged 1 commit into
mainfrom
fix/l3d-close-phase
Oct 30, 2026
Merged

tae898 merged 1 commit into
mainfrom
fix/l3d-close-phase

Conversation

@tae898

@tae898 tae898 commented Oct 30, 2026

Copy link
Copy Markdown

The 2026-10-30 audit of the 12 OK dense rows of the 2026-11 pass found mem_probe_phases=load,index_build,warmup on every one: no mem_queries_*, mem_idle_* or disk_queries_* columns, so the page column mem_queries_owned_mib is blank for dense, and no JVM working set after GC exists for dense rows.

Cause: memory_probe.phases_for_mark derives queries and idle (and the forced GC) only from the close-start mark. l1_tpc.py, l2_graph.py, e2_hybrid.py, l3_sparse.py and l4_tsbs.py wrap their end in _beat.phase("close"); l3d_dense.py had no close phase.

Change (benchmarks/experiments/l3d_dense.py): the close is now wrapped in _beat.phase("close"), after every timed operation (cold queries, beam sweep, mutation). _t starts inside the phase, so the probe's idle sleep and forced GC on close-start are not priced into close_s. Without BENCH_MEMPROBE_DIR the mark only prints a PHASE line. dense_multipass_driver.py does not call l3d_dense.main, so it gets no second close. This is an additive instrument amendment, approved on 2026-10-30, outside every timer; the other audit gaps stay deferred.

Test: test_dense_lane_emits_the_close_mark_after_every_timed_mark in benchmarks/experiments/test_memory_probe.py reads the lane's marker sequence from the l3d_dense.py source. On the old code it fails with phases ['load', 'index_build', 'warmup']; with the change test_memory_probe.py passes (18 passed, 1 skipped).

Smoke: one dense cell through the real runner (--lanes l3d --backends sqlite_vec_dense --scale tiny --reps 1 --mem 4g, BENCH_CPUSET=4-5, taskset -c 4,5 nice -n 19, own results file). The row has mem_probe_phases=load,index_build,warmup,queries,idle, mem_queries_owned_mib=39.1, mem_idle_owned_mib=39.1, disk_queries_mib=57.1, close_s=0.01. A non-JVM arm was used (dbbench:arcadedb is not on this laptop and no image was built), so the JVM working set after GC is not exercised by the smoke; that path is the same close-start hook the other lanes already use.

🤖 Generated with Claude Code

…JVM working set after GC

Audit of the 12 OK dense rows of the 2026-11 pass (2026-10-30): all had mem_probe_phases=load,index_build,warmup, with no mem_queries_*, mem_idle_* or disk_queries_* columns, so the page column mem_queries_owned_mib is blank for dense and no JVM working set after GC exists for dense rows.

Cause: memory_probe.phases_for_mark derives `queries` and `idle` (and the forced GC) only from the `close-start` mark. l1, l2, e2, l3s and l4 wrap their end in _beat.phase("close"); l3d_dense.py did not.

Fix: wrap the close in _beat.phase("close"), after every timed operation (cold queries, beam sweep, mutation). `_t` starts inside the phase, so the probe's idle sleep and forced GC on close-start are not priced into close_s. Without BENCH_MEMPROBE_DIR the mark only prints a PHASE line. dense_multipass_driver.py does not call l3d_dense.main, so it gets no second close.

Approved additive instrument amendment mid-pass (2026-10-30), outside every timer; the other audit gaps stay deferred.

Test: test_dense_lane_emits_the_close_mark_after_every_timed_mark reads the lane's own marker sequence from the l3d_dense.py source; it fails on the old code (phases load,index_build,warmup) and passes now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 4c4d1ac into main Oct 30, 2026
4 checks passed
@tae898
tae898 deleted the fix/l3d-close-phase branch October 30, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant