Repository navigation
l3d: emit the close phase so dense rows record queries, idle and JVM working set after GC - #556
Merged
Merged
Conversation
…JVM working set after GC
Audit of the 12 OK dense rows of the 2026-11 pass (2026-10-30): all had mem_probe_phases=load,index_build,warmup, with no mem_queries_*, mem_idle_* or disk_queries_* columns, so the page column mem_queries_owned_mib is blank for dense and no JVM working set after GC exists for dense rows.
Cause: memory_probe.phases_for_mark derives `queries` and `idle` (and the forced GC) only from the `close-start` mark. l1, l2, e2, l3s and l4 wrap their end in _beat.phase("close"); l3d_dense.py did not.
Fix: wrap the close in _beat.phase("close"), after every timed operation (cold queries, beam sweep, mutation). `_t` starts inside the phase, so the probe's idle sleep and forced GC on close-start are not priced into close_s. Without BENCH_MEMPROBE_DIR the mark only prints a PHASE line. dense_multipass_driver.py does not call l3d_dense.main, so it gets no second close.
Approved additive instrument amendment mid-pass (2026-10-30), outside every timer; the other audit gaps stay deferred.
Test: test_dense_lane_emits_the_close_mark_after_every_timed_mark reads the lane's own marker sequence from the l3d_dense.py source; it fails on the old code (phases load,index_build,warmup) and passes now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 2026-10-30 audit of the 12 OK dense rows of the 2026-11 pass found
mem_probe_phases=load,index_build,warmupon every one: nomem_queries_*,mem_idle_*ordisk_queries_*columns, so the page columnmem_queries_owned_mibis blank for dense, and no JVM working set after GC exists for dense rows.Cause:
memory_probe.phases_for_markderivesqueriesandidle(and the forced GC) only from theclose-startmark.l1_tpc.py,l2_graph.py,e2_hybrid.py,l3_sparse.pyandl4_tsbs.pywrap their end in_beat.phase("close");l3d_dense.pyhad no close phase.Change (
benchmarks/experiments/l3d_dense.py): the close is now wrapped in_beat.phase("close"), after every timed operation (cold queries, beam sweep, mutation)._tstarts inside the phase, so the probe's idle sleep and forced GC onclose-startare not priced intoclose_s. WithoutBENCH_MEMPROBE_DIRthe mark only prints aPHASEline.dense_multipass_driver.pydoes not calll3d_dense.main, so it gets no second close. This is an additive instrument amendment, approved on 2026-10-30, outside every timer; the other audit gaps stay deferred.Test:
test_dense_lane_emits_the_close_mark_after_every_timed_markinbenchmarks/experiments/test_memory_probe.pyreads the lane's marker sequence from thel3d_dense.pysource. On the old code it fails with phases['load', 'index_build', 'warmup']; with the changetest_memory_probe.pypasses (18 passed, 1 skipped).Smoke: one dense cell through the real runner (
--lanes l3d --backends sqlite_vec_dense --scale tiny --reps 1 --mem 4g,BENCH_CPUSET=4-5,taskset -c 4,5 nice -n 19, own results file). The row hasmem_probe_phases=load,index_build,warmup,queries,idle,mem_queries_owned_mib=39.1,mem_idle_owned_mib=39.1,disk_queries_mib=57.1,close_s=0.01. A non-JVM arm was used (dbbench:arcadedbis not on this laptop and no image was built), so the JVM working set after GC is not exercised by the smoke; that path is the sameclose-starthook the other lanes already use.🤖 Generated with Claude Code