Skip to content

Stage generator: the paper-2611 profile narrows the 2026-11 subset (core lanes, fp32 dense, N=3) - #509

Merged
tae898 merged 1 commit into
mainfrom
subset-paper-profile
Oct 24, 2026
Merged

tae898 merged 1 commit into
mainfrom
subset-paper-profile

Conversation

@tae898

@tae898 tae898 commented Oct 24, 2026 •

Copy link
Copy Markdown

Narrows the 2026-11 DBBench subset to what the paper needs (the owner's decisions of 2026-10-25; the paper's experiments must finish by 2026-11-10). The narrowing is a named profile, make_2611_stages.py --profile paper-2611, which is now the default. Every dropped cell is listed by --check with its reason, so nothing is silently missing.

What the profile does

  1. Core lanes only: graph, cross-model, documents, dense. The second list (sparse, time series, lifecycle) is out for the comparators and for the ArcadeDB chain. --second-list is refused unless --profile full is given.
  2. Neo4j and Memgraph stay in the graph lane (and cross-model where they are: Memgraph, and Neo4j through the composed stack). Their dense cells are out.
  3. The dense lane is fp32 only for every engine: no _int8 and no _pq arm, comparators' and ArcadeDB's. ArcadeDB keeps arcadedb_dense_embedded and arcadedb_dense_server. The drop is derived from runner.LANES (QUANTIZED_DENSE is the arm-name rule), so a newly registered int8 arm is dropped without anyone typing its name.
  4. The target is N=3 (TARGET_REPS). --reps still defaults to 5, so the chain keeps Stage generator: exact pass order at N=5 (T1 1-3, T2 1-3, then T1/T2 alternating) and the ArcadeDB go marker #503's order with passes 4 and 5 after pass 3 (they run only if time is left), and the ArcadeDB marker gate is unchanged. --project now prices the profile's target repetitions (3) unless --project-reps 5 is given.

--profile full is the whole subset as it was before.

Dropped cells (lane cells per pass, both chains; each cell counted once, under the first reason)

reason cells arms
second list (sparse, time series, lifecycle) 148 lanes l3s, l4, lifecycle
Neo4j and Memgraph dense 8 neo4j_dense, neo4j_dense_int8, memgraph_dense, memgraph_dense_int8
quantized dense, other engines 12 arcadedb_dense_embedded_int8, arcadedb_dense_server_int8, qdrant_dense_int8, mongodb_dense_int8, arangodb_dense_int8, sqlite_vec_dense_int8 (and the _pq arms when BENCH_DENSE_PQ=1)

Why: the second list is out of scope for the paper. Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper.

Proof and tests

  • --check prints the dropped cells and runs profile_problems on both chains (comparators and ArcadeDB) whatever the arguments select: no lane of the second list, no quantized dense arm, no Neo4j or Memgraph dense cell, ArcadeDB's dense arms exactly its fp32 pair, no stale drop rule, no arm-name skip that would also drop an arm of another lane. The order proofs (plan_problems, schedule and prefixes) run as before.
  • --check --drafts DIR lints written scripts for the same (lane read back from --lanes, arms from BACKENDS).
  • stage_drafts_2611.sha256 is regenerated with test_stage_generator_2611.py --rewrite (229 scripts, from 319: the second list and the dropped arms are gone). The hash cases now use the default profile.
  • New tests in test_stage_generator_2611.py pin the dropped arms by name, the drafts of both chains, the refusal of --second-list, the BENCH_DENSE_PQ=1 case, the profile proof catching a planted dropped arm or lane, the draft lint, the order with passes 4 and 5 after 3, and the projection default. Tests that used --second-list through the CLI now pass --profile full.
  • Run locally pinned to cores 0-3: make_2611_stages.py --check, --check --drafts on a generated folder (0 problems), version_pin_check.py (0 disagreements), and the benchmarks pytest CI runs (1073 passed, 24 skipped). fairness_check.py needs results/runs.jsonl and does not apply to this change.
  • CAMPAIGN.md, COMPARATORS.md and FAIRNESS.md say what the subset stages now and why.

No benchmark ran, no docker, no mini.

🤖 Generated with Claude Code

…ore lanes, fp32 dense, N=3 target)

The paper's experiments must finish by 2026-11-10, so make_2611_stages.py takes a named profile, `--profile paper-2611` (the default), with the drops listed and proved by `--check` instead of silently missing:

- core lanes only (graph, cross-model, documents, dense): no sparse, time series or lifecycle cell, for the comparators and the ArcadeDB chain; `--second-list` needs `--profile full`
- Neo4j and Memgraph keep graph and cross-model, but no dense cell
- the dense lane is fp32 only for every engine: no `_int8` and no `_pq` arm, ArcadeDB's included (embedded and served fp32 stay)
- target N=3 (TARGET_REPS); the passes 4 and 5 stay after pass 3 in #503's order, with the ArcadeDB marker gate unchanged

`--check` prints each dropped reason with its lane-cell count and arms, runs profile_problems on both chains whatever the arguments select, and `--check --drafts` lints written scripts for the same. `--profile full` is the whole subset as before. stage_drafts_2611.sha256 is regenerated with `test_stage_generator_2611.py --rewrite`.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit e7e2d0b into main Oct 24, 2026
4 checks passed
@tae898
tae898 deleted the subset-paper-profile branch October 24, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant