Skip to content

Evaluate the deterministic engine against pandas, SQLite and DuckDB - #8

Merged
fortunexbt merged 7 commits into
mainfrom
eval/2026-10
Oct 23, 2026
Merged

fortunexbt merged 7 commits into
mainfrom
eval/2026-10

Conversation

@fortunexbt

Copy link
Copy Markdown
Owner

Adds eval/, a reproducible evaluation of the deterministic engine (CSV reader, parser, calculations). No model is loaded or called. It also fixes three engine bugs the evaluation found. Each fix has its own commit and regression test.

What it measures:

  • Real questions: 954 WikiTableQuestions questions (CC BY-SA 4.0) on Wikipedia tables. 554 of them are whole-table aggregates, identified by SQUALL's SQL (CC BY-SA 4.0); the other 400 are a fixed sample where a refusal is correct. The ground truth is SQUALL's SQL run in DuckDB, cross-checked with pandas and kept only when it matches the human answer.
  • The same calculations in Tablebeam, pandas, SQLite and DuckDB, each with its defaults, plus 162 calculations over four real CSVs with awkward number formats: Treasury Debt to the Penny (public domain), Wikidata X IDs (CC0), and Paris and Montpellier rent control (Licence Ouverte 2.0). Those are scored against exact fractions.
  • Provenance: the displayed rows, the downloaded rows, the scope and the written-out arithmetic must reproduce every answer.

Headline (95% Wilson intervals), in the README's wording:

  • Supported WikiTableQuestions questions (n=554): 79.1% correct (75.5–82.2), 20.6% refused, 0.4% wrong (0.1–1.3). Before the fixes: 79.2% correct, 14.6% refused, 6.1% wrong (4.4–8.5).
  • In their original wording the parser answers none of them, and it is never wrong (0.0%, 0.0–0.4, n=954).
  • Silent-wrong rate on the hard-format CSVs (n=162): Tablebeam 0.0% (0.0–2.3), pandas 19.8%, SQLite 38.9%, DuckDB 33.3%. Tablebeam refuses 37%, mostly Montpellier's "18,1" decimals.

Fixes:

  • Refuse single-number answers over a table's own Total or Career row instead of counting it twice. This caused 32 of the 34 wrong answers.
  • Round displayed values from the decimal, so 32.36875 shows as 32.3688.
  • The demo showed TRUE/FALSE group labels as 1 and 0.

Verification: pytest -q passes (397 tests). Two runs of python eval/run_eval.py produce byte-identical eval/results.json and eval/results.md. The README's new Evaluation section has the full tables and the limits.

Real questions from WikiTableQuestions with SQUALL's SQL as ground truth,
four executors (Tablebeam, pandas, SQLite, DuckDB) on the same calculation
specs, four real CSVs with hard number formats, and provenance checks on
every answer. Inputs are pinned with SHA-256 digests and reruns are
byte-identical.
bool is an Integral, so the demo's exact-cell formatter printed 1 and 0
for boolean group labels and cells. Found by the evaluation's provenance
check on Montpellier's rent-control table (count by meuble).
A mean of exactly 32.36875 is stored just below the half and showed as
32.3687; summation noise does the same to 24.81875. Strip float noise at
15 significant digits, then round the six shown digits half up. Found by
the evaluation's display check on Paris's rent-control table.
Tables copied from web pages often end with a Total or Career row. A sum,
mean, median, maximum or row count over the whole column then counts
everything twice, silently. On the evaluation's WikiTableQuestions specs
this was the largest cause of wrong answers. Refuse with the row and the
reason instead; minimums, grouped answers and filtered rows are unaffected.
Silent wrong answers on the WikiTableQuestions specs fall from 34 of 554
to 2; provenance and display checks now pass for all 542 answers.
@fortunexbt
fortunexbt merged commit b94ae02 into main Oct 23, 2026
6 checks passed
@fortunexbt
fortunexbt deleted the eval/2026-10 branch October 23, 2026 08:57
fortunexbt added a commit that referenced this pull request Oct 23, 2026
* Add a reproducible evaluation of the deterministic engine

Real questions from WikiTableQuestions with SQUALL's SQL as ground truth,
four executors (Tablebeam, pandas, SQLite, DuckDB) on the same calculation
specs, four real CSVs with hard number formats, and provenance checks on
every answer. Inputs are pinned with SHA-256 digests and reruns are
byte-identical.

* Show TRUE/FALSE group labels as True and False in the demo

bool is an Integral, so the demo's exact-cell formatter printed 1 and 0
for boolean group labels and cells. Found by the evaluation's provenance
check on Montpellier's rent-control table (count by meuble).

* Round displayed values from the decimal, not the binary float

A mean of exactly 32.36875 is stored just below the half and showed as
32.3687; summation noise does the same to 24.81875. Strip float noise at
15 significant digits, then round the six shown digits half up. Found by
the evaluation's display check on Paris's rent-control table.

* Refuse single-number answers over a table's own total row

Tables copied from web pages often end with a Total or Career row. A sum,
mean, median, maximum or row count over the whole column then counts
everything twice, silently. On the evaluation's WikiTableQuestions specs
this was the largest cause of wrong answers. Refuse with the row and the
reason instead; minimums, grouped answers and filtered rows are unaffected.

* Record retrieval dates and rerun the evaluation after the fixes

Silent wrong answers on the WikiTableQuestions specs fall from 34 of 554
to 2; provenance and display checks now pass for all 542 answers.

* Document the evaluation's method, numbers and limits

* Name a total row by its label when the index is not a position
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant