Repository navigation
Evaluate the deterministic engine against pandas, SQLite and DuckDB - #8
Merged
Merged
Conversation
Real questions from WikiTableQuestions with SQUALL's SQL as ground truth, four executors (Tablebeam, pandas, SQLite, DuckDB) on the same calculation specs, four real CSVs with hard number formats, and provenance checks on every answer. Inputs are pinned with SHA-256 digests and reruns are byte-identical.
bool is an Integral, so the demo's exact-cell formatter printed 1 and 0 for boolean group labels and cells. Found by the evaluation's provenance check on Montpellier's rent-control table (count by meuble).
A mean of exactly 32.36875 is stored just below the half and showed as 32.3687; summation noise does the same to 24.81875. Strip float noise at 15 significant digits, then round the six shown digits half up. Found by the evaluation's display check on Paris's rent-control table.
Tables copied from web pages often end with a Total or Career row. A sum, mean, median, maximum or row count over the whole column then counts everything twice, silently. On the evaluation's WikiTableQuestions specs this was the largest cause of wrong answers. Refuse with the row and the reason instead; minimums, grouped answers and filtered rows are unaffected.
Silent wrong answers on the WikiTableQuestions specs fall from 34 of 554 to 2; provenance and display checks now pass for all 542 answers.
fortunexbt
added a commit
that referenced
this pull request
Oct 23, 2026
* Add a reproducible evaluation of the deterministic engine Real questions from WikiTableQuestions with SQUALL's SQL as ground truth, four executors (Tablebeam, pandas, SQLite, DuckDB) on the same calculation specs, four real CSVs with hard number formats, and provenance checks on every answer. Inputs are pinned with SHA-256 digests and reruns are byte-identical. * Show TRUE/FALSE group labels as True and False in the demo bool is an Integral, so the demo's exact-cell formatter printed 1 and 0 for boolean group labels and cells. Found by the evaluation's provenance check on Montpellier's rent-control table (count by meuble). * Round displayed values from the decimal, not the binary float A mean of exactly 32.36875 is stored just below the half and showed as 32.3687; summation noise does the same to 24.81875. Strip float noise at 15 significant digits, then round the six shown digits half up. Found by the evaluation's display check on Paris's rent-control table. * Refuse single-number answers over a table's own total row Tables copied from web pages often end with a Total or Career row. A sum, mean, median, maximum or row count over the whole column then counts everything twice, silently. On the evaluation's WikiTableQuestions specs this was the largest cause of wrong answers. Refuse with the row and the reason instead; minimums, grouped answers and filtered rows are unaffected. * Record retrieval dates and rerun the evaluation after the fixes Silent wrong answers on the WikiTableQuestions specs fall from 34 of 554 to 2; provenance and display checks now pass for all 542 answers. * Document the evaluation's method, numbers and limits * Name a total row by its label when the index is not a position
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
eval/, a reproducible evaluation of the deterministic engine (CSV reader, parser, calculations). No model is loaded or called. It also fixes three engine bugs the evaluation found. Each fix has its own commit and regression test.What it measures:
Headline (95% Wilson intervals), in the README's wording:
"18,1"decimals.Fixes:
Verification:
pytest -qpasses (397 tests). Two runs ofpython eval/run_eval.pyproduce byte-identicaleval/results.jsonandeval/results.md. The README's new Evaluation section has the full tables and the limits.