Skip to content

test: prove first execution succeeds after reset_state#4530

Draft
laverem wants to merge 7 commits into
delta-io:mainfrom
laverem:df-54-add-reset-state-test
Draft

test: prove first execution succeeds after reset_state#4530
laverem wants to merge 7 commits into
delta-io:mainfrom
laverem:df-54-add-reset-state-test

Conversation

@laverem

@laverem laverem commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Follow-up to the DataFusion 54 upgrade (#4525). Adds unit tests for
ArrowStreamBatchGenerator in python/src/writer.rs covering DF54's
LazyMemoryExec::execute behavior, which calls reset_state on the first
execution:

  • first reset_state hands the reader to a fresh generator that yields the
    original batch (the regression that was breaking the first write)
  • a second reset_state is exhausted and errors loudly rather than silently
    yielding no rows

AI disclosure: tests were AI-generated and human verified locally with
cargo test --lib writer::tests.

Add unit tests for ArrowStreamBatchGenerator covering DF54's
LazyMemoryExec::execute behavior: the first reset_state hands the
reader to a fresh generator that yields the original batch, while a
second reset is exhausted and errors loudly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Esteban Laver <esteban.laver@enterprisedb.com>
@laverem laverem requested a review from ion-elgreco as a code owner June 11, 2026 20:44
@github-actions github-actions Bot added binding/python Issues for the Python package ai The issue or task is AI generated or contributed. labels Jun 11, 2026

@ethan-tyler ethan-tyler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow up!! I hate to be nit picky but the AI comments are overly verbose and the datafusion version is irrelevant.

@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.90%. Comparing base (6420bc1) to head (739267a).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4530      +/-   ##
==========================================
+ Coverage   79.84%   79.90%   +0.05%     
==========================================
  Files         172      172              
  Lines       62882    62915      +33     
  Branches    62882    62915      +33     
==========================================
+ Hits        50211    50274      +63     
+ Misses       9722     9691      -31     
- Partials     2949     2950       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ethan-tyler ethan-tyler marked this pull request as draft June 15, 2026 14:43
@laverem laverem force-pushed the df-54-add-reset-state-test branch from 5bc83d6 to 739267a Compare June 16, 2026 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai The issue or task is AI generated or contributed. binding/python Issues for the Python package

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants