Skip to content

Conversation

@ashishnegi
Copy link
Contributor

@ashishnegi ashishnegi commented May 1, 2025

Rationale for this change

We saw crash with call stack:

error: runtime error: invalid memory address or nil pointer dereference
runtime.gopanic
	runtime/panic.go:792
runtime.panicmem
	runtime/panic.go:262
runtime.sigpanic
	runtime/signal_unix.go:925
github.com/apache/arrow-go/v18/arrow/memory.(*Buffer).Bytes
	github.com/apache/arrow-go/[email protected]/arrow/memory/buffer.go:106
github.com/apache/arrow-go/v18/parquet/file.(*page).Data
	github.com/apache/arrow-go/[email protected]/parquet/file/page_reader.go:90
github.com/apache/arrow-go/v18/parquet/file.(*columnWriter).TotalBytesWritten
	github.com/apache/arrow-go/[email protected]/parquet/file/column_writer.go:203
github.com/apache/arrow-go/v18/parquet/file.(*rowGroupWriter).Close
	github.com/apache/arrow-go/[email protected]/parquet/file/row_group_writer.go:237
github.com/apache/arrow-go/v18/parquet/file.(*Writer).FlushWithFooter
	github.com/apache/arrow-go/[email protected]/parquet/file/file_writer.go:229
github.com/apache/arrow-go/v18/parquet/file.(*Writer).Close
	github.com/apache/arrow-go/[email protected]/parquet/file/file_writer.go:215

On code inspection, i see that if WriteDataPage fails, w.pages will still be holding released Buffer/Pages.
On error handling, we close the parquet file, which tries to access w.pages through TotalBytesWritten() - causing crash.

What changes are included in this PR?

The PR includes a bug fix which removes the to-be released pages from w.pages.

Are these changes tested?

Added unit test.

Without the fix: test panics

=== RUN   TestWriteDataFailure
--- FAIL: TestWriteDataFailure (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x1026e4184]

goroutine 21 [running]:
testing.tRunner.func1.2({0x1031c7de0, 0x104016500})
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:1734 +0x1ac
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:1737 +0x334
panic({0x1031c7de0?, 0x104016500?})
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/panic.go:792 +0x124
github.com/apache/arrow-go/v18/arrow/memory.(*Buffer).Bytes(...)
	/Users/ashishnegi/work/repos/arrow-go/arrow/memory/buffer.go:112
github.com/apache/arrow-go/v18/parquet/file.(*page).Data(...)
	/Users/ashishnegi/work/repos/arrow-go/parquet/file/page_reader.go:90
github.com/apache/arrow-go/v18/parquet/file.(*columnWriter).TotalBytesWritten(...)
	/Users/ashishnegi/work/repos/arrow-go/parquet/file/column_writer.go:209
github.com/apache/arrow-go/v18/parquet/file_test.TestWriteDataFailure(0x14000103180)
	/Users/ashishnegi/work/repos/arrow-go/parquet/file/column_writer_test.go:121 +0x4b4
testing.tRunner(0x14000103180, 0x10339e3d8)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:1792 +0xe4
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:1851 +0x374
FAIL	github.com/apache/arrow-go/v18/parquet/file	0.764s

With the fix:

=== RUN   TestWriteDataFailure
--- PASS: TestWriteDataFailure (0.00s)

Are there any user-facing changes?

No

@ashishnegi ashishnegi requested a review from zeroshade as a code owner May 1, 2025 21:26
Comment on lines +425 to 433
for i, p := range w.pages {
defer p.Release()
if err = w.WriteDataPage(p); err != nil {
// To keep pages in consistent state,
// remove the pages that will be released using above defer call.
w.pages = w.pages[i+1:]
return err
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an easy way to add a test case for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a unit test and updated PR description.

@ashishnegi ashishnegi requested a review from zeroshade May 2, 2025 15:01
Copy link
Member

@zeroshade zeroshade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks! Will merge after CI finishes

@ashishnegi
Copy link
Contributor Author

@zeroshade Thanks for quick review!

I am wondering if this can be back ported to latest stable release v18.2.0 ?

@zeroshade
Copy link
Member

I actually think it's about time for an 18.3.0 release :) so I'll start preparing that

@zeroshade zeroshade merged commit b624664 into apache:main May 2, 2025
23 checks passed
@ashishnegi
Copy link
Contributor Author

ashishnegi commented May 6, 2025

Hi @zeroshade
Can i expect v18.3.0 this week ?
Thanks for your help!

@zeroshade
Copy link
Member

Hi! Sorry, I'll kick off the RC and the vote on the mailing list today. With luck I should be able to release v18.3.0 by Friday

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants