Skip to content

fix SpillBuf bug when getting disk size of a closed file #1020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lihao712 merged 1 commit into from
Jun 9, 2025
+12 −2
Merged

fix SpillBuf bug when getting disk size of a closed file #1020

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions spark-extension/src/main/scala/org/apache/spark/sql/blaze/memory/SpillBuf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ class FileBasedSpillBuf(
with Logging {

private var readPosition: Long = 0
private var closedFileDiskUsage: Long = 0

override def write(buf: ByteBuffer): Unit = {
val startTimeNs = System.nanoTime()
Expand All @@ -126,12 +127,21 @@ class FileBasedSpillBuf(
}

override val memUsed: Long = 0
override def diskUsed: Long = fileChannel.size()
override def diskUsed: Long = {
if (fileChannel.isOpen) {
fileChannel.size()
} else {
closedFileDiskUsage
}
}
override def diskIOTime: Long = diskIOTimeNs
override def size: Long = numWrittenBytes

override def release(): Unit = {
fileChannel.close()
if (fileChannel.isOpen) {
closedFileDiskUsage = fileChannel.size()
fileChannel.close()
}
if (!file.delete()) {
logWarning(s"Was unable to delete spill file: ${file.getAbsolutePath}")
}
Expand Down
Loading