Skip to content

Make io compression zstd level configurable#1111

Merged
richox merged 3 commits intoapache:masterfrom
turboFei:zstd_level
Aug 7, 2025
Merged

Make io compression zstd level configurable#1111
richox merged 3 commits intoapache:masterfrom
turboFei:zstd_level

Conversation

@turboFei
Copy link
Member

@turboFei turboFei commented Aug 6, 2025

Which issue does this PR close?

Closes #.

Rationale for this change

Make zstd level configurable.
https://github.com/apache/spark/blob/bca5f3a0f09e9020f4bda9b07ce70d7c685f9549/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2130-L2136

  private[spark] val IO_COMPRESSION_ZSTD_LEVEL =
    ConfigBuilder("spark.io.compression.zstd.level")
      .doc("Compression level for Zstd compression codec. Increasing the compression " +
        "level will result in better compression at the expense of more CPU and memory")
      .version("2.3.0")
      .intConf
      .createWithDefault(1)

What changes are included in this PR?

Are there any user-facing changes?

@turboFei turboFei requested a review from merrily01 August 6, 2025 09:56
Copy link
Member

@merrily01 merrily01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@richox richox merged commit f44b2cf into apache:master Aug 7, 2025
73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants