[feature](storage) Add OzoneProperties to support Apache Ozone#60809
Merged
morningman merged 4 commits intoapache:masterfrom Feb 27, 2026
Merged
[feature](storage) Add OzoneProperties to support Apache Ozone#60809morningman merged 4 commits intoapache:masterfrom
morningman merged 4 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
There was a problem hiding this comment.
Pull request overview
This pull request adds support for fs.s3a.* property aliases to enable Hadoop-style configuration for S3 and MinIO storage backends. This allows users familiar with Hadoop S3A filesystem properties to use those standard property names when configuring Doris data sources, particularly useful for Iceberg and Ozone S3 gateway integrations.
Changes:
- Added
fs.s3a.*aliases for S3 and MinIO storage property configurations (endpoint, region, credentials, connection settings, path-style access) - Comprehensive test coverage for the new alias property mapping in both S3 and MinIO property classes
- Updated property identifiers to recognize
fs.s3a.*properties for storage type detection
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| S3Properties.java | Added fs.s3a.* aliases to ConnectorProperty annotations for endpoint, region, access key, secret key, session token, connection settings, and path-style access |
| MinioProperties.java | Added fs.s3a.* aliases to ConnectorProperty annotations and updated IDENTIFIERS set to include the new aliases |
| S3PropertiesTest.java | Added comprehensive test case validating fs.s3a.* properties map correctly to S3Properties fields and backend configuration |
| MinioPropertiesTest.java | Added two test cases: one for basic fs.s3a.* property mapping and another for fs.defaultFS integration |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
TPC-H: Total hot run time: 28737 ms |
TPC-DS: Total hot run time: 183375 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
xylaaaaa
changed the title
xylaaaaa
changed the title
Contributor
|
run buildall |
morningman
previously approved these changes
Feb 26, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 28875 ms |
TPC-DS: Total hot run time: 183658 ms |
github-actions
bot
removed
the
approved
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 28916 ms |
TPC-DS: Total hot run time: 185302 ms |
morningman
approved these changes
Feb 27, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
CalvinKirs
approved these changes
Feb 27, 2026
github-actions bot
pushed a commit
that referenced
this pull request
Feb 27, 2026
## Proposed changes Add dedicated `OzoneProperties` to support Apache Ozone S3 Gateway as an explicit storage backend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Add dedicated
OzonePropertiesto support Apache Ozone S3 Gateway as an explicit storage backend.Changes
OzonePropertiesclass — extendsAbstractS3CompatibleProperties, supportsozone.*ands3.*aliases (endpoint, region, access key, secret key, session token, path style, connection settings).fs.ozone.support=trueis provided.ozone/s3g).fs.ozone.support=true,MinioProperties.guessIsMe(...)no longer preempts Ozone.OzonePropertiesTestnow coverss3.*binding, explicit support requirement, and rejection path forfs.s3a.*-only Ozone config.Further information
Apache Ozone exposes an S3-compatible gateway (S3G). For Doris Ozone configs in this PR:
fs.ozone.support=true.s3.*keys for endpoint/AK/SK/region/path-style.warehouse/fs.defaultFSmay still bes3a://...URIs as required by Iceberg/Hadoop path semantics.Checklist
Local verification (2026-02-26)
Environment:
127.0.0.1:9030http://127.0.0.1:9878thrift://127.0.0.1:9083Results:
(1, Alice, 2026-01-01)(2, Bob, 2026-01-02)Iceberg Catalog Examples (Ozone, updated)
1) Hadoop Catalog (no HMS dependency)
2) HMS Catalog (production path with Hive Metastore)
Additional verification (2026-02-27): ozone.* aliases
This PR now also validates explicit
ozone.*property aliases (in addition tos3.*) for Iceberg catalogs.Unit tests
OzonePropertiesTestaddedtestCreateAllWithDefaultFsAndOzoneProperties()OzonePropertiesTestMinioPropertiesTestS3PropertiesTestRuntime verification (local)
Environment:
127.0.0.1:9030http://127.0.0.1:9878thrift://127.0.0.1:9083Results:
ozone.*: create catalog/database/table + insert/select succeeded.ozone.*: create catalog/database/table + insert/select succeeded.s3.*control path still succeeds (no regression).Hadoop Catalog with ozone.* (validated)
HMS Catalog with ozone.* (validated)
apache/doris-website#3421