Hdfs ec merge#22
Merged
asfgit merged 217 commits intoapache:trunkfrom Sep 30, 2015
Merged
Conversation
…d directories ( Contributed by Zhe Zhang )
…buted by Zhe Zhang
…g (Kai Zheng via umamahesh)
…buted by Jing Zhao.
…ntributed by Jing Zhao.
…eNode. Contributed by Jing Zhao.
…o print striped blocks. Contributed by Takuya Fukudome.
…oup ( Contributed by Kai Zheng )
…om striped files. Contributed by Jing Zhao.
…-7903 and HDFS-7435. Contributed by Zhe Zhang.
…iped blocks. Contributed by Kai Sasaki.
…eplicatedBlocks and PendingReplicationBlocks. Contributed by Jing Zhao.
…s to DataNode. Contributed by Zhe Zhang.
…-7903, HDFS-7435 and HDFS-7930 (this commit is for HDFS-7930 only)
…ocks. Contributed by GAO Rui.
…mage. Contributed by Hui Zheng.
…-7903, HDFS-7435, HDFS-7930, HDFS-7960 (this commit is for HDFS-7960)
zhe-thoughts
force-pushed
the
HDFS-EC-Merge
branch
from
3cda072 to
53358fe
Compare
…ation factor on EC files. Contributed by Gao Rui.
…am. Contributed by Walter Su.
… and eliminate notion of EC zones.
…C policy. Contributed by J.Andreina. Change-Id: I9211d9728480225a407d82e6c0bea1a928adfa11
…ipedFile (Contributed by J.Andreina)
…ma Maheswara Rao G and Zhe Zhang.
…taNode. Contributed by Rakesh R. Change-Id: I9429706ae3c9b10a9274c07b98da6ed54cce192b
…w Wang. Change-Id: Ic3ec1f29fef0e27c46fff66fd28a51f8c4c61e71
…pecifying policy. Contributed by Zhe Zhang. Change-Id: I82919e55c4a0d8646d25a53aac757098a56b3029
…tanode.stripedread.timeout.millis. Contributed by Andrew Wang. Change-Id: Ice86c5d46d29e94087c0f186b941d5394e7ac0e6
…ning. Contributed by Rakesh R. Change-Id: Ic248999a7f8e5e740d49c9b10abcf16f66dd0f98
…ail to process remaining tasks due to one invalid ECTask. Contributed by Uma Maheswara Rao G.
…lErasureCodingPolicies. Contributed by Rakesh R. Change-Id: Id30a1ce9e2ce676d00a9220a2d3f14b8d9f00527
zhe-thoughts
force-pushed
the
HDFS-EC-Merge
branch
2 times, most recently
from
5ec5d29 to
b8393ed
Compare
Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java Change-Id: I8511c4d64b0959e79129febc179845a3892fedcc
…oo verbose and hurt performance. Contributed by Rui Li
zhe-thoughts
force-pushed
the
HDFS-EC-Merge
branch
from
b8393ed to
7bff8ca
Compare
… from ErasureCodingPolicy. Contributed by Vinayakumar B. Change-Id: Ic56da0b426f47c63dac440aef6f5fc8554f6cf13
…putStream. Contributed by Jing Zhao and Walter Su.
Conflicts: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java Change-Id: Ic7946c4ea35bed587fe879ce58b959b25ecc0823
mekasone
pushed a commit
to mekasone/hadoop
that referenced
this pull request
Feb 19, 2017
Add log message.
Amithsha
pushed a commit
to Amithsha/hadoop
that referenced
this pull request
Jan 28, 2021
Add greensight timeline collector dependency
saxenapranav
pushed a commit
to saxenapranav/hadoop
that referenced
this pull request
Apr 19, 2023
Mkdirs code for blob endpoint.
singer-bin
pushed a commit
to singer-bin/hadoop
that referenced
this pull request
Dec 19, 2024
Pull request for Sandy Ryza's fix for PARQUET-25. Author: Sandy Ryza <sandy.ryza@cloudera.com> Closes apache#22 from tomwhite/PARQUET-25-unbound-record-filter-configurable and squashes the following commits: a9d3fdc [Sandy Ryza] PARQUET-25. Pushdown predicates only work with hardcoded arguments.
LuciferYang
added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 16, 2026
…lockSize + getAclStatus. RPCs apache#21–23. Three PATH_READ reads using the shared xattrReadPilot helper (reusable for any read-only PATH_READ dispatch). apache#21 getStoragePolicy (PATH_READ) — returns BlockStoragePolicy from the target INode's storage policy ID. Pre-resolved-IIP overload added to FSDirAttrOp. apache#22 getPreferredBlockSize (PATH_READ) — returns the preferred block size from INodeFile.valueOf. Pre-resolved-IIP overload added to FSDirAttrOp (2 lines; no permission check needed beyond the traversal check in the pilot body). apache#23 getAclStatus (PATH_READ) — returns AclStatus built from INodeAttributes. Pre-resolved-IIP overload added to FSDirAclOp; skips the .snapshot-dir special case (pilot rejects .snapshot paths in envelope). Tests (3 new): - getStoragePolicyOnFile — set HOT then read back - getPreferredBlockSizeOnFile — create with 8192 block size, verify DFSClient.getBlockSize returns 8192 - getAclStatusOnFile — set ACL then read back via getAclStatus Verification: mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o \ -Dtest='TestFSNamesystemFGLIIP' => Tests run: 120, Failures: 0, Errors: 0, Skipped: 0 Post-gate RPC counter: 23 migrated. Spec: docs/fgl/HDFS-17385-wave4-pilot-design.md §6.3
LuciferYang
added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 16, 2026
RPCs apache#24–27. Continues the mechanical post-gate cadence. apache#24 checkAccess (PATH_READ) — validates FsAction access on a target. Reuses xattrReadPilot for traversal + permission check. FileNotFoundException for absent target, checkPathAccess for the requested mode. apache#25 getErasureCodingPolicy (PATH_READ) — returns the EC policy of a target file/dir. Pre-resolved-IIP overload added to FSDirErasureCodingOp; skips the .snapshot-dir and reserved-name special cases (pilot rejects both in envelope). apache#26 setErasureCodingPolicy (PATH_WRITE) — sets EC policy via xattr on a directory target. Pre-resolved-IIP overload uses setErasureCodingPolicyXAttr under fsd.writeLock. hasWriteLock(FS) assertion passes via HELD_IIP_WRITE. apache#27 unsetErasureCodingPolicy (PATH_WRITE) — removes EC policy xattr. Pre-resolved-IIP overload uses removeErasureCodingPolicyXAttr under fsd.writeLock. Also added FSDirAttrOp pre-resolved-IIP overloads: - getStoragePolicy(fsd, pc, bm, iip) — reused by RPC apache#21 - getPreferredBlockSize(fsd, iip) — reused by RPC apache#22 - getAclStatus(fsd, pc, iip) — reused by RPC apache#23 Verification: mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o \ -Dtest='TestFSNamesystemFGLIIP' => Tests run: 120, Failures: 0, Errors: 0, Skipped: 0 Post-gate RPC counter: 27 migrated. Spec: docs/fgl/HDFS-17385-wave4-pilot-design.md §6.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.