update Shell.java to avoid OOM#30
Closed
huahuiyang wants to merge 2 commits intoapache:trunkfrom
Closed
Conversation
avoid NodeManager out of memory, in case running a command which has verbose error log continuously
avoid NodeManager out of memory, in case running a command which has verbose error log continuously
mekasone
pushed a commit
to mekasone/hadoop
that referenced
this pull request
Feb 19, 2017
Update DCOS.
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
- Defined APIs for specifying different types of windows - sessions, tumbling, global and keyed variants. - Defined APIs for specifying early and late triggers for a window. - Standardized all above Window types to be expressed as a combination of default, early and late triggers. - Defined classes for different types of trigger specifications. - Hide the WindowState class from programmers and move it from samza-api to samza-operator. We can choose to add it later if need be. - Removed some implementation classes in Window and Trigger. We can revisit them later when we implement Windows. - New API for specifying Time durations meaningfully in Samza. - Unit tests for most of the above changes. - Misc. Documentation, readability related changes to public APIs. Author: vjagadish1989 <jvenkatr@linkedin.com> Reviewers: Yi Pan <nickpan47@gmail.com>, Prateek Maheshwari <pmaheshw@linkedin.com> Closes apache#30 from vjagadish/samza-operator-v3
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
…y exceptions Author: vjagadish1989 <jvenkatr@linkedin.com> Reviewers: Jake Maes <jacob.maes@gmail.com> Closes apache#30 from vjagadish1989/samza-1077
saxenapranav
pushed a commit
to saxenapranav/hadoop
that referenced
this pull request
Apr 26, 2023
Mkdir create explicit
singer-bin
pushed a commit
to singer-bin/hadoop
that referenced
this pull request
Dec 19, 2024
This uses the existing dictionary support introduced for int96. Encoding and ParquetProperties have been updated to use the dictionary supporting classes, when requested for write or present during read. This also fixes a bug in the fixed dictionary values writer, where the length was hard-coded for int96, 12 bytes. Author: Ryan Blue <rblue@cloudera.com> Closes apache#30 from rdblue/PARQUET-63-add-fixed-dictionary-support and squashes the following commits: bc34a34 [Ryan Blue] PARQUET-63: Enable dictionary encoding for FIXED.
Contributor
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
LuciferYang
added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 16, 2026
Fourth pilot mode promoted. ANCESTOR_WRITE shares the same lock
plan as PATH_WRITE (read ancestors, write target) — the distinct
mode exists for semantic clarity: ANCESTOR_WRITE signals "subtree
operation" where the write lock on the target (subtree root)
protects all descendants.
Mode promotion:
- IIPAcquireMode.ANCESTOR_WRITE: DEFERRED → PILOT.
- INodeLockManager.computeMaxLockDepth: ANCESTOR_WRITE falls
through to the PATH_WRITE case (same depth computation).
- Only RENAME_WRITE remains deferred.
RPC apache#30 — delete -r (ANCESTOR_WRITE):
First subtree operation in the pilot. The outer delete method
now tries two pilot paths:
1. deletePilot (PARENT_WRITE) — single-file deletes (existing)
2. deleteRecursivePilot (ANCESTOR_WRITE) — recursive directory
deletes (new). Tried only when recursive=true AND the
single-file pilot missed (target is a directory).
deleteRecursivePilot:
- Acquires ANCESTOR_WRITE (read ancestors, write subtree root)
- Phase B: target exists, ancestorsAllowMutate, target directory
has no snapshot feature (isSnapshottable / isWithSnapshot)
- DFSUtil.checkProtectedDescendants for non-empty recursive dirs
- Permission: parentAccess=WRITE, subAccess=ALL (legacy parity)
- Delegates to FSDirDeleteOp.deleteInternal (same as single-file)
Concurrency semantic: a concurrent PATH_READ on a descendant of
the deleted subtree is serialized at the subtree root — the
reader needs a read lock on the root as an ancestor, which
conflicts with our write lock. This is the correct isolation
for subtree deletion.
Tests (4 new):
- deleteRecursiveNonEmptyDirectory — sub1/sub2/file tree
- deleteRecursiveDeeplyNested — a/b/c/d/e/file tree
- deleteRecursiveOnFileFallsToSingleFilePath — recursive=true
on a file still works (via the single-file pilot path)
- parallelRecursiveDeleteDisjointTrees — 8 concurrent recursive
deletes on disjoint subtrees
TestIIPAcquireMode + TestINodeLockManager updated:
- ANCESTOR_WRITE moved from DEFERRED_MODES to PILOT_MODES
- pilotExercisesTwoWriteModes → pilotExercisesThreeWriteModes
- deferredModeThrowsUnsupported updated to only RENAME_WRITE
Verification:
mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o \
-Dtest='TestFSNamesystemFGLIIP'
=> Tests run: 133, Failures: 0, Errors: 0, Skipped: 0
Post-gate RPC counter: 30 migrated. 4 of 5 pilot modes implemented.
Only RENAME_WRITE remains deferred.
Spec: docs/fgl/HDFS-17385-wave4-pilot-design.md §1.4, §2.4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
avoid out of memory(such as NodeManager OOM), in case running a command which has verbose error log continuously