-
Notifications
You must be signed in to change notification settings - Fork 7
improve stoppable check APIs to include failure details #63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -396,7 +396,7 @@ public static boolean isInstanceStable(HelixDataAccessor dataAccessor, String in | |
| */ | ||
| public static boolean siblingNodesActiveReplicaCheck(HelixDataAccessor dataAccessor, | ||
| String instanceName) { | ||
| return siblingNodesActiveReplicaCheck(dataAccessor, instanceName, Collections.emptySet()); | ||
| return siblingNodesActiveReplicaCheckWithDetails(dataAccessor, instanceName, Collections.emptySet()).isPassed(); | ||
| } | ||
|
|
||
| /** | ||
|
|
@@ -416,6 +416,29 @@ public static boolean siblingNodesActiveReplicaCheck(HelixDataAccessor dataAcces | |
| */ | ||
| public static boolean siblingNodesActiveReplicaCheck(HelixDataAccessor dataAccessor, | ||
| String instanceName, Set<String> toBeStoppedInstances) { | ||
| return siblingNodesActiveReplicaCheckWithDetails(dataAccessor, instanceName, toBeStoppedInstances).isPassed(); | ||
| } | ||
|
|
||
| /** | ||
| * Check if sibling nodes of the instance meet min active replicas constraint with details | ||
| * Two instances are sibling of each other if they host the same partition. And sibling nodes | ||
| * that are in toBeStoppableInstances will be presumed to be stopped. | ||
| * WARNING: The check uses ExternalView to reduce network traffic but suffer from accuracy | ||
| * due to external view propagation latency | ||
| * | ||
| * This method returns detailed information about the first partition that fails the check, | ||
| * including resource name, partition name, current active replicas, and required minimum. | ||
| * | ||
| * TODO: Use in memory cache and query instance's currentStates | ||
| * | ||
| * @param dataAccessor A helper class to access the Helix data. | ||
| * @param instanceName An instance to be evaluated against this check. | ||
| * @param toBeStoppedInstances A set of instances presumed to be are already stopped. And it | ||
| * shouldn't contain the `instanceName` | ||
| * @return MinActiveReplicaCheckResult with pass/fail status and details of first failure | ||
| */ | ||
| public static MinActiveReplicaCheckResult siblingNodesActiveReplicaCheckWithDetails( | ||
| HelixDataAccessor dataAccessor, String instanceName, Set<String> toBeStoppedInstances) { | ||
| PropertyKey.Builder propertyKeyBuilder = dataAccessor.keyBuilder(); | ||
| List<String> resources = dataAccessor.getChildNames(propertyKeyBuilder.idealStates()); | ||
|
|
||
|
|
@@ -464,15 +487,15 @@ public static boolean siblingNodesActiveReplicaCheck(HelixDataAccessor dataAcces | |
| } | ||
| } | ||
| if (numHealthySiblings < minActiveReplicas) { | ||
| _logger.info( | ||
| "Partition {} doesn't have enough active replicas in sibling nodes. NumHealthySiblings: {}, minActiveReplicas: {}", | ||
| partition, numHealthySiblings, minActiveReplicas); | ||
| return false; | ||
| _logger.warn( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would suggest to move this to debug log instead. IDTS it qualifies as WARN.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Nope. we are still failing fast and provide only the details of first partition that's violating the min active replica check. And I chose warn as this may require attention and helpful while debugging.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wait, I just noticed, why are we not collecting all the failed partitions? |
||
| "Instance {} min active replica check failed: Resource {} partition {} has {}/{} active replicas", | ||
| instanceName, resourceName, partition, numHealthySiblings, minActiveReplicas); | ||
| return MinActiveReplicaCheckResult.failed(resourceName, partition, numHealthySiblings, minActiveReplicas); | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return true; | ||
| return MinActiveReplicaCheckResult.passed(); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.helix.util; | ||
|
|
||
| /** | ||
| * Result container for minimum active replica validation checks. | ||
| * Provides details about validation outcome and specific failure information. | ||
| */ | ||
| public class MinActiveReplicaCheckResult { | ||
| private final boolean passed; | ||
| private final String resourceName; | ||
| private final String partitionName; | ||
| private final int currentActiveReplicas; | ||
| private final int requiredMinActiveReplicas; | ||
|
|
||
| private MinActiveReplicaCheckResult(boolean passed, String resourceName, String partitionName, | ||
| int currentActiveReplicas, int requiredMinActiveReplicas) { | ||
| this.passed = passed; | ||
| this.resourceName = resourceName; | ||
| this.partitionName = partitionName; | ||
| this.currentActiveReplicas = currentActiveReplicas; | ||
| this.requiredMinActiveReplicas = requiredMinActiveReplicas; | ||
| } | ||
|
|
||
| public static MinActiveReplicaCheckResult passed() { | ||
| return new MinActiveReplicaCheckResult(true, null, null, -1, -1); | ||
| } | ||
|
|
||
| public static MinActiveReplicaCheckResult failed(String resourceName, String partitionName, | ||
| int currentActiveReplicas, int requiredMinActiveReplicas) { | ||
| return new MinActiveReplicaCheckResult(false, resourceName, partitionName, | ||
| currentActiveReplicas, requiredMinActiveReplicas); | ||
| } | ||
|
|
||
| public boolean isPassed() { | ||
| return passed; | ||
| } | ||
|
|
||
| public String getResourceName() { | ||
| return resourceName; | ||
| } | ||
|
|
||
| public String getPartitionName() { | ||
| return partitionName; | ||
| } | ||
|
|
||
| public int getCurrentActiveReplicas() { | ||
| return currentActiveReplicas; | ||
| } | ||
|
|
||
| public int getRequiredMinActiveReplicas() { | ||
| return requiredMinActiveReplicas; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| if (passed) { | ||
| return "MIN_ACTIVE_REPLICA_CHECK_FAILED: passed"; | ||
| } else { | ||
| return String.format("MIN_ACTIVE_REPLICA_CHECK_FAILED: Resource %s partition %s has %d/%d active replicas", | ||
| resourceName, partitionName, currentActiveReplicas, requiredMinActiveReplicas); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current implementation doesnt have 1:1 backward compatibility, even if includeDetails is not passed, still we will enter this method and will check all the paritions nonetheless.
I will propose to add a flag in this method like "failOnFirst" which will be true if "includeDetails" wasnt passed or is false, and bail out on first failure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implementation is fully backward compatible with existing. We still fail fast and provide details only for the first partition violating the min active replica check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just noticed. Why are we not collecting all the failed partitions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea was to provide an indication to the failed partitions since anyways the result will be truncated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The result will be truncated in Nimbus UI/Customized status. But we can add logging on ACM side for earlier debugging if we have this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with many partitions part. Here's what I think can be smartly done.
if error list < 10, list all, and for all remaining, just give a count summary.
Maintaining backward compatibility is simple like I mentioned in the original comment with the use of "failOnFirst"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imho, as already mentioned in response to other comment above, we need to iterate on this feature.
Adding more partition details in string format, I'm not very much convinced.
want to see customers feedback on the current version and iterate based on that.
We need a better request/response modelling to handle all other checks failures too in addition to MIN_ACTIVE_REPLICA check.
Have some thoughts on it. But want to iterate based on the feedback.
Adding more flags in the current API makes them irrelevant for other flavours of the APIs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few observations:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thought about it during the implementation and discussed with @LZD-PratyushBhatt too at that time.
Count info may not provide any info on which users can take an action without additional debugging/api calls which they are doing it today.
And another reason for not including it is, don't want to pack too many details in a string message. Right approach is to add all these flags/details should be part of the request/response model.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By count, I was mentioning the count of INITIAL state (like OFFLINE) and ERROR state.
I was against just printing the first partition info, because if you have 10 partition in ERROR state, you fix one, that doesn't affect 9 others. Rebalancing doesn't have any affect on ERROR partitions.
A count summary atleast let's the USER know that they need to look at Current state of the participant to get list of all the initial/error state partitions.
Anyways, my comment/suggestion is non-binding and I won't block the PR because of this.
Overall LGTM