Repository navigation
fix(#941): support PagerDuty incident lookup for HCP clusters - #942
fullsend-ai-coder[bot] wants to merge 5 commits into
Conversation
HCP clusters use region-based PagerDuty services rather than per-cluster services keyed by DNS base domain. This caused osdctl cluster context and org context to return zero PD incidents for HCP clusters. For HCP clusters, use the cluster's region ID to query PD services and filter incidents by matching cluster ID in the first trigger log entry's EventDetails. Classic clusters continue to use the existing DNS-based lookup unchanged. Changes: - pkg/provider/pagerduty: add WithClusterID builder method and incidentMatchesCluster filter that checks EventDetails for cluster_id when clusterID is set on the client - cmd/cluster/context: detect HCP clusters in setup() to use region-based PD service query and cluster ID filtering - cmd/org/context: extend NewPDClient to accept clusterID, detect HCP clusters in FetchContext to use region-based lookup with cluster ID filtering Note: golangci-lint was not available in the sandbox. go vet and gofmt passed on all changed packages. Closes #941
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: fullsend-ai-coder[bot] The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @fullsend-ai-coder[bot]. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
There was a problem hiding this comment.
Both cmd/cluster/context.go (line 392) and cmd/org/context.go (line 218) pass the internal ID, PagerDuty alerts contain the external UUID
This means incidentMatchesCluster always returns false, and all incidents are filtered out. Confirmed in a live test: all 10 services show "None".
Fix: use ExternalID() instead of ID() in both places:
cmd/cluster/context.go:392:WithClusterID(o.externalClusterID)instead ofWithClusterID(o.clusterID)cmd/org/context.go:218:cluster.ExternalID()instead ofcluster.ID()
baseDomain works for classic clusters because PD services are named after the cluster's DNS domain
For HCP, the PR overwrites baseDomain with the AWS region string (us-east-1). This technically works because GetPDServiceIDs just passes whatever is in baseDomain to PD's Query field but it's using a DNS-domain-shaped variable
to carry a region name through a fuzzy text search, which is why it accidentally matches 10 unrelated services.
A cleaner design would separate the concern: either introduce a distinct field/parameter for the PD service query string (so it's clear what's being searched and why), or construct the actual expected service name prefix e.g. rhobs-hcp-prod-us-east-1
Querying PD with just the region string (e.g. us-east-1) returns ~10 services via substring match across all environments and service types. Only the 2 prod RHOBS HCP services are relevant for a production cluster.
- rhobs-hcp-prod-critical-us-east-1 (relevant)
- rhobs-hcp-prod-warning-us-east-1 (relevant)
- rhobs-hcp-stage-us-east-1 (wrong env)
- rhobs-infra-{critical,warning,stage}-us-east-1 (infra, not cluster alerts)
- osd-hypershift-us-east-1 (legacy)
- ...
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 10:45 AM UTC · Completed 11:01 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.83 |
Both cmd/cluster/context.go and cmd/org/context.go were passing the internal cluster ID to WithClusterID(), but PagerDuty alerts contain the external UUID. This caused incidentMatchesCluster to always return false, filtering out all incidents for HCP clusters. Use ExternalID() instead of ID() in both locations so the cluster ID matches what PagerDuty stores in alert EventDetails. Addresses #942
🔧 Fix agent — iteration 1 (human-triggered)Fixed the core bug: both cluster and org context commands were passing the internal cluster ID to PagerDuty incident filtering, but PD alerts contain the external UUID. Changed to use ExternalID() in both locations. The design concern about baseDomain carrying a region string is valid but not a correctness issue once the ID fix is applied. Fixed (2):
Disagreed (1):
Tests: passed CI inspections (6):
Decision points
Next steps:
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
🤖 Finished Review · ✅ Success · Started 11:03 AM UTC · Completed 11:21 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.23 |
|
Risk Assessment: moderate (2/5) DetailsRoutine bot-authored bug fix with good test coverage, no security or CI concerns; code age of several affected files (1-3 years) and historical fix commits on cluster/context.go are the primary risk drivers, yielding a moderate composite score. Previous runRisk Assessment: moderate (2/5) DetailsA bot-authored, well-tested bug fix for a specific HCP PagerDuty lookup gap; moderate line count and blast radius are offset by clean history, zero protected/security-sensitive paths, and a tightly scoped linked issue, yielding a composite of ~1.7 rounded to 2. Previous run (2)Risk Assessment: moderate (2/5) DetailsBot-authored bug fix adding HCP-specific PagerDuty routing with solid test coverage (40%) across a medium-sized change; stable files with no recent churn and well-scoped issue alignment keep risk at moderate. Previous run (3)Risk Assessment: moderate (2/5) DetailsA small bot-authored bug fix with good test coverage and no security or CI impact; risk is held at moderate primarily by the staleness of four of five touched files (most last modified 6-36 months ago) and the lack of a feature flag guarding the changed lookup path. |
ReviewFindingsMedium
Low
Next steps:
Previous runReviewFindingsMedium
Low
Next steps:
Previous run (2)ReviewFindingsMedium
Low
Next steps:
Previous run (3)ReviewFindingsMedium
Low
Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 11:23 AM UTC · Completed 11:38 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.76 |
…er ID Align the WithClusterID guard in generateContextData() with the region check in setup() so that HCP clusters without a region do not silently drop all PD alerts. Add cluster ID filtering to GetHistoricalAlertsForCluster so historical alert summaries for HCP clusters only include incidents from the target cluster, not all clusters in the region. Addresses #942
🔧 Fix agent — iteration 2 (bot-triggered)Fixed both medium-severity logic errors: aligned the WithClusterID guard condition in cluster context to match the region check (preventing silent alert loss for HCP clusters without a region), and added cluster ID filtering to GetHistoricalAlertsForCluster (preventing inflated historical alert counts from unrelated clusters in the same region). Added tests for the new historical filtering. Disagreed on four low-severity refactoring/naming suggestions as out of scope for this bug-fix PR. Fixed (2):
Disagreed (4):
Tests: passed CI inspections (6):
Decision points
Next steps:
|
|
🤖 Review · ❌ Terminated · Started 11:39 AM UTC · Ended 11:55 AM UTC Commit: |
|
🤖 Finished Review · ✅ Success · Started 11:39 AM UTC · Completed 11:55 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.85 |
|
🤖 Finished Fix · ✅ Success · Started 11:57 AM UTC · Completed 12:13 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.29 |
- Extract duplicated HCP region guard (Hypershift().Enabled() && Region() != nil && Region().ID() != "") into utils.HCPRegionID() helper, used by all three call sites in cluster/context.go and org/context.go to eliminate divergence risk - Add gomock matchers (hasIncludes/hasNoIncludes) to verify that HCP tests assert opts.Includes contains "first_trigger_log_entries" and classic cluster tests assert Includes is empty Addresses #942
🔧 Fix agent — iteration 3 (bot-triggered)Addressed both Medium findings: extracted the duplicated HCP guard into a shared utils.HCPRegionID helper (eliminating divergence risk across three call sites), and added gomock matchers verifying that opts.Includes is set correctly for HCP vs classic clusters. Disagreed with two Low findings (FetchContext integration test and per-incident logging) with reasoning. Fixed (2):
Disagreed (2):
Tests: passed CI inspections (5):
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 12:15 PM UTC · Completed 12:33 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.25 |
|
🤖 Finished Fix · ✅ Success · Started 12:35 PM UTC · Completed 12:51 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.14 |
|
/fs-fix-stop |
|
Fix agent disabled for this PR. Remove the |
…w feedback on PR #942 - Rename baseDomain/WithBaseDomain to serviceQuery/WithServiceQuery in PD client to accurately describe the dual-purpose parameter (DNS base domain for classic, region ID for HCP) - Rename contextOptions.baseDomain to pdServiceQuery in cluster context - Add warning log in incidentMatchesCluster when EventDetails is nil to aid debugging when HCP incidents are filtered out - Rewrite HCPRegionID doc comment to be cluster-generic, not PD-specific - Move WithClusterID test into Client Creation describe block where other builder method tests live - Add alternate key name tests (clusterID, cluster-id) for GetHistoricalAlertsForCluster HCP filtering Addresses #942
🔧 Fix agent — iteration 4 (bot-triggered)Addressed 6 of 8 review findings with code changes: renamed baseDomain/WithBaseDomain to pdServiceQuery/WithServiceQuery for naming coherence, added warning log for nil EventDetails, rewrote HCPRegionID doc comment, moved WithClusterID test to correct describe block, and added alternate key tests for historical alerts. Disagreed on 2 findings: narrowing key variants requires infrastructure knowledge outside PR scope, and FetchContext integration testing requires substantial OCM mock infrastructure. Fixed (6):
Disagreed (2):
Tests: passed CI inspections (8):
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 12:53 PM UTC · Completed 1:11 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.12 |
There was a problem hiding this comment.
Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:
pkg/provider/pagerduty/pagerduty.go(file-level): Line 178 · [medium] logic-error
Pre-existing bug now more impactful for HCP: the incidents slice is declared outside the outer service-ID loop and is never reset between iterations. Incidents from earlier services accumulate and inflate historical alert counts for subsequent service IDs.
Suggested fix: Reset the incidents slice at the start of each outer loop iteration.
| @@ -106,21 +115,33 @@ func (c *client) GetFiringAlertsForCluster(pdServiceIDs []string) (map[string][] | |||
| var incidentListOffset uint = 0 | |||
There was a problem hiding this comment.
[medium] logic-error
Pre-existing bug now more impactful for HCP: incidentListOffset is declared outside the outer for loop over service IDs and is never reset to 0 between iterations. If the first service requires pagination, subsequent services begin querying at a non-zero offset, silently omitting their first pages of incidents.
Suggested fix: Reset incidentListOffset to 0 at the top of each outer loop iteration.
| GetServiceLogs func(string, time.Time, bool, bool) ([]*v1.LogEntry, error) | ||
| GetJiraIssues func(clusterID, externalID, filter string) ([]jira.Issue, error) | ||
| NewPDClient func(baseDomain string) (PDClient, error) | ||
| NewPDClient func(baseDomain, clusterID string) (PDClient, error) |
There was a problem hiding this comment.
[low] naming-conventions
The first parameter of NewPDClient is named baseDomain but its semantics changed: for HCP clusters the caller passes a region ID. In cmd/cluster/context.go the equivalent was renamed to pdServiceQuery, creating an inconsistency.
Suggested fix: Rename the parameter to serviceQuery in both the type definition and the function literal.
| } | ||
|
|
||
| pdClient := pagerduty.NewClient().WithBaseDomain(baseDomain) | ||
| pdClient := pagerduty.NewClient().WithServiceQuery(baseDomain) |
There was a problem hiding this comment.
[low] edge-case
validatePDIncidents was updated with the API rename but does not add HCP support. For HCP clusters, PD incident validation is silently skipped.
| } | ||
| } | ||
|
|
||
| func TestNewPDClient_PassesClusterID(t *testing.T) { |
There was a problem hiding this comment.
[low] test-adequacy
TestNewPDClient_PassesClusterID verifies parameter forwarding but does not exercise the HCP routing logic in FetchContext that determines which values to pass based on HCPRegionID.
Suggested fix: Add integration-level tests for the FetchContext HCP routing branch.
| return c | ||
| } | ||
|
|
||
| func (c *client) WithClusterID(clusterID string) *client { |
There was a problem hiding this comment.
[low] naming-conventions
WithClusterID uses all-caps ID while the existing WithTeamIdList uses mixed-case Id, creating a naming inconsistency in the builder API.
Suggested fix: Use consistent casing: either WithClusterId to match existing WithTeamIdList, or rename WithTeamIdList to WithTeamIDList for Go-idiomatic consistency.
| // For HCP clusters, PD services are organized by region | ||
| // rather than per-cluster DNS domain. Use the region as | ||
| // the query and filter incidents by cluster ID. | ||
| baseDomain := cluster.DNS().BaseDomain() |
There was a problem hiding this comment.
[low] naming-coherence
The local variable baseDomain is overwritten with a region ID for HCP clusters, reusing the same variable for two semantically distinct values.
Suggested fix: Introduce a serviceQuery variable initialized to baseDomain, then assign regionID to serviceQuery for HCP clusters.
|
@fullsend-ai-coder[bot]: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Summary
osdctl cluster contextandosdctl org contextcommandsEventDetailsApproach
PagerDuty provider (
pkg/provider/pagerduty):WithClusterID()builder method on the PD clientclusterIDis set,GetFiringAlertsForClusterincludesfirst_trigger_log_entriesin the PD API request and filters incidents by checkingEventDetailsfor a matching cluster ID (supportscluster_id,clusterID, andcluster-idkey names)clusterIDis empty (classic clusters), behavior is unchangedCluster context (
cmd/cluster/context.go):setup(), detects HCP clusters viacluster.Hypershift().Enabled()and uses the region ID instead of DNS base domain for the PD service querygenerateContextData(), passes the cluster ID to the PD client for HCP clustersOrg context (
cmd/org/context.go):NewPDClientfunction signature to accept bothbaseDomainandclusterIDFetchContext, detects HCP clusters and passes region + cluster ID for filteringTesting
incidentMatchesClustercovering all key name variants, non-matching, nil EventDetails, and missing key scenariosGetFiringAlertsForClusterwith cluster ID filtering: matching, non-matching, alternate keys, and classic (empty clusterID) behaviorNewPDClientsignature in org contextCloses #941
Post-script verification
agent/941-hcp-pagerduty-incidents)13f62d5fbd5f130e861aa956a4709499d18144e9..HEAD)