Skip to content

fix(#941): support PagerDuty incident lookup for HCP clusters - #942

Open
fullsend-ai-coder[bot] wants to merge 5 commits into
masterfrom
agent/941-hcp-pagerduty-incidents
Open

fullsend-ai-coder[bot] wants to merge 5 commits into
masterfrom
agent/941-hcp-pagerduty-incidents

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

  • Fixes PagerDuty incident lookup for HCP (Hosted Control Plane) clusters in both osdctl cluster context and osdctl org context commands
  • HCP clusters use region-based PD services instead of per-cluster DNS-based services, so the existing DNS base domain query returned zero results
  • For HCP clusters, queries PD services by the cluster's AWS region and filters incidents by matching cluster ID in the alert's EventDetails
  • Classic clusters continue to use the existing DNS-based lookup unchanged

Approach

PagerDuty provider (pkg/provider/pagerduty):

  • Added WithClusterID() builder method on the PD client
  • When clusterID is set, GetFiringAlertsForCluster includes first_trigger_log_entries in the PD API request and filters incidents by checking EventDetails for a matching cluster ID (supports cluster_id, clusterID, and cluster-id key names)
  • When clusterID is empty (classic clusters), behavior is unchanged

Cluster context (cmd/cluster/context.go):

  • In setup(), detects HCP clusters via cluster.Hypershift().Enabled() and uses the region ID instead of DNS base domain for the PD service query
  • In generateContextData(), passes the cluster ID to the PD client for HCP clusters

Org context (cmd/org/context.go):

  • Extended NewPDClient function signature to accept both baseDomain and clusterID
  • In FetchContext, detects HCP clusters and passes region + cluster ID for filtering

Testing

  • Added unit tests for incidentMatchesCluster covering all key name variants, non-matching, nil EventDetails, and missing key scenarios
  • Added integration-level tests for GetFiringAlertsForCluster with cluster ID filtering: matching, non-matching, alternate keys, and classic (empty clusterID) behavior
  • Added test for updated NewPDClient signature in org context
  • All existing tests continue to pass (25 pagerduty tests, 7 org context tests, 16 cluster context tests)

Closes #941

Post-script verification

  • Branch is not main/master (agent/941-hcp-pagerduty-incidents)
  • Secret scan passed (gitleaks — 13f62d5fbd5f130e861aa956a4709499d18144e9..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

HCP clusters use region-based PagerDuty services rather than
per-cluster services keyed by DNS base domain. This caused
osdctl cluster context and org context to return zero PD
incidents for HCP clusters.

For HCP clusters, use the cluster's region ID to query PD
services and filter incidents by matching cluster ID in the
first trigger log entry's EventDetails. Classic clusters
continue to use the existing DNS-based lookup unchanged.

Changes:
- pkg/provider/pagerduty: add WithClusterID builder method and
  incidentMatchesCluster filter that checks EventDetails for
  cluster_id when clusterID is set on the client
- cmd/cluster/context: detect HCP clusters in setup() to use
  region-based PD service query and cluster ID filtering
- cmd/org/context: extend NewPDClient to accept clusterID,
  detect HCP clusters in FetchContext to use region-based
  lookup with cluster ID filtering

Note: golangci-lint was not available in the sandbox. go vet
and gofmt passed on all changed packages.

Closes #941
@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fullsend-ai-coder[bot]
Once this PR has been reviewed and has the lgtm label, please assign tafhim for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 4, 2026
@openshift-ci

openshift-ci Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Hi @fullsend-ai-coder[bot]. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@RaphaelBut

Copy link
Copy Markdown
Contributor

/ok-to-test

@openshift-ci openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2026

@RaphaelBut RaphaelBut left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both cmd/cluster/context.go (line 392) and cmd/org/context.go (line 218) pass the internal ID, PagerDuty alerts contain the external UUID

This means incidentMatchesCluster always returns false, and all incidents are filtered out. Confirmed in a live test: all 10 services show "None".

Fix: use ExternalID() instead of ID() in both places:

  • cmd/cluster/context.go:392: WithClusterID(o.externalClusterID) instead of WithClusterID(o.clusterID)
  • cmd/org/context.go:218: cluster.ExternalID() instead of cluster.ID()

baseDomain works for classic clusters because PD services are named after the cluster's DNS domain
For HCP, the PR overwrites baseDomain with the AWS region string (us-east-1). This technically works because GetPDServiceIDs just passes whatever is in baseDomain to PD's Query field but it's using a DNS-domain-shaped variable
to carry a region name through a fuzzy text search, which is why it accidentally matches 10 unrelated services.

A cleaner design would separate the concern: either introduce a distinct field/parameter for the PD service query string (so it's clear what's being searched and why), or construct the actual expected service name prefix e.g. rhobs-hcp-prod-us-east-1

Querying PD with just the region string (e.g. us-east-1) returns ~10 services via substring match across all environments and service types. Only the 2 prod RHOBS HCP services are relevant for a production cluster.

  • rhobs-hcp-prod-critical-us-east-1 (relevant)
  • rhobs-hcp-prod-warning-us-east-1 (relevant)
  • rhobs-hcp-stage-us-east-1 (wrong env)
  • rhobs-infra-{critical,warning,stage}-us-east-1 (infra, not cluster alerts)
  • osd-hypershift-us-east-1 (legacy)
  • ...

@RaphaelBut

Copy link
Copy Markdown
Contributor

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 10:45 AM UTC · Completed 11:01 AM UTC

Commit: de7a4d6 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.83

Both cmd/cluster/context.go and cmd/org/context.go were passing the
internal cluster ID to WithClusterID(), but PagerDuty alerts contain
the external UUID. This caused incidentMatchesCluster to always return
false, filtering out all incidents for HCP clusters.

Use ExternalID() instead of ID() in both locations so the cluster ID
matches what PagerDuty stores in alert EventDetails.

Addresses #942
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Fixed the core bug: both cluster and org context commands were passing the internal cluster ID to PagerDuty incident filtering, but PD alerts contain the external UUID. Changed to use ExternalID() in both locations. The design concern about baseDomain carrying a region string is valid but not a correctness issue once the ID fix is applied.

Fixed (2):

  1. Both cmd/cluster/context.go and cmd/org/context.go pass the internal ID instead of the external UUID for PagerDuty cluster matching (cmd/cluster/context.go): Changed WithClusterID(o.clusterID) to WithClusterID(o.externalClusterID) so the external UUID is used for PD incident filtering
  2. cmd/org/context.go passes cluster.ID() (internal) instead of cluster.ExternalID() (external UUID) (cmd/org/context.go): Changed cluster.ID() to cluster.ExternalID() so the external UUID is used for PD incident filtering

Disagreed (1):

  1. baseDomain overloaded with region string matches too many PD services: Valid design concern but not a correctness bug for this PR. Once the external cluster ID fix is applied, incidentMatchesCluster correctly filters incidents by cluster UUID regardless of how many services are queried. Introducing a distinct query parameter or constructing service name prefixes would be a separate refactoring effort beyond this PR's scope. The reviewer acknowledged it 'technically works' and framed the suggestion as 'a cleaner design would' rather than a required fix.

Tests: passed

CI inspections (6):

  1. ci/prow/build (passing, success): Build passed.
  2. ci/prow/format (passing, success): Format check passed.
  3. ci/prow/lint (passing, success): Lint passed.
  4. ci/prow/test (passing, success): Tests passed.
  5. ci/prow/verify-docs (passing, success): Doc verification passed.
  6. tide (passing, pending): Merge queue is pending, waiting for required checks and approvals.
Decision points
  • Whether to refactor baseDomain usage for HCP region queries (alternatives: Introduce a separate PD service query field and construct specific service name prefix, Keep region-as-baseDomain approach since incident filtering ensures correctness; rationale: The current approach produces correct results after the external ID fix. The broader refactor is a valid improvement but out of scope for this bug fix PR.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: c7cd3dde-7ad8-4f48-bbad-c7cede2fefcb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:03 AM UTC · Completed 11:21 AM UTC

Commit: 23f1923 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.23

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Routine bot-authored bug fix with good test coverage, no security or CI concerns; code age of several affected files (1-3 years) and historical fix commits on cluster/context.go are the primary risk drivers, yielding a moderate composite score.

Previous run

Risk Assessment: moderate (2/5)

Details

A bot-authored, well-tested bug fix for a specific HCP PagerDuty lookup gap; moderate line count and blast radius are offset by clean history, zero protected/security-sensitive paths, and a tightly scoped linked issue, yielding a composite of ~1.7 rounded to 2.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Bot-authored bug fix adding HCP-specific PagerDuty routing with solid test coverage (40%) across a medium-sized change; stable files with no recent churn and well-scoped issue alignment keep risk at moderate.

Previous run (3)

Risk Assessment: moderate (2/5)

Details

A small bot-authored bug fix with good test coverage and no security or CI impact; risk is held at moderate primarily by the staleness of four of five touched files (most last modified 6-36 months ago) and the lack of a feature flag guarding the changed lookup path.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Review

Findings

Medium

  • [logic-error] pkg/provider/pagerduty/pagerduty.go:115 — Pre-existing bug now more impactful for HCP: incidentListOffset is declared outside the outer for loop over service IDs and is never reset to 0 between iterations. If the first service requires pagination, subsequent services begin querying at a non-zero offset, silently omitting their first pages of incidents.
    Remediation: Reset incidentListOffset to 0 at the top of each outer loop iteration.

  • [logic-error] pkg/provider/pagerduty/pagerduty.go:178 — Pre-existing bug now more impactful for HCP: the incidents slice is declared outside the outer service-ID loop and is never reset between iterations. Incidents from earlier services accumulate and inflate historical alert counts for subsequent service IDs.
    Remediation: Reset the incidents slice at the start of each outer loop iteration.

Low

  • [naming-conventions] cmd/org/context.go:44 — The first parameter of NewPDClient is named baseDomain but its semantics changed: for HCP clusters the caller passes a region ID. In cmd/cluster/context.go the equivalent was renamed to pdServiceQuery, creating an inconsistency.
    Remediation: Rename the parameter to serviceQuery in both the type definition and the function literal.

  • [edge-case] cmd/account/aws-creds.go:230 — validatePDIncidents was updated with the API rename but does not add HCP support. For HCP clusters, PD incident validation is silently skipped.

  • [test-adequacy] cmd/org/context_test.go:129 — TestNewPDClient_PassesClusterID verifies parameter forwarding but does not exercise the HCP routing logic in FetchContext that determines which values to pass based on HCPRegionID.
    Remediation: Add integration-level tests for the FetchContext HCP routing branch.

  • [naming-conventions] pkg/provider/pagerduty/pagerduty.go:54 — WithClusterID uses all-caps ID while the existing WithTeamIdList uses mixed-case Id, creating a naming inconsistency in the builder API.
    Remediation: Use consistent casing: either WithClusterId to match existing WithTeamIdList, or rename WithTeamIdList to WithTeamIDList for Go-idiomatic consistency.

  • [naming-coherence] cmd/org/context.go:214 — The local variable baseDomain is overwritten with a region ID for HCP clusters, reusing the same variable for two semantically distinct values.
    Remediation: Introduce a serviceQuery variable initialized to baseDomain, then assign regionID to serviceQuery for HCP clusters.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run

Review

Findings

Medium

  • [edge case / silent data loss] pkg/provider/pagerduty/pagerduty.go:158 — For HCP clusters, incidentMatchesCluster returns false whenever EventDetails is nil or lacks a recognized cluster ID key. If the PagerDuty API fails to populate FirstTriggerLogEntry despite the include[]=first_trigger_log_entries parameter (e.g., manually created incidents or email-integration incidents), every incident will silently be dropped. While fail-closed is the safe default for region-based services, there is no logging to indicate incidents were filtered out, making debugging difficult when a user reports "no incidents found" for an HCP cluster with genuine alerts.
    Remediation: Add a debug/warning log line when EventDetails is nil for an incident being filtered.

Low

  • [naming-coherence] cmd/cluster/context.go:59 — The contextOptions.baseDomain field is overloaded to carry either a DNS base domain (Classic) or an AWS region ID (HCP). Inline comments mitigate confusion at the assignment site, but the field name itself is misleading for future maintainers.
    Remediation: Rename the field to pdServiceQuery or similar.

  • [naming-coherence] pkg/provider/pagerduty/pagerduty.go:46 — WithBaseDomain now accepts either a DNS base domain or a region ID. The method name no longer describes its argument.
    Remediation: Consider renaming to WithServiceQuery or adding a separate WithRegion path.

  • [naming-coherence] pkg/utils/ocm.go:750 — HCPRegionID doc comment is PagerDuty-specific, but the function is a general HCP cluster property detector.
    Remediation: Rewrite doc comment to describe the return value without PD-specific rationale.

  • [architectural-coherence] pkg/provider/pagerduty/pagerduty.go:161 — incidentMatchesCluster probes three key variants (cluster_id, clusterID, cluster-id). The canonical key used by HCP PD alerting is not documented in the PR or the linked issue.
    Remediation: Confirm the canonical key and narrow the list to known valid values.

  • [test adequacy] cmd/org/context_test.go:129 — TestNewPDClient_PassesClusterID tests the two-parameter function signature but not the FetchContext integration path (HCP detection, region routing, external ID selection).

  • [test adequacy] pkg/provider/pagerduty/pagerduty_test.go:431 — GetHistoricalAlertsForCluster HCP filtering tests only use the cluster_id key, not alternate key names (clusterID, cluster-id).

  • [code organization] pkg/provider/pagerduty/pagerduty_test.go:193 — Context("WithClusterID") is nested inside Describe("Provider Functionality") instead of Describe("Client Creation") where all other With* setter tests live.
    Remediation: Move the block into Describe("Client Creation").


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (2)

Review

Findings

Medium

  • [test-adequacy] pkg/provider/pagerduty/pagerduty_test.go:241 — All HCP filtering tests use gomock.Any() for the ListIncidentsOptions argument, so they never verify that opts.Includes = []string{"first_trigger_log_entries"} is actually set when clusterID is non-empty. The Includes parameter is the runtime mechanism that causes the PagerDuty API to populate FirstTriggerLogEntry.EventDetails in the response. Without it, EventDetails will be nil at runtime and incidentMatchesCluster will return false for every incident, silently filtering out ALL HCP incidents. The tests mask this because mock responses have EventDetails manually populated.
    Remediation: Use a gomock matcher on the ListIncidentsOptions argument in the HCP test cases to assert that Includes contains "first_trigger_log_entries" when the client has a non-empty clusterID.

  • [architectural-coherence] cmd/cluster/context.go:392 — The three-part HCP guard cluster.Hypershift().Enabled() && cluster.Region() != nil && cluster.Region().ID() != "" is duplicated verbatim in three separate call sites: setup() at line 201, generateContextData() at line 392, and org/context.go FetchContext() at line 216. A comment in generateContextData() explicitly warns that the guard "must match the region check in setup()", confirming the divergence risk. If one instance is updated without the others, baseDomain and clusterID filtering will silently mismatch.
    Remediation: Extract the guard into a helper function (e.g., hcpRegionID(cluster) string returning the region ID or empty string) and use it in all three call sites.

Low

  • [test-adequacy] cmd/org/context_test.go:129 — TestNewPDClient_PassesClusterID only tests that the NewPDClient function field accepts two string arguments by calling it directly. It does not exercise the HCP branching logic in FetchContext that decides when to set baseDomain to the region ID and populate clusterID.

  • [edge-case] pkg/provider/pagerduty/pagerduty.go:161 — incidentMatchesCluster checks three hardcoded key names (cluster_id, clusterID, cluster-id) against EventDetails. If a different variant is used by any alert source, incidents will be silently dropped for HCP clusters. No logging or metric exists when EventDetails is non-nil but contains no recognized cluster ID key.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (3)

Review

Findings

Medium

  • [logic-error] cmd/cluster/context.go:391 — The guard condition for setting WithClusterID (o.cluster.Hypershift().Enabled()) is weaker than the guard used to override baseDomain to the region ID (line 201: o.cluster.Hypershift().Enabled() && o.cluster.Region() != nil && o.cluster.Region().ID() != ""). If an HCP cluster has no region set, baseDomain remains the DNS base domain (classic-style lookup), but clusterID filtering is still applied. Since classic/DNS-based PD services likely do not include cluster_id in their EventDetails, incidentMatchesCluster would return false for every incident, silently dropping all PD alerts for the cluster. By contrast, cmd/org/context.go correctly aligns both conditions.
    Remediation: Align the guard at line 391 with the one at line 201 — change if o.cluster.Hypershift().Enabled() to if o.cluster.Hypershift().Enabled() && o.cluster.Region() != nil && o.cluster.Region().ID() != "", or extract the HCP-with-region check into a helper boolean set once in setup() and reuse it in both locations.

  • [logic-error] pkg/provider/pagerduty/pagerduty.go:169 — GetHistoricalAlertsForCluster does not apply the cluster ID filtering that GetFiringAlertsForCluster now applies when c.clusterID is set. For HCP clusters, PD services are region-based and contain incidents from many clusters. When the --full flag is used on an HCP cluster, the historical alert summary will include incidents from all clusters in the region, producing inflated and misleading occurrence counts.
    Remediation: Apply the same cluster ID filtering in GetHistoricalAlertsForCluster. When c.clusterID is non-empty, include first_trigger_log_entries in the API request and skip incidents where incidentMatchesCluster returns false.

Low

  • [architectural-coherence] pkg/provider/pagerduty/pagerduty.go:36 — The baseDomain field in the client struct is repurposed to carry a region ID for HCP clusters. The field name and every non-HCP call site treat it as a DNS base domain. This dual-purpose semantics may confuse future readers.
    Remediation: Rename the baseDomain field to something semantically neutral such as serviceQuery / WithServiceQuery, or add a separate region field alongside baseDomain.

  • [edge-case] pkg/provider/pagerduty/pagerduty.go:156 — incidentMatchesCluster checks for three key name variants (cluster_id, clusterID, cluster-id) in EventDetails. If the actual alerting system uses a different key name, the filter would silently reject all incidents. Additionally, EventDetails is typed as map[string]string, meaning nested or non-string values would be silently dropped during deserialization.

  • [architectural-coherence] cmd/org/context.go:216 — The HCP detection and baseDomain-to-region substitution logic is duplicated in both cmd/cluster/context.go and cmd/org/context.go. Both guard conditions and the override assignment are identical. This duplication is a contributing factor to the guard mismatch bug above.
    Remediation: Extract the HCP baseDomain/clusterID derivation into a shared helper in the utils package.

  • [naming-convention] pkg/provider/pagerduty/pagerduty.go:51 — The new builder method WithClusterID uses Go-idiomatic uppercase ID, while the pre-existing WithTeamIdList uses mixed-case Id. Same divergence in struct fields: clusterID vs teamIds.
    Remediation: The new code follows Go convention (ID in all caps). If consistency is desired, rename the pre-existing WithTeamIdList/teamIds to WithTeamIDList/teamIDs.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 11:23 AM UTC · Completed 11:38 AM UTC

Commit: 23f1923 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.76

…er ID

Align the WithClusterID guard in generateContextData() with the region
check in setup() so that HCP clusters without a region do not silently
drop all PD alerts. Add cluster ID filtering to GetHistoricalAlertsForCluster
so historical alert summaries for HCP clusters only include incidents
from the target cluster, not all clusters in the region.

Addresses #942
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (bot-triggered)

Fixed both medium-severity logic errors: aligned the WithClusterID guard condition in cluster context to match the region check (preventing silent alert loss for HCP clusters without a region), and added cluster ID filtering to GetHistoricalAlertsForCluster (preventing inflated historical alert counts from unrelated clusters in the same region). Added tests for the new historical filtering. Disagreed on four low-severity refactoring/naming suggestions as out of scope for this bug-fix PR.

Fixed (2):

  1. Guard condition for WithClusterID is weaker than the guard for baseDomain override, causing silent alert loss for HCP clusters without a region (cmd/cluster/context.go): Aligned the guard at line 391 with the region check at line 201 by adding Region() != nil and Region().ID() != "" conditions, preventing cluster ID filtering from being applied when baseDomain is still a DNS domain
  2. GetHistoricalAlertsForCluster does not apply cluster ID filtering for HCP clusters, inflating historical alert counts with incidents from all clusters in the region (pkg/provider/pagerduty/pagerduty.go): Added cluster ID filtering to GetHistoricalAlertsForCluster: when clusterID is set, includes first_trigger_log_entries in the API request and skips incidents that do not match via incidentMatchesCluster. Added two tests covering HCP filtering and classic (empty clusterID) passthrough behavior

Disagreed (4):

  1. baseDomain field repurposed to carry a region ID for HCP clusters; rename to something semantically neutral: Valid design concern but a refactoring effort beyond this bug-fix PR's scope. The field works correctly for both use cases after the guard alignment fix. Renaming would touch every call site and the public builder API, which is a separate change.
  2. incidentMatchesCluster checks only three key name variants; a different key name would silently reject all incidents: The three variants (cluster_id, clusterID, cluster-id) cover the known alert formats. Adding speculative key names without evidence of other formats would not improve correctness. If a new format is discovered, it can be added then.
  3. HCP detection and baseDomain-to-region substitution logic is duplicated in cmd/cluster/context.go and cmd/org/context.go: Valid concern but extracting a shared helper is a refactoring effort beyond this PR's scope. The two locations now have aligned guard conditions, reducing the risk the reviewer identified as a contributing factor.
  4. WithClusterID uses Go-idiomatic uppercase ID while pre-existing WithTeamIdList uses mixed-case Id: The new code follows Go convention (ID in all caps). Renaming the pre-existing WithTeamIdList/teamIds would be a separate cleanup PR that touches unrelated code.

Tests: passed

CI inspections (6):

  1. ci/prow/build (passing, success): Build passed.
  2. ci/prow/format (passing, success): Format check passed.
  3. ci/prow/lint (passing, success): Lint passed.
  4. ci/prow/test (passing, success): Tests passed.
  5. ci/prow/verify-docs (passing, success): Doc verification passed.
  6. tide (unrelated, pending): Merge queue pending on approved and lgtm labels, not related to this PR's code changes.
Decision points
  • Whether to rename baseDomain or extract shared helpers alongside the bug fixes (alternatives: Rename baseDomain to serviceQuery and extract a shared HCP helper, Keep refactoring out of scope and fix only the correctness bugs; rationale: Mixing refactoring with correctness fixes increases diff size and review burden. The two medium-severity bugs are addressed with minimal, targeted changes that do not require restructuring.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Review · ❌ Terminated · Started 11:39 AM UTC · Ended 11:55 AM UTC

Commit: b88c660 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:39 AM UTC · Completed 11:55 AM UTC

Commit: b88c660 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.85

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 11:57 AM UTC · Completed 12:13 PM UTC

Commit: b88c660 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.29

- Extract duplicated HCP region guard (Hypershift().Enabled() &&
  Region() != nil && Region().ID() != "") into utils.HCPRegionID()
  helper, used by all three call sites in cluster/context.go and
  org/context.go to eliminate divergence risk
- Add gomock matchers (hasIncludes/hasNoIncludes) to verify that HCP
  tests assert opts.Includes contains "first_trigger_log_entries"
  and classic cluster tests assert Includes is empty

Addresses #942
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 3 (bot-triggered)

Addressed both Medium findings: extracted the duplicated HCP guard into a shared utils.HCPRegionID helper (eliminating divergence risk across three call sites), and added gomock matchers verifying that opts.Includes is set correctly for HCP vs classic clusters. Disagreed with two Low findings (FetchContext integration test and per-incident logging) with reasoning.

Fixed (2):

  1. HCP filtering tests use gomock.Any() and never verify that opts.Includes contains first_trigger_log_entries when clusterID is non-empty (pkg/provider/pagerduty/pagerduty_test.go): Added custom gomock matchers (hasIncludes/hasNoIncludes) that verify ListIncidentsOptions.Includes. HCP test cases now assert Includes contains 'first_trigger_log_entries', and classic cluster tests assert Includes is empty. Applied to both GetFiringAlertsForCluster and GetHistoricalAlertsForCluster test contexts.
  2. Three-part HCP guard duplicated verbatim in three call sites with divergence risk (pkg/utils/ocm.go): Extracted the guard into utils.HCPRegionID(cluster) which returns the region ID for HCP clusters or empty string for classic clusters. Replaced all three call sites (setup() and generateContextData() in cmd/cluster/context.go, FetchContext() in cmd/org/context.go) with the single helper, eliminating the comment warning about keeping guards in sync.

Disagreed (2):

  1. TestNewPDClient_PassesClusterID only tests function signature, not HCP branching in FetchContext: Testing the HCP branching logic in FetchContext would require building full OCM cluster objects with Hypershift enabled and a Region set, plus mocking subscriptions, OCM client, and cluster lookups. The existing test validates the interface contract. The guard duplication risk (the root cause this finding depends on) is already addressed by extracting utils.HCPRegionID, which is a single tested function. Adding an integration-level test for FetchContext HCP branching is disproportionate effort for a Low finding.
  2. incidentMatchesCluster uses hardcoded key names with no logging for unrecognized keys: Adding per-incident logging in a tight filter loop would generate excessive noise in production when region-based PD services contain incidents from many clusters. The pagerduty package has no structured logger (only fmt.Printf for errors). If all HCP incidents are silently dropped, the user observes an empty alert list, which is already a visible signal. The three key variants (cluster_id, clusterID, cluster-id) cover the known PagerDuty event detail conventions.

Tests: passed

CI inspections (5):

  1. ci/prow/build (passing, success): Build passed.
  2. ci/prow/lint (passing, success): Lint passed.
  3. ci/prow/test (passing, success): Tests passed.
  4. ci/prow/verify-docs (passing, success): Documentation verification passed.
  5. ci/prow/format (unrelated, failure): Format job failed but gofmt, go vet, and go mod tidy all pass locally. The format target also runs mockgen and ci-build (goreleaser), which are CI infrastructure steps unrelated to this PR's code changes. Unable to access Prow logs to confirm, but all other Prow checks (build, lint, test, verify-docs) pass. Recommend re-running the format job after this push. If it persists, investigate the mockgen or ci-build steps in the format target.

Strategy change: Previous iterations made incremental point fixes for individual findings. This iteration takes a structural approach by extracting shared logic into a helper function, addressing the root cause of guard duplication rather than patching individual instances.

Decision points
  • Where to place the HCPRegionID helper function (alternatives: Add to pkg/provider/pagerduty (near PD client), Add to pkg/utils/ocm.go (shared cluster utilities), Add to a new shared package; rationale: pkg/utils/ocm.go already imports cmv1.Cluster and is imported by both cmd/cluster and cmd/org. The function is about cluster topology, not PagerDuty internals. Adding to pagerduty would introduce an OCM SDK dependency into the PD provider.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:15 PM UTC · Completed 12:33 PM UTC

Commit: 998eb7c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.25

fullsend-ai-review[bot]

This comment was marked as outdated.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 12:35 PM UTC · Completed 12:51 PM UTC

Commit: 998eb7c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.14

@RaphaelBut

Copy link
Copy Markdown
Contributor

/fs-fix-stop

@github-actions github-actions Bot added the fullsend-no-fix Skip bot-triggered fix agent runs label Sep 23, 2026
@github-actions

Copy link
Copy Markdown

Fix agent disabled for this PR. Remove the fullsend-no-fix label or use /fs-fix to re-engage.

…w feedback on PR #942

- Rename baseDomain/WithBaseDomain to serviceQuery/WithServiceQuery in
  PD client to accurately describe the dual-purpose parameter (DNS base
  domain for classic, region ID for HCP)
- Rename contextOptions.baseDomain to pdServiceQuery in cluster context
- Add warning log in incidentMatchesCluster when EventDetails is nil to
  aid debugging when HCP incidents are filtered out
- Rewrite HCPRegionID doc comment to be cluster-generic, not PD-specific
- Move WithClusterID test into Client Creation describe block where
  other builder method tests live
- Add alternate key name tests (clusterID, cluster-id) for
  GetHistoricalAlertsForCluster HCP filtering

Addresses #942
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 4 (bot-triggered)

Addressed 6 of 8 review findings with code changes: renamed baseDomain/WithBaseDomain to pdServiceQuery/WithServiceQuery for naming coherence, added warning log for nil EventDetails, rewrote HCPRegionID doc comment, moved WithClusterID test to correct describe block, and added alternate key tests for historical alerts. Disagreed on 2 findings: narrowing key variants requires infrastructure knowledge outside PR scope, and FetchContext integration testing requires substantial OCM mock infrastructure.

Fixed (6):

  1. incidentMatchesCluster returns false with no logging when EventDetails is nil, making HCP debugging difficult (pkg/provider/pagerduty/pagerduty.go): Added a warning log line (fmt.Printf) when EventDetails is nil for an incident being filtered, following the existing logging pattern in the file
  2. contextOptions.baseDomain field name is misleading when it carries a region ID for HCP clusters (cmd/cluster/context.go): Renamed the struct field from baseDomain to pdServiceQuery to accurately describe its dual-purpose role
  3. WithBaseDomain method name no longer describes its argument since it now accepts region IDs (pkg/provider/pagerduty/pagerduty.go): Renamed WithBaseDomain to WithServiceQuery with a doc comment explaining the dual-purpose parameter, updated the internal field from baseDomain to serviceQuery, and updated all callers (cluster/context.go, org/context.go, account/aws-creds.go)
  4. HCPRegionID doc comment is PagerDuty-specific but the function is a general HCP cluster property detector (pkg/utils/ocm.go): Rewrote the doc comment to describe the return value generically without PagerDuty-specific rationale
  5. GetHistoricalAlertsForCluster HCP filtering tests only use the cluster_id key, not alternate key names (pkg/provider/pagerduty/pagerduty_test.go): Added two new test cases covering the clusterID and cluster-id alternate key names for GetHistoricalAlertsForCluster HCP filtering
  6. WithClusterID context is nested inside Provider Functionality instead of Client Creation where other With setter tests live* (pkg/provider/pagerduty/pagerduty_test.go): Moved the WithClusterID context block from Describe(Provider Functionality) into Describe(Client Creation) alongside other builder method tests

Disagreed (2):

  1. incidentMatchesCluster probes three key variants and the canonical key is not documented: The three key variants (cluster_id, clusterID, cluster-id) represent a defensive approach against different alert source naming conventions. Without access to all HCP PD alerting configurations, narrowing the list risks silently dropping real incidents. The current approach is safer and already well-tested. Documenting a canonical key requires knowledge of the alerting infrastructure that is outside this PR's scope.
  2. TestNewPDClient_PassesClusterID does not test the FetchContext integration path (HCP detection, region routing, external ID selection): Testing the FetchContext HCP integration path requires mocking the OCM SDK cluster objects with Hypershift and Region builders, which is a substantial test infrastructure effort beyond the scope of this review finding. The existing test validates the function signature contract, and the PD provider tests cover the filtering logic thoroughly.

Tests: passed

CI inspections (8):

  1. ci/prow/test (passing, success): Unit tests passed.
  2. ci/prow/lint (passing, success): Linting passed.
  3. ci/prow/verify-docs (passing, success): Docs verification passed.
  4. ci/prow/build (passing, pending): Build check still pending on prior commit.
  5. ci/prow/format (passing, pending): Format check still pending on prior commit.
  6. CodeRabbit (passing, success): External review bot passed.
  7. tide (passing, pending): Merge queue status pending.
  8. dependabot (passing, skipped): Auto-merge workflow skipped, not applicable to this PR.

Strategy change: Iteration 4 escalation: shifted from logic/test fixes to naming coherence and observability improvements, which prior iterations did not address.

Decision points
  • Strategy escalation at iteration 4: previous iterations focused on logic and test fixes; this iteration implements the naming renames that prior iterations did not address (alternatives: Repeat similar logic-focused fixes, Address naming coherence and observability findings as the primary strategy; rationale: The reviewer's feedback shifted from logic errors to naming coherence and observability. Addressing these concerns directly rather than repeating prior approaches.)
  • Keep three cluster ID key variants vs narrow to one canonical key (alternatives: Narrow to cluster_id only, Keep all three variants as defensive check; rationale: Without confirmed canonical key from PD alerting infrastructure, narrowing risks silent data loss for HCP clusters using a different variant.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-coder fullsend-ai-coder Bot added the needs-human Agent loop needs human intervention label Sep 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:53 PM UTC · Completed 1:11 PM UTC

Commit: 2e97188 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.12

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:

  • pkg/provider/pagerduty/pagerduty.go (file-level): Line 178 · [medium] logic-error

Pre-existing bug now more impactful for HCP: the incidents slice is declared outside the outer service-ID loop and is never reset between iterations. Incidents from earlier services accumulate and inflate historical alert counts for subsequent service IDs.

Suggested fix: Reset the incidents slice at the start of each outer loop iteration.

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

@@ -106,21 +115,33 @@ func (c *client) GetFiringAlertsForCluster(pdServiceIDs []string) (map[string][]
var incidentListOffset uint = 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

Pre-existing bug now more impactful for HCP: incidentListOffset is declared outside the outer for loop over service IDs and is never reset to 0 between iterations. If the first service requires pagination, subsequent services begin querying at a non-zero offset, silently omitting their first pages of incidents.

Suggested fix: Reset incidentListOffset to 0 at the top of each outer loop iteration.

Comment thread cmd/org/context.go
GetServiceLogs func(string, time.Time, bool, bool) ([]*v1.LogEntry, error)
GetJiraIssues func(clusterID, externalID, filter string) ([]jira.Issue, error)
NewPDClient func(baseDomain string) (PDClient, error)
NewPDClient func(baseDomain, clusterID string) (PDClient, error)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] naming-conventions

The first parameter of NewPDClient is named baseDomain but its semantics changed: for HCP clusters the caller passes a region ID. In cmd/cluster/context.go the equivalent was renamed to pdServiceQuery, creating an inconsistency.

Suggested fix: Rename the parameter to serviceQuery in both the type definition and the function literal.

Comment thread cmd/account/aws-creds.go
}

pdClient := pagerduty.NewClient().WithBaseDomain(baseDomain)
pdClient := pagerduty.NewClient().WithServiceQuery(baseDomain)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

validatePDIncidents was updated with the API rename but does not add HCP support. For HCP clusters, PD incident validation is silently skipped.

Comment thread cmd/org/context_test.go
}
}

func TestNewPDClient_PassesClusterID(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test-adequacy

TestNewPDClient_PassesClusterID verifies parameter forwarding but does not exercise the HCP routing logic in FetchContext that determines which values to pass based on HCPRegionID.

Suggested fix: Add integration-level tests for the FetchContext HCP routing branch.

return c
}

func (c *client) WithClusterID(clusterID string) *client {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] naming-conventions

WithClusterID uses all-caps ID while the existing WithTeamIdList uses mixed-case Id, creating a naming inconsistency in the builder API.

Suggested fix: Use consistent casing: either WithClusterId to match existing WithTeamIdList, or rename WithTeamIdList to WithTeamIDList for Go-idiomatic consistency.

Comment thread cmd/org/context.go
// For HCP clusters, PD services are organized by region
// rather than per-cluster DNS domain. Use the region as
// the query and filter incidents by cluster ID.
baseDomain := cluster.DNS().BaseDomain()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] naming-coherence

The local variable baseDomain is overwritten with a region ID for HCP clusters, reusing the same variable for two semantically distinct values.

Suggested fix: Introduce a serviceQuery variable initialized to baseDomain, then assign regionID to serviceQuery for HCP clusters.

@openshift-ci

openshift-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

@fullsend-ai-coder[bot]: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/format 2e97188 link true /test format
ci/prow/build 2e97188 link true /test build

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fullsend-no-fix Skip bot-triggered fix agent runs needs-human Agent loop needs human intervention ok-to-test Indicates a non-member PR verified by an org member that is safe to test. risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ROSAENG-435] osdctl cluster context should list PagerDuty incidents for HCP

2 participants