Skip to content

fix(e2e): read VMB backup type from status.includedVolumes[].type - #2531

Merged
openshift-merge-bot[bot] merged 3 commits into
openshift:oadp-devfrom
kaovilai:fix-e2e-vmb-backup-type-included-volumes
Nov 9, 2026
Merged

openshift-merge-bot[bot] merged 3 commits into
openshift:oadp-devfrom
kaovilai:fix-e2e-vmb-backup-type-included-volumes

Conversation

@kaovilai

@kaovilai kaovilai commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Problem

GetVMBBackupType reads VirtualMachineBackup.status.type. KubeVirt removed that field (kubevirt/kubevirt f1a69f53c, "drop the whole-backup type") in favour of per-volume status.includedVolumes[].type (c32da47a4). On current KubeVirt (community HCO v1.20.0 in virt-kdm-e2e-test-aws) it is never populated, so "full backup, incremental chain, restart, and max-limit fallback" polls until the ephemeral VMB is deleted and times out after 300s, even though the backup itself succeeded.

Change

GetVMBBackupType derives the type from status.includedVolumes[].type, mirroring kubevirt-datamover-controller (which records one type per checkpoint): Incremental if any included volume is incremental, Full only if every volume is full. Unknown volume types are rejected. There is no fallback to status.type; this targets current KubeVirt.

Relation to #2529

#2529 avoids the timeout by reading the DataUpload's expected-backup-type annotation. That value is written by kubevirt-datamover-controller itself, so the "expected vs. actual backup type must not mismatch" assertion can no longer fail. This PR keeps the test checking the type virt-controller actually performed. It conflicts with #2529 on GetVMBBackupType, so one should supersede the other.

Merge order

  1. First: fix: read VMB backup type from status.includedVolumes[].type migtools/kubevirt-datamover-controller#255 (controller reads status.includedVolumes[].type), built into the KDM image that OADP ships and tests with.
  2. Then: this PR.

Merging this PR first is safe but not useful: on current KubeVirt the controller still reports full for incremental backups until migtools/kubevirt-datamover-controller#255 lands, so the incremental assertion here is expected to fail. CI for this PR pulls the KDM fix via the Depends-On line below.

Depends-On: migtools/kubevirt-datamover-controller#255

Testing

Compiles and passes go vet ./tests/e2e/....

5.0-virt-kdm-e2e-test-aws on 0f9ac65, with Depends-On active for this job (openshift/release#88443): oadp-depends-on-build found the Depends-On: line below, built the head of migtools/kubevirt-datamover-controller#255 (4e755aa) inside the test cluster and swapped it in as RELATED_IMAGE_KUBEVIRT_DATAMOVER_CONTROLLER; the e2e step succeeded: https://prow.ci.openshift.org/view/gs/test-platform-results-public/pr-logs/pull/openshift_oadp-operator/2531/pull-ci-openshift-oadp-operator-oadp-dev-5.0-virt-kdm-e2e-test-aws/2118734212679864320

An earlier run of the same job on the same commit, before Depends-On worked, ran the stock controller and failed at backup 4 (virt_backup_restore_suite_test.go:926): the stock controller read the removed status.type, logged Backup type mismatch detected ... expected=incremental actual=full on backups 2 and 3 even though the VMBs reported Full, Incremental, Incremental, and kept the chain length at 1. That run also shows the helper reads includedVolumes[].type correctly.

The Depends-On line can be dropped once #255 is merged and the shipped KDM image carries the fix.

Note

Responses generated with Claude (model: Claude Sonnet 5.5)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Updated end-to-end backup checks to read backup types from included volumes. A backup is classified as incremental if any included volume is incremental; otherwise, it is classified as full.
    • Expanded coverage for case-insensitive backup types and missing or invalid volume details.

KubeVirt dropped VirtualMachineBackup.status.type (kubevirt/kubevirt
f1a69f53c) in favour of per-volume status.includedVolumes[].type, so
GetVMBBackupType never saw a type on current KubeVirt and timed out in the
virt-kdm incremental sequence test.

Derive the type from the included volumes: Incremental only if every
volume is incremental, otherwise Full. This keeps the test checking the
backup type virt-controller actually performed rather than the
controller's own expected-backup-type annotation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <11228024+kaovilai@users.noreply.github.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 3, 2026
@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Walkthrough

The backup type helper now derives the backup type from per-volume values in status.includedVolumes[].type. E2E backup type checks and descriptions now refer to that field.

Changes

Virtual machine backup type

Layer / File(s) Summary
Derive and verify backup type
tests/e2e/lib/virt_helpers.go, tests/e2e/virt_backup_restore_suite_test.go
GetVMBBackupType returns an error if the included-volumes list is missing or empty, a volume type is missing, or a type is unrecognized. It returns Incremental if any volume type is incremental; otherwise, it returns Full. Comparisons ignore case. E2E checks and descriptions identify status.includedVolumes[].type as the backup type field.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · up to 0f9ac

No actionable merge-blocking defect was established; the change is mergeable with normal validation.

Pre-merge checks: Passed 15

Check name Status Explanation
Title check Passed The title names the change: GetVMBBackupType now reads the VirtualMachineBackup backup type from status.includedVolumes[].type. It matches the main change in both the raw summary and the PR objectives…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names Passed PASS: The pull request changes helper logic, comments, and assertion messages only. No Ginkgo test title was added or modified. The base and head declarations are identical, and the existing titles co…
Test Structure And Quality Passed The pull request does not add a new Ginkgo It block or change test setup and cleanup structure. It updates the existing backup-type helper and assertion text only. The relevant checks use a 5-minute…
Microshift Test Compatibility Passed PASS. The PR adds no new Ginkgo tests or test containers. The base and head contain the same It, PIt, Describe, and Context declarations. The changed test lines only update comments and assertion mess…
Single Node Openshift (Sno) Test Compatibility Passed PASS: The pull request does not add a new Ginkgo test. The only test-file changes update comments and assertion messages from Status.Type to status.includedVolumes[].type; all It, Describe, `C…
Topology-Aware Scheduling Compatibility Passed The pull request changes only E2E helper and test files. The diff adds no deployment manifests, operator/controller code, replica settings, affinity, topology spread constraints, node selectors, toler…
Ote Binary Stdout Contract Passed The pull request introduces no process-level stdout writes. Its added code only reads VMB status, returns errors, and changes comments and assertion messages. The diff adds no fmt.Print*, print/printl…
Ipv6 And Disconnected Network Test Compatibility Passed PASS. The pull request changes only VMB backup-type parsing and assertion text. It adds no new Ginkgo test declarations and introduces no IPv4 literals, IP parsing, network policies, URL construction,…
No-Weak-Crypto Passed PASS: The pull request changes only VMB status parsing and test assertions in two Go files. The added diff contains no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom crypto, or non-constant-time sec…
Container-Privileges Passed The pull request changes only two Go test files: tests/e2e/lib/virt_helpers.go and tests/e2e/virt_backup_restore_suite_test.go. The diff adds no container or Kubernetes manifests and introduces no…
No-Sensitive-Data-In-Logs Passed The diff adds no new log statements. New errors report the VMB namespace/name and the included-volume type; the type is an expected Full/Incremental status enum, not a credential, token, PII field…
Description check Passed The description explains why the change is needed and provides test commands and CI results. The Problem and Testing sections cover the template requirements.
Full details: Title check

Explanation

The title names the change: GetVMBBackupType now reads the VirtualMachineBackup backup type from status.includedVolumes[].type. It matches the main change in both the raw summary and the PR objectives. The fix(e2e) prefix is conventional and the title is specific.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 2 files.

Full details: Stable And Deterministic Test Names

Explanation

PASS: The pull request changes helper logic, comments, and assertion messages only. No Ginkgo test title was added or modified. The base and head declarations are identical, and the existing titles contain no generated names, timestamps, node or namespace values, IPs, UUIDs, or legacy metadata prefixes.

Full details: Test Structure And Quality

Explanation

The pull request does not add a new Ginkgo It block or change test setup and cleanup structure. It updates the existing backup-type helper and assertion text only. The relevant checks use a 5-minute Eventually timeout (lines 932-934 and 1080-1084), and the existing test has bounded setup waits plus shared AfterEach cleanup (lines 948-979 and 753-755). The updated assertion messages identify status.includedVolumes[].type. The informing recommendation does not apply because no new test was added.

Full details: Microshift Test Compatibility

Explanation

PASS. The PR adds no new Ginkgo tests or test containers. The base and head contain the same It, PIt, Describe, and Context declarations. The changed test lines only update comments and assertion messages, while the helper changes backup-status parsing. Therefore, the MicroShift compatibility check does not apply.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request does not add a new Ginkgo test. The only test-file changes update comments and assertion messages from Status.Type to status.includedVolumes[].type; all It, Describe, Context, and PIt declarations are unchanged between the reviewed base and head. The helper changes also add no multi-node or HA assumptions.

Full details: Topology-Aware Scheduling Compatibility

Explanation

The pull request changes only E2E helper and test files. The diff adds no deployment manifests, operator/controller code, replica settings, affinity, topology spread constraints, node selectors, tolerations, or PDBs. Therefore, it introduces no topology-dependent scheduling constraints.

Full details: Ote Binary Stdout Contract

Explanation

The pull request introduces no process-level stdout writes. Its added code only reads VMB status, returns errors, and changes comments and assertion messages. The diff adds no fmt.Print*, print/println, klog, os.Stdout, or suite-setup output. Existing log.Printf/log.Println calls are unchanged and use Go's standard logger, which defaults to stderr.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS. The pull request changes only VMB backup-type parsing and assertion text. It adds no new Ginkgo test declarations and introduces no IPv4 literals, IP parsing, network policies, URL construction, or external connectivity. The existing public CirrOS URLs and other network-related code appear only in unchanged context.

Full details: No-Weak-Crypto

Explanation

PASS: The pull request changes only VMB status parsing and test assertions in two Go files. The added diff contains no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom crypto, or non-constant-time secret/token comparison. strings.EqualFold is used only for the non-secret backup type values Full and Incremental.

Full details: Container-Privileges

Explanation

The pull request changes only two Go test files: tests/e2e/lib/virt_helpers.go and tests/e2e/virt_backup_restore_suite_test.go. The diff adds no container or Kubernetes manifests and introduces no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or root-user settings.

Full details: No-Sensitive-Data-In-Logs

Explanation

The diff adds no new log statements. New errors report the VMB namespace/name and the included-volume type; the type is an expected Full/Incremental status enum, not a credential, token, PII field, hostname, or customer payload. Existing assertion messages continue to include the DataUpload name, and no sensitive value is newly logged.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML; Path: .coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: d42a886b-a7b7-41ec-baa6-bb2638ab22f1
Commits

Reviewing files that changed from the base of the PR and between a77cc3c and 0f9ac65.

Files selected for processing: 1

  • tests/e2e/lib/virt_helpers.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2026

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

Comment thread tests/e2e/lib/virt_helpers.go Resolved
Comment thread tests/e2e/lib/virt_helpers.go Resolved
Only Full and Incremental are valid; treating any other value as Full
could let a full-backup assertion pass without evidence of a full backup.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <11228024+kaovilai@users.noreply.github.com>
@kaovilai

kaovilai commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

/test all

@kaovilai

kaovilai commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

/retest

Re-running jobs that failed with S3 403 InvalidAccessKeyId after the CI AWS key was rotated; see #2532.

Note

Responses generated with Claude (model: Claude Sonnet 5.5)

@kaovilai

kaovilai commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

/retest

Re-running jobs that failed with S3 403 InvalidAccessKeyId after the CI AWS key was rotated again; see #2532.

Note

Responses generated with Claude (model: Claude Sonnet 5.5)

kubevirt-datamover-controller records one type per checkpoint and keeps a
mixed backup (some volumes full, some incremental) incremental so the parent
chain is preserved. Mirror that in GetVMBBackupType.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <11228024+kaovilai@users.noreply.github.com>
@kaovilai

kaovilai commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

/test 5.0-e2e-test-aws

@openshift-ci

openshift-ci Bot commented Nov 5, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai, shubham-pampattiwar, sseago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kaovilai

kaovilai commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

/retest

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

The maxIncrementalBackups step in the combined test fails because maxIncrementalBackups counts per-chain, and the VM restart earlier in the test resets the chain. After the restart the per-chain incremental count is only 1, so incrementalCount(1) < maxIncrementalBackups(2) — the controller correctly annotates incremental rather than full.

#2529 fixes this by separating maxIncrementalBackups into a standalone It block with its own VM (no prior restart) and a clean chain (maxInc=1: full → incremental → forced full). Whichever PR survives the conflict on GetVMBBackupType will need that standalone-test approach to avoid this per-chain counting issue.

Also note: the Depends-On for migtools/kubevirt-datamover-controller#255 was not picked up by CI — the build log shows ci-operator tagged the base-branch image (konveyor/kubevirt-datamover-controller:oadp-dev) rather than building from the KDM PR source. However, this is secondary — the chain-count issue would cause the same failure regardless of which KDM image ran.


AI-generated. Review for accuracy.

@kaovilai

kaovilai commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

The Depends-On point is right and matches what I found: no image-swap artifacts for that job, so the controller there did not include migtools/kubevirt-datamover-controller#255.

I don't think the restart explanation holds, though, and I'd rather not restructure the test on it yet:

  • incrementalCount is checkpointLookup.ChainLength - 1 (kubevirt-datamover-controller kubevirt_dataupload_controller.go:887-893), read from the checkpoint chain in the BSL. A VM restart does not modify that index, and the docs say a restart keeps the chain.
  • In the 0f9ac65 run, chainLength=1 already at backup 2's lookup was not caused by a restart (the restart happens after backup 2). It was caused by the controller recording each backup as full after the false expected=incremental actual=full mismatch.
  • The VMB dumps in that run show Full, Incremental, Incremental for backups 1 to 3, i.e. KubeVirt did keep doing incremental backups across the restart.

With #255, the expected sequence is chain length 2 after backup 2, still 2 after the restart, 3 after backup 3, then incrementalCount(2) >= maxIncrementalBackups(2) forces backup 4 to full. I have not confirmed that on a cluster, because #255's image has not run in this job yet. If it still fails after #255 is in the image, the standalone-It approach from #2529 is a reasonable fallback, and I'll revisit it then.

Note

Responses generated with Claude (model: Claude Sonnet 5.5)

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

The maxIncrementalBackups step in the combined test fails because maxIncrementalBackups counts per-chain, and the VM restart earlier in the test resets the chain. After the restart the per-chain incremental count is only 1, so incrementalCount(1) < maxIncrementalBackups(2) — the controller correctly annotates incremental rather than full.

#2529 fixes this by separating maxIncrementalBackups into a standalone It block with its own VM (no prior restart) and a clean chain (maxInc=1: full → incremental → forced full). Whichever PR survives the conflict on GetVMBBackupType will need that standalone-test approach to avoid this per-chain counting issue.


AI-generated. Review for accuracy.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

Agreed—the earlier explanation that the VM restart reset the BSL checkpoint chain was wrong, and I withdraw it. I also verified that the max-incremental count is checkpointLookup.ChainLength - 1.

One timeline clarification from the run: backup 2’s pre-backup lookup had chainLength=1, which is expected for the single full backup already in the chain. After backup 2, the controller logged expected=incremental actual=full and the uploader recorded that checkpoint as full; backup 3’s lookup therefore still had chainLength=1. That lookup is after the VM restart, but the earlier mismatch—not the restart—explains the chain length. The VMB dumps show Full, Incremental, Incremental in status.includedVolumes[].type for backups 1–3. Backup 4 then failed because its DataUpload annotation was incremental while the test expected full.

The Prow run imported the prebuilt konveyor/kubevirt-datamover-controller:oadp-dev image; it built no controller image, so it did not exercise KDM PR #255. The chain sequence with #255 is still unverified on a cluster. I agree with holding off on the standalone-It restructure until a run includes #255’s image and confirms whether the failure remains.


AI-assisted response


AI-generated. Review for accuracy.

@sseago

sseago commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@kaovilai A restart keeps the chain in S3, but upon VM restart, the checkpoint in the VM itself gets lost, so while OADP asks for an incremental backup, kubevirt forces a full one, and then our post-kubevirt processing takes "full" from the status and updates the chain accordingly. When I was debugging this particular failure in my own (now closed) PR, that's exactly what I saw happening. Because the restart caused the chain to reset, we didn't hit max chain length.

@sseago

sseago commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Although it's not 100% clear what's happening now, and I never really got to see whether the restructuring fixed CI since that's when the tests all stopped working.

@sseago

sseago commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/test 5.0-e2e-test-aws

@sseago

sseago commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/retest

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

Two Depends-On issues on this PR:

1. virt-kdm-e2e-test-aws doesn't have the oadp-depends-on-build step at all

openshift/release#84228 added the step to the e2e-test-aws test definitions in openshift-oadp-operator-oadp-dev__5.0.yaml, but not to virt-kdm-e2e-test-aws in the same file. The KDM controller's own config (migtools-kubevirt-datamover-controller-oadp-dev.yaml) was updated, but that only applies to PRs on the KDM repo — not oadp-operator PRs. The virt-kdm-e2e-test-aws test definition needs DEPENDS_ON_CANDIDATES, oadp-depends-on-build, and oadp-apply-depends-on-images added.

2. e2e-test-aws has the step, but it's crashing

The oadp-depends-on-build step ran but failed after 19 seconds in the e2e-test-aws job. It printed Fetching PR description for openshift/oadp-operator#2531 and then died with exit 1 under set -o errexit — no further output. The crash is either the unauthenticated curl -sf to api.github.com hitting a rate limit / network issue, or the grep -Po body extraction failing on the response. This killed the entire pre phase so no e2e tests ran.


AI-generated. Review for accuracy.

@sseago

sseago commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@kaovilai

kaovilai commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

/hold

Holding this PR until migtools/kubevirt-datamover-controller#255 merges, per the merge order in this description (that PR first, then this one). Without the hold, tide merges this PR as soon as the lgtm label lands, which would put it ahead of that PR. In that order, the incremental-backup assertion in this PR fails on oadp-dev until the KDM controller fix ships.

Release the hold with /hold cancel once migtools/kubevirt-datamover-controller#255 has merged. That PR still needs a second approving review: branch protection requires 2 and it has 1 (sseago, on the current head).

Note

Responses generated with Claude (model: Claude Sonnet 5.5)

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 9, 2026
@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 9, 2026
@kaovilai

kaovilai commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

/hold cancel

migtools/kubevirt-datamover-controller#255 has merged; releasing this per the hold condition.

Note

Responses generated with Claude (model: GLM 5.3)

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 9, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 625bd21 into openshift:oadp-dev Nov 9, 2026
23 checks passed
@sseago

sseago commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

/cherry-pick oadp-1.6

@openshift-cherrypick-robot

Copy link
Copy Markdown
Contributor

@sseago: new pull request created: #2540

Details

In response to this:

/cherry-pick oadp-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kaovilai added a commit to kaovilai/oadp-operator that referenced this pull request Nov 10, 2026
KubeVirt dropped VirtualMachineBackup.status.type (kubevirt/kubevirt
f1a69f53c) in favour of per-volume status.includedVolumes[].type, so
GetVMBBackupType never saw a type on current KubeVirt and timed out in the
virt-kdm incremental sequence test.

Derive the type from the included volumes: Incremental only if every
volume is incremental, otherwise Full. This keeps the test checking the
backup type virt-controller actually performed rather than the
controller's own expected-backup-type annotation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tiger Kaovilai <11228024+kaovilai@users.noreply.github.com>

(cherry picked from commit 51dac1b)
backport of openshift#2531 to oadp-1.6 —
unblocks the known oadp-1.6 base-branch KDM flake (see PR discussion)
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
shakyav pushed a commit to shakyav/release that referenced this pull request Nov 10, 2026
…openshift#88443)

* Wire Depends-On image overrides into all oadp-operator e2e presubmits

Follow-up to openshift#84228, which wired
the Depends-On cross-repo image override only into the e2e-test-aws
presubmits. Extend the same wiring (DEPENDS_ON_CANDIDATES copied
per-branch from that file's e2e-test-aws, oadp-depends-on-build pre-step,
oadp-apply-depends-on-images test-step) to every remaining oadp-operator
e2e presubmit:

- e2e-test-cli-aws (oadp-1.4/1.5/1.6/dev, incl. allnamespaces on 5.1)
- virt-kdm-e2e-test-aws / virt-nokdm-e2e-test-aws (oadp-1.6, oadp-dev 5.0)
- e2e-test-kubevirt-aws / e2e-test-hcp-aws (oadp-1.5, oadp-1.6)
- e2e-test-aws-allnamespaces (oadp-dev 5.1)

so a Depends-On: line in a PR description takes effect for the job
that actually exercises the affected specs -- e.g. the virt-kdm job
needed to validate openshift/oadp-operator#2531
together with
migtools/kubevirt-datamover-controller#255
pre-merge.

Not wired, deliberately: periodics/postsubmits (no PR description to
resolve), tls-pqc-readiness/tls13-adherence scanner jobs, oadp-1.3__4.14
(EOL, also excluded by 84228's scope), and hypershift-oadp-plugin e2e
(installs via hypershift workflow + dependency_overrides, not an OLM
Subscription).

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: survive GitHub API unavailability and fix body extraction

The step failed every job it ran in since
openshift#84228 merged, for two
independent reasons:

1. The anonymous api.github.com fetch fails from build-farm step pods
   (their shared egress IP sits over GitHub's anonymous rate limit;
   observed as an immediate failure in every run so far -- 84228's own
   rehearsals, the 88443 rehearsal, and
   openshift/oadp-operator#2531 post-merge
   e2e-test-aws run, all dying at this step before any test ran). With
   set -e this turned every wired presubmit red.

2. Even on a successful fetch, the PR-body extraction never matched:
   GitHub returns pretty-printed JSON ("body": "..." with a space after
   the colon) but the pattern required "body":" with none, so the
   extraction came back empty and the grep no-match exit code killed
   the step under errexit+pipefail.

Fixes:

- Fetch resilience: 3 direct retries, then 3 retries through a one-shot
  curl pod on the test cluster's own egress (different NAT IP than the
  build farm's), then degrade to a no-op with a loud warning and a
  depends-on-fetch-failed.txt marker artifact -- a broken description
  fetch must never fail the job itself.
- Extraction: tolerate the whitespace after the JSON key and neutralize
  the grep no-match exit code.

Validated locally against openshift/oadp-operator#2531:
the fixed extraction finds its Depends-On line naming
migtools/kubevirt-datamover-controller#255,
and a forced all-fetches-failed run logs the warning, writes the marker,
and exits 0.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: align body-prefix strip with the extraction regex

The bash glob strip required at least one space after the JSON colon
while the extraction regex tolerates zero; a compact payload would have
kept its "body":" prefix. Same sed-based regex semantics on both
sides now. Review finding on
openshift#88443.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: fix SC2155 in fetch fallback pod name

Split local declaration from assignment per shellcheck (the only
finding from ci/prow/step-registry-shellcheck on
openshift#88443).

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* Address review: mktemp scratch files, bounded fetch curls, 1.5 virt/hcp candidates

- oadp-depends-on-build: use mktemp for the PR-JSON/curl-stderr scratch
  files and the dependency tarball instead of predictable /tmp names
  (CWE-377), and bound both GitHub fetch curls with connect/max timeouts
  so a stalled connection cannot block the fallback attempts or the
  no-op path. The tarball path also moves to mktemp (same pattern,
  pre-existing from openshift#84228).
- oadp-1.5__4.19/4.20: e2e-test-kubevirt-aws gains the
  migtools/kubevirt-velero-plugin candidate and e2e-test-hcp-aws gains
  openshift/hypershift-oadp-plugin, so a Depends-On naming those repos
  resolves on the jobs that actually exercise them (both entries already
  present in the 1.6/dev lists these were copied from).

Review findings on openshift#88443.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

---------

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
bpresnel-rh pushed a commit to bpresnel-rh/release that referenced this pull request Nov 11, 2026
…openshift#88443)

* Wire Depends-On image overrides into all oadp-operator e2e presubmits

Follow-up to openshift#84228, which wired
the Depends-On cross-repo image override only into the e2e-test-aws
presubmits. Extend the same wiring (DEPENDS_ON_CANDIDATES copied
per-branch from that file's e2e-test-aws, oadp-depends-on-build pre-step,
oadp-apply-depends-on-images test-step) to every remaining oadp-operator
e2e presubmit:

- e2e-test-cli-aws (oadp-1.4/1.5/1.6/dev, incl. allnamespaces on 5.1)
- virt-kdm-e2e-test-aws / virt-nokdm-e2e-test-aws (oadp-1.6, oadp-dev 5.0)
- e2e-test-kubevirt-aws / e2e-test-hcp-aws (oadp-1.5, oadp-1.6)
- e2e-test-aws-allnamespaces (oadp-dev 5.1)

so a Depends-On: line in a PR description takes effect for the job
that actually exercises the affected specs -- e.g. the virt-kdm job
needed to validate openshift/oadp-operator#2531
together with
migtools/kubevirt-datamover-controller#255
pre-merge.

Not wired, deliberately: periodics/postsubmits (no PR description to
resolve), tls-pqc-readiness/tls13-adherence scanner jobs, oadp-1.3__4.14
(EOL, also excluded by 84228's scope), and hypershift-oadp-plugin e2e
(installs via hypershift workflow + dependency_overrides, not an OLM
Subscription).

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: survive GitHub API unavailability and fix body extraction

The step failed every job it ran in since
openshift#84228 merged, for two
independent reasons:

1. The anonymous api.github.com fetch fails from build-farm step pods
   (their shared egress IP sits over GitHub's anonymous rate limit;
   observed as an immediate failure in every run so far -- 84228's own
   rehearsals, the 88443 rehearsal, and
   openshift/oadp-operator#2531 post-merge
   e2e-test-aws run, all dying at this step before any test ran). With
   set -e this turned every wired presubmit red.

2. Even on a successful fetch, the PR-body extraction never matched:
   GitHub returns pretty-printed JSON ("body": "..." with a space after
   the colon) but the pattern required "body":" with none, so the
   extraction came back empty and the grep no-match exit code killed
   the step under errexit+pipefail.

Fixes:

- Fetch resilience: 3 direct retries, then 3 retries through a one-shot
  curl pod on the test cluster's own egress (different NAT IP than the
  build farm's), then degrade to a no-op with a loud warning and a
  depends-on-fetch-failed.txt marker artifact -- a broken description
  fetch must never fail the job itself.
- Extraction: tolerate the whitespace after the JSON key and neutralize
  the grep no-match exit code.

Validated locally against openshift/oadp-operator#2531:
the fixed extraction finds its Depends-On line naming
migtools/kubevirt-datamover-controller#255,
and a forced all-fetches-failed run logs the warning, writes the marker,
and exits 0.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: align body-prefix strip with the extraction regex

The bash glob strip required at least one space after the JSON colon
while the extraction regex tolerates zero; a compact payload would have
kept its "body":" prefix. Same sed-based regex semantics on both
sides now. Review finding on
openshift#88443.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* oadp-depends-on-build: fix SC2155 in fetch fallback pod name

Split local declaration from assignment per shellcheck (the only
finding from ci/prow/step-registry-shellcheck on
openshift#88443).

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

* Address review: mktemp scratch files, bounded fetch curls, 1.5 virt/hcp candidates

- oadp-depends-on-build: use mktemp for the PR-JSON/curl-stderr scratch
  files and the dependency tarball instead of predictable /tmp names
  (CWE-377), and bound both GitHub fetch curls with connect/max timeouts
  so a stalled connection cannot block the fallback attempts or the
  no-op path. The tarball path also moves to mktemp (same pattern,
  pre-existing from openshift#84228).
- oadp-1.5__4.19/4.20: e2e-test-kubevirt-aws gains the
  migtools/kubevirt-velero-plugin candidate and e2e-test-hcp-aws gains
  openshift/hypershift-oadp-plugin, so a Depends-On naming those repos
  resolves on the jobs that actually exercise them (both entries already
  present in the 1.6/dev lists these were copied from).

Review findings on openshift#88443.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>

---------

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-authored-by: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
…ends-On

The KDM (kubevirt-datamover-controller/-plugin) virt-kdm-e2e-test-aws jobs
run oadp-operator's e2e suite from a pre-built image
(konveyor/oadp-operator-e2e-tests), whose build/ci-Dockerfile bakes a
release branch's source in. A Depends-On: line naming an
openshift/oadp-operator PR therefore only ever swapped the operator's
manager image (MANAGER_IMAGE) -- the PR's e2e TEST CODE changes never
executed in the KDM job, so a KDM PR whose spec changes live in an
unmerged oadp-operator PR (e.g.
migtools/kubevirt-datamover-controller#255 needing
openshift/oadp-operator#2531 GetVMBBackupType
rework) could only be tested from the oadp-operator side, and needed
/override otherwise.

Add a TESTS_SOURCE sentinel to oadp-depends-on-build's
DEPENDS_ON_CANDIDATES: for a matching candidate, the depended-on PR's
source tarball is copied to
${SHARED_DIR}/depends-on-tests-source-<org>-<repo>.tar.gz instead of being
built as an image, and nothing is appended to depends-on-images.txt (the
image-apply consumers never see it). The KDM configs' e2e step unpacks
that tarball into a fresh directory when present, verifies it is
openshift/oadp-operator source via go.mod, and runs
make TEST_VIRT_KDM=true test-e2e from it; with no tarball the baked-in
tests run exactly as before. konveyor/oadp-operator-e2e-tests carries the
Go toolchain and the repo's make test-e2e dependencies (built FROM
quay.io/konveyor/builder with kubectl, virtctl, and pre-downloaded
modules baked in), which is what makes this viable.

All 4 migtools KDM configs (controller/plugin x oadp-dev/oadp-1.6) declare
openshift/oadp-operator TESTS_SOURCE; the controller configs keep their
existing MANAGER_IMAGE line so a single Depends-On line resolves both the
manager image and the test source. MANAGER_IMAGE behavior is unchanged,
and jobs without a matching Depends-On line are unaffected. The
openshift-priv KDM mirror configs are left unchanged for a follow-up.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
…_DIR

SHARED_DIR is a ~1 MiB Kubernetes Secret (documented across the
step-registry: qe-agent, quay, stackrox, telcov, aro-hcp steps), while
openshift/oadp-operator source archives are 10+ MB compressed (PR
openshift/oadp-operator#2531 is 11.2 MiB) -- the tarball could never
transit it, so the original copy-the-tarball design would have failed
the positive path.

The resolver now writes a one-line pointer file
(${SHARED_DIR}/depends-on-tests-source-<org>-<repo>.txt) containing the
depended-on PR's archive URL, and each wired e2e step reads the pointer
and fetches the tarball itself:
- the pointer must name exactly an openshift/oadp-operator PR archive
  URL (it decides what gets executed as test code, so anything else
  fails loudly),
- the fetch uses curl --retry and fails loudly rather than silently
  falling back to baked-in tests,
- the go.mod module check is unchanged.

The resolver-side tarball fetch is kept as fail-fast validation (and is
still required by image candidates). oadp-operator-e2e-tests carries
curl (its build/ci-Dockerfile installs kubectl/virtctl with it), so the
consuming image is fine.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
Observed live in the openshift#88500 rehearsal of
virt-kdm-e2e-test-aws: in a pj-rehearse run, ci-operator's
REPO_OWNER/REPO_NAME/PULL_NUMBER identify the openshift/release PR being
rehearsed, not a PR of the repo under test -- so a literal Depends-On
line in the release PR's own description gets resolved by the rehearsal
(that release PR's example line pulled openshift/oadp-operator#2531
manager image and test source into the rehearsal), and the target repo
is always built from its branch head, so a Depends-On-resolved rehearsal
tests branch-head code against the depended-on PR's code.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
…ends-On

The KDM (kubevirt-datamover-controller/-plugin) virt-kdm-e2e-test-aws jobs
run oadp-operator's e2e suite from a pre-built image
(konveyor/oadp-operator-e2e-tests), whose build/ci-Dockerfile bakes a
release branch's source in. A Depends-On: line naming an
openshift/oadp-operator PR therefore only ever swapped the operator's
manager image (MANAGER_IMAGE) -- the PR's e2e TEST CODE changes never
executed in the KDM job, so a KDM PR whose spec changes live in an
unmerged oadp-operator PR (e.g.
migtools/kubevirt-datamover-controller#255 needing
openshift/oadp-operator#2531 GetVMBBackupType
rework) could only be tested from the oadp-operator side, and needed
/override otherwise.

Add a TESTS_SOURCE sentinel to oadp-depends-on-build's
DEPENDS_ON_CANDIDATES: for a matching candidate, the depended-on PR's
source tarball is copied to
${SHARED_DIR}/depends-on-tests-source-<org>-<repo>.tar.gz instead of being
built as an image, and nothing is appended to depends-on-images.txt (the
image-apply consumers never see it). The KDM configs' e2e step unpacks
that tarball into a fresh directory when present, verifies it is
openshift/oadp-operator source via go.mod, and runs
make TEST_VIRT_KDM=true test-e2e from it; with no tarball the baked-in
tests run exactly as before. konveyor/oadp-operator-e2e-tests carries the
Go toolchain and the repo's make test-e2e dependencies (built FROM
quay.io/konveyor/builder with kubectl, virtctl, and pre-downloaded
modules baked in), which is what makes this viable.

All 4 migtools KDM configs (controller/plugin x oadp-dev/oadp-1.6) declare
openshift/oadp-operator TESTS_SOURCE; the controller configs keep their
existing MANAGER_IMAGE line so a single Depends-On line resolves both the
manager image and the test source. MANAGER_IMAGE behavior is unchanged,
and jobs without a matching Depends-On line are unaffected. The
openshift-priv KDM mirror configs are left unchanged for a follow-up.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
…_DIR

SHARED_DIR is a ~1 MiB Kubernetes Secret (documented across the
step-registry: qe-agent, quay, stackrox, telcov, aro-hcp steps), while
openshift/oadp-operator source archives are 10+ MB compressed (PR
openshift/oadp-operator#2531 is 11.2 MiB) -- the tarball could never
transit it, so the original copy-the-tarball design would have failed
the positive path.

The resolver now writes a one-line pointer file
(${SHARED_DIR}/depends-on-tests-source-<org>-<repo>.txt) containing the
depended-on PR's archive URL, and each wired e2e step reads the pointer
and fetches the tarball itself:
- the pointer must name exactly an openshift/oadp-operator PR archive
  URL (it decides what gets executed as test code, so anything else
  fails loudly),
- the fetch uses curl --retry and fails loudly rather than silently
  falling back to baked-in tests,
- the go.mod module check is unchanged.

The resolver-side tarball fetch is kept as fail-fast validation (and is
still required by image candidates). oadp-operator-e2e-tests carries
curl (its build/ci-Dockerfile installs kubectl/virtctl with it), so the
consuming image is fine.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
kaovilai added a commit to kaovilai/release that referenced this pull request Nov 16, 2026
Observed live in the openshift#88500 rehearsal of
virt-kdm-e2e-test-aws: in a pj-rehearse run, ci-operator's
REPO_OWNER/REPO_NAME/PULL_NUMBER identify the openshift/release PR being
rehearsed, not a PR of the repo under test -- so a literal Depends-On
line in the release PR's own description gets resolved by the rehearsal
(that release PR's example line pulled openshift/oadp-operator#2531
manager image and test source into the rehearsal), and the target repo
is always built from its branch head, so a Depends-On-resolved rehearsal
tests branch-head code against the depended-on PR's code.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Co-Authored-By: Claude Code (model: GLM 5.3) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants