Repository navigation
OADP-8924: Treat missing openshift-apiserver config as no internal registry - #489
Conversation
GetRegistryInfo returned the NotFound error from the openshift-apiserver/config ConfigMap Get. Clusters without that ConfigMap (e.g. HCP guest clusters, where openshift-apiserver runs on the management cluster) then failed every backup/restore item action that calls GetRegistryInfo, such as pods: error executing custom action (groupResource=pods ...): rpc error: code = Unknown desc = configmaps "config" not found On NotFound, cache and return an empty registry hostname with no error, matching the existing handling of a missing docker-registry service on K8s <= 1.11. Other errors are still returned. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Co-Authored-By: Claude Code (model: Claude Sonnet 5.5) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
|
/cherry-pick oadp-1.6 |
|
@kaovilai: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@kaovilai: This pull request references OADP-8924 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved registry discovery can silently omit local image contents from otherwise successful backups.
1 open finding
What changed in this PR
This PR prevents missing API-server configuration from failing pod backups by treating ConfigMap NotFound responses as an empty internal registry hostname.
Changes:
- Cache an empty hostname on NotFound while preserving other errors.
- Add HTTP-backed tests for missing configuration, successful discovery, errors, and caching.
| File | Description |
|---|---|
velero-plugins/common/shared.go |
Handle missing registry configuration without returning an error. |
velero-plugins/common/shared_test.go |
Test registry lookup results and caching behavior. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@kaovilai: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
redhat-chai-bot
left a comment
There was a problem hiding this comment.
Review: OADP-8924 — Downstream consumer safety analysis
I've traced every downstream consumer of GetRegistryInfo() to verify the PR's claim that an empty registry hostname is safe. The claim holds — the fix is correct and safe.
Fix analysis
The production change is minimal and correct: apierrors.IsNotFound(err) on the ConfigMap GET, then log, cache empty, return ("", nil). This mirrors the existing K8s ≤ 1.11 pattern (docker-registry service not found → same graceful empty return). Other errors (500, 403, network) are still propagated — good.
Bonus coverage: A ConfigMap GET against a nonexistent namespace (e.g. openshift-apiserver namespace absent entirely on HCP guest) also returns NotFound, so apierrors.IsNotFound catches that scenario too.
Downstream consumer verification
I audited every file that reads BackupRegistryHostname / RestoreRegistryHostname or calls GetRegistryInfo():
| Consumer | Empty registry behavior | Safe? |
|---|---|---|
util.go SwapContainerImageRefs |
if oldRegistry == "" || newRegistry == "" { return } — immediate no-op |
✅ |
imagestreamtag/restore.go |
localImage := len(backupInternalRegistry) > 0 && ... — guards on non-empty |
✅ |
imagestream/backup.go |
Only logs internalRegistry, no error on empty |
✅ |
imagestream/restore.go |
GetSrcAndDestRegistryInfo reads annotation — empty means no local images identified |
✅ |
deployment/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
pod/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
daemonset/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
statefulset/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
cronjob/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
job/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
replicaset/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
replicationcontroller/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
deploymentconfig/restore.go |
Calls SwapContainerImageRefs (guarded) |
✅ |
buildconfig/restore.go |
Passes annotation through to UpdateCommonSpec |
✅ |
common/restore.go |
Also calls GetRegistryInfo() — fixed by same shared function |
✅ |
The SwapContainerImageRefs guard at util.go:92-94 is the ultimate safety net — every resource restore plugin flows through it.
Test quality
The test is well-structured:
- Uses
httptestserver — no production refactoring needed, no real API dependency - Covers the three key paths: 404 (fix target), 200 (existing happy path), 500 (error passthrough)
- Verifies caching semantics (404/200 cache → 1 API call; 500 doesn't cache → 2 API calls)
- Properly sets
serverVersionto Minor=30 to reach the ConfigMap code path - Returns empty ImageStream list to ensure Step 1 falls through
- Resets
registryInfobetween subtests
One scenario to note (not a blocker)
If the internal registry IS configured on an HCP guest but ImageStreams in openshift are somehow unpopulated (Step 1 misses it), this fix means we silently skip image copy instead of crashing. That's strictly better than the current behavior (crash the whole backup), and it's a pre-existing discovery gap — not something this PR introduces.
CI passes (images ✅, unit-test ✅).
AI-generated. Review for accuracy.
|
/lgtm |
Add a test case to the external HCP backup/restore suite that deploys a workload pod on the guest cluster and includes the guest namespace in the Velero backup. This exercises the openshift-velero-plugin GetRegistryInfo() code path that fails on HCP guest clusters where the openshift-apiserver/config ConfigMap is absent (fixed in openshift/openshift-velero-plugin#489). Changes: - hcp_backup_restore_suite_test.go: add AdditionalBackupNamespaces field to HCPBackupRestoreCase and wire it into runHCPBackupAndRestore so tests can include extra namespaces in the backup. - hcp_external_cluster_backup_restore_suite_test.go: add new test "HCP external cluster guest pod backup test (OADP-8924)" with helpers to create and validate a test pod on the guest cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a test case to the external HCP backup/restore suite that deploys a workload pod on the guest cluster and includes the guest namespace in the Velero backup. This exercises the openshift-velero-plugin GetRegistryInfo() code path that fails on HCP guest clusters where the openshift-apiserver/config ConfigMap is absent (fixed in openshift/openshift-velero-plugin#489). Changes: - hcp_backup_restore_suite_test.go: add AdditionalBackupNamespaces field to HCPBackupRestoreCase and wire it into runHCPBackupAndRestore so tests can include extra namespaces in the backup. - hcp_external_cluster_backup_restore_suite_test.go: add new test "HCP external cluster guest pod backup test (OADP-8924)" with helpers to create and validate a test pod on the guest cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a test case to the external HCP backup/restore suite that deploys a workload pod on the guest cluster and includes the guest namespace in the Velero backup. This exercises the openshift-velero-plugin GetRegistryInfo() code path that fails on HCP guest clusters where the openshift-apiserver/config ConfigMap is absent (fixed in openshift/openshift-velero-plugin#489). Changes: - hcp_backup_restore_suite_test.go: add AdditionalBackupNamespaces field to HCPBackupRestoreCase and wire it into runHCPBackupAndRestore so tests can include extra namespaces in the backup. - hcp_external_cluster_backup_restore_suite_test.go: add new test "HCP external cluster guest pod backup test (OADP-8924)" with helpers to create and validate a test pod on the guest cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a test case to the external HCP backup/restore suite that deploys a workload pod on the guest cluster and includes the guest namespace in the Velero backup. This exercises the openshift-velero-plugin GetRegistryInfo() code path that fails on HCP guest clusters where the openshift-apiserver/config ConfigMap is absent (fixed in openshift/openshift-velero-plugin#489). Changes: - hcp_backup_restore_suite_test.go: add AdditionalBackupNamespaces field to HCPBackupRestoreCase and wire it into runHCPBackupAndRestore so tests can include extra namespaces in the backup. - hcp_external_cluster_backup_restore_suite_test.go: add new test "HCP external cluster guest pod backup test (OADP-8924)" with helpers to create and validate a test pod on the guest cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a test case to the external HCP backup/restore suite that deploys a workload pod on the guest cluster and includes the guest namespace in the Velero backup. This exercises the openshift-velero-plugin GetRegistryInfo() code path that fails on HCP guest clusters where the openshift-apiserver/config ConfigMap is absent (fixed in openshift/openshift-velero-plugin#489). Changes: - hcp_backup_restore_suite_test.go: add AdditionalBackupNamespaces field to HCPBackupRestoreCase and wire it into runHCPBackupAndRestore so tests can include extra namespaces in the backup. - hcp_external_cluster_backup_restore_suite_test.go: add new test "HCP external cluster guest pod backup test (OADP-8924)" with helpers to create and validate a test pod on the guest cluster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
github dumb. closes this due to openshift/release#87392 merging as if it was an issue being fixed. /reopen |
|
@kaovilai: Reopened this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kaovilai, sseago, weshayutin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@kaovilai: new pull request created: #498 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@kaovilai: new pull request created: #499 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |

Problem
Backing up a
virt-launcherpod fails with:GetRegistryInfo(velero-plugins/common/shared.go) looks for the internal registry hostname in theopenshift-apiserver/configConfigMap on K8s minor > 11, after theopenshiftnamespace imagestream lookup yields nothing. On clusters without that ConfigMap the Get returns NotFound, which was returned as-is.common/backup.goExecutepropagates it, and sinceAppliesToincludespods, the whole pod backup fails.When this happens
Both must hold in the cluster where Velero runs: the
openshift-apiserver/configConfigMap is absent, and theopenshiftnamespace imagestream lookup yields no hostname.cluster-openshift-apiserver-operatoron standard OCP. On an HCP (HyperShift) guest no operator runs in the guest, and hypershift creates no such ConfigMap there (the config lives in a ConfigMap namedopenshift-apiserverin the management cluster's control-plane namespace). Based on reading hypershift and operator code; not yet checked on a live guest.openshiftand the registry is up, so the imagestream lookup should succeed and the ConfigMap is never read. The bug is expected on a guest where the imagestream lookup comes up empty, most likely with the ImageRegistry capability disabled (or the registryRemoved), or on non-OpenShift clusters.Removedthe ConfigMap exists without the hostname key, which the plugin already handles.The failure is not specific to virtualization: any backup or restore touching the
AppliesTokinds hits the same call.Fix
On
apierrors.IsNotFound, log, cache an empty registry hostname, and return"", nil. This matches the existing handling of a missingdocker-registryservice on K8s <= 1.11. Other errors are still returned.Consumers of an empty hostname:
imagestreamtag/restore.goguards onlen(backupInternalRegistry) > 0.buildconfig/restore.goandutil.go(SwapContainerImageRefs) are no-ops or pass the annotation through.imagestream/backup.gopasses it toimagecopy.CopyLocalImageStreamImages, which skips internal-registry images when the hostname is empty. On such a cluster, internal images are not copied during backup. This is the same outcome as for any cluster where the registry cannot be discovered, and it is preferable to failing every pod backup.common/restore.gohas the same call and is fixed by the same change.Testing
New
TestGetRegistryInfoConfigMapinvelero-plugins/common/shared_test.gouses anhttptestserver as the API (no production refactor needed) and callsGetRegistryInfotwice per case:("", nil), cached, one API requestThe 404 case fails on unfixed
oadp-devand passes with the fix.go test ./...passes exceptbuildandimagestream, which need/usr/local/kubebuilder/bin/etcd(envtest) and fail for that reason in this environment.Not yet verified on a live HCP guest cluster.
Workaround until this lands
Use the hotfix image, which contains this fix. Multi-arch (linux/amd64 + linux/arm64), built from 6020363, for testing:
To check whether a cluster is exposed (both must hold: the ConfigMap is absent and the
openshiftnamespace has no imagestreams):Backports to
oadp-X.Ybranches to follow after this lands.Notes
spec.backupImages: falseon the DataProtectionApplication does not avoid this bug. In oadp-operator it only omits theOPENSHIFT_IMAGESTREAM_BACKUPenv var on the Velero container (internal/controller/velero.go), skips registry secret handling, and drops the BSL prefix requirement. In this plugin that env var is read only byimagecopy/imagestream.go. The registry lookup is not gated by it:common/backup.go:66andcommon/restore.go:66callGetRegistryInfofor every item the common plugin handles, so the NotFound is still returned. No DPA setting is known to avoid the failing call.config.yaml, transient 5xx) still fail the item; client-go retries a 5xx only when the response carriesRetry-After. Making those non-fatal would also changecommon/restore.goand is left out of this PR.Note
Responses generated with Claude (model: Claude Sonnet 5.5)
Summary by CodeRabbit