Suspected race condition in .status.phase update: .status.phase is Pending even though all workloads and components are running.

[jgwest]

Describe the bug

Suspected race condition in operator code that updates .status.phase field.

When running this test, .status.phase is stuck in Pending, even though all workloads and components are running.

• Thus the .status.phase should instead be Available

This can be seen in test 1-108_alternate_cluster_roles_cluster_scoped_instance, for example.

Summarizing 1 Failure:
  [FAIL] GitOps Operator Sequential E2E Tests 1-108_alternate_cluster_roles_cluster_scoped_instance [It] verifies that you can add alternate namespaces to ARGOCD_CLUSTER_CONFIG_NAMESPACES, and that the clusterrole and binding created by this feature can be disabled via DefaultClusterScopedRoleDisabled
  /go/src/github.com/redhat-developer/gitops-operator/test/openshift/e2e/ginkgo/sequential/1-108_alternate_cluster_roles_cluster_scoped_instance_test.go:96
[jgw@localhost-lan argocd-operator]$ 

The actual workloads are running correctly, implying it's an issue with the operator not detecting that the workload status has changed.

  'kubectl get all -n alternate-role' output:
  Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
  NAME                                      READY   STATUS    RESTARTS   AGE
  pod/argocd-application-controller-0       1/1     Running   0          8m10s
  pod/argocd-redis-7ccfdcdc47-xvtrj         1/1     Running   0          8m10s
  pod/argocd-repo-server-754667b7f9-24zjw   1/1     Running   0          8m10s
  pod/argocd-server-d444fdf4f-7jsn5         1/1     Running   0          8m10s

  NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
  service/argocd-metrics          ClusterIP   172.30.198.73    <none>        8082/TCP            8m11s
  service/argocd-redis            ClusterIP   172.30.101.3     <none>        6379/TCP            8m11s
  service/argocd-repo-server      ClusterIP   172.30.107.132   <none>        8081/TCP,8084/TCP   8m10s
  service/argocd-server           ClusterIP   172.30.252.7     <none>        80/TCP,443/TCP      8m10s
  service/argocd-server-metrics   ClusterIP   172.30.255.169   <none>        8083/TCP            8m10s

  NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/argocd-redis         1/1     1            1           8m10s
  deployment.apps/argocd-repo-server   1/1     1            1           8m10s
  deployment.apps/argocd-server        1/1     1            1           8m10s

  NAME                                            DESIRED   CURRENT   READY   AGE
  replicaset.apps/argocd-redis-7ccfdcdc47         1         1         1       8m10s
  replicaset.apps/argocd-repo-server-754667b7f9   1         1         1       8m10s
  replicaset.apps/argocd-server-d444fdf4f         1         1         1       8m10s

  NAME                                             READY   AGE
  statefulset.apps/argocd-application-controller   1/1     8m10s

Test failure looks like this:

�[38;5;243m------------------------------�[0m
�[0mGitOps Operator Sequential E2E Tests �[38;5;243m1-108_alternate_cluster_roles_cluster_scoped_instance �[0m�[1mverifies that you can add alternate namespaces to ARGOCD_CLUSTER_CONFIG_NAMESPACES, and that the clusterrole and binding created by this feature can be disabled via DefaultClusterScopedRoleDisabled�[0m
�[38;5;243m/go/src/github.com/redhat-developer/gitops-operator/test/openshift/e2e/ginkgo/sequential/1-108_alternate_cluster_roles_cluster_scoped_instance_test.go:75�[0m
  Deleting Namespace gitops-e2e-test-e3b8d281-f074
  Object exists in ExistByName: openshift-gitops
  ArgoCD status is now Available
  �[1mSTEP:�[0m creating new namespace alternate-role �[38;5;243m@ 06/18/25 11:33:02.511�[0m
  �[1mSTEP:�[0m adding alternate-role to ARGOCD_CLUSTER_CONFIG_NAMESPACES in Subscription �[38;5;243m@ 06/18/25 11:33:02.667�[0m
  �[1mSTEP:�[0m creating an ArgoCD instance in the new namespace, and waiting for it to be available �[38;5;243m@ 06/18/25 11:33:03.07�[0m
  ArgoCD status is not yet Available
  ArgoCD status is not yet Available
(...)
  ArgoCD status is not yet Available
  ArgoCD status is not yet Available
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/redhat-developer/gitops-operator/test/openshift/e2e/ginkgo/sequential/1-108_alternate_cluster_roles_cluster_scoped_instance_test.go:96 �[38;5;243m@ 06/18/25 11:41:13.177�[0m
  executing command: [kubectl logs pod/openshift-gitops-operator-controller-manager-65d94b7c58-br76j manager -n openshift-gitops-operator]