fix(controller): honor scaleDownDelaySeconds before scaling down the intermediate RS

[andrewjamesbrown]

We are experiencing a problem with our use of Istio and Argo Rollouts. During an "interrupted canary", we see:

• Set weight to 0 for all Istio VirtualServices
• Update DestinationRule to point to new canary
• Scale down intermediate replicaset
• Set weight to 0 for all Istio VirtualServices (this is done twice in 1.9 to avoid setting the weight prior to the DR - see fix(trafficrouting): ensure early SetWeight runs after checkReplicasAvailable #4639 and fix(trafficrouting): set weight before hash for new canary #4564

The issue occurs because the Istio control plane is async - it takes time for the configuration to sync to all the proxies, and during this time, we've started to scale down the intermediate replicaset. In this case, it's desirable to delay the scale-down of the intermediate replicaset until the Istio data plane is in sync.

We see this on a 1.9-equivalent release that we build internally.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• My builds are green. Try syncing with master if they are not.
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• I have run all tests locally (including the flaky ones) and they pass on my workstation
• I have used LLM/AI/Agent tools for this PR but I am responsible for all code of this PR
• I understand what the code does and WHY/HOW it works in several scenarios
• I know if my code is just adding new functionality or changing old functionality for existing users
• My organization is added to USERS.md.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 45m 23s ⏱️
120 tests 106 ✅  7 💤 7 ❌
488 runs  453 ✅ 28 💤 7 ❌

For more details on these failures, see this check.

Results for commit f67f89f.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 474 tests   2 473 ✅  3m 19s ⏱️
  129 suites      0 💤
    1 files        1 ❌

For more details on these failures, see this check.

Results for commit f67f89f.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.06%. Comparing base (2ccdae8) to head (3c3e116).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4645      +/-   ##
==========================================
+ Coverage   85.03%   85.06%   +0.02%     
==========================================
  Files         164      164              
  Lines       18989    18992       +3     
==========================================
+ Hits        16148    16155       +7     
+ Misses       1993     1991       -2     
+ Partials      848      846       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Hariharasuthan99]

LGTM!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Hariharasuthan99]

Hello @andrewjamesbrown, I see after you merged master, one of the unit test has failed, could you check and see if you could fix it?