Robust Zero-Downtime CA cert rotation

[c-pius]

Description

Improve our zero-downtime rotation of the CA certificate for the Watch Mechanism. The overall concept is documented here #2986.

This epic serves to orchestrate all tasks needed to get the concept fully realized.

Acceptance Criteria

• no-downtime for the watch mechanism when CA cert is rotated
• CA cert rotation can be triggered as often as we want without breaking flows
• stable e2e testing

Reasons

Current zero-downtime solution works in general, we observed some issues however:

• rotation is missing idempotency
  • if rotation is triggered too frequently, the setup breaks because certain actions (i.e., server cert switch) has not been performed yet
• rotation leads to errors in client certificates (issuing secret not found) propagating to Kyma error status
• existing e2e test is quite flaky

Attachments

[c-pius]

Regarding the next renewal of certificates (dates are the same for dev, stage and prod):

• CA cert Not After: 2026-04-16T13:13:12Z
• CA cert Renewal Time: 2026-02-14T13:13:12Z
• Currently set cert switch before duration 1491h = 62.125d
• => cert switch will happen 2026-02-13T10:13:12Z
  • should be good cert-switch wise
  • the Issuing certificate as Secret does not exist we may still face