Add config option [secrets]backends_order

[moiseenkov]

Introduce a new configuration option for specifying secret backends load order:

[secrets]backends_order = custom,environment_variable,metastore

The default value represents current behavior, thus nothing will change for existing users.

[moiseenkov]

@eladkal , please take a look at the updates.

[potiuk]

I was initially against making it configurable, but seeing the simplicity and flexibility, I am in.

[potiuk]

@eladkal ?

[VladaZakharova]

hi there!
@potiuk
Can we merge this one please?

[VladaZakharova]

Hi @potiuk @eladkal ! Are there some other changes we need to make here? Or we can merge this one?

[eladkal]

We are on feature freeze for Airflow 3.
https://lists.apache.org/thread/r26htzl0w3th7pw0l1y31g6s14qbtwwt

[potiuk]

Yeah. I think that might be 3.1

[eladkal]

prevent accidental merge. We are on feature freeze for Airflow 3.
PR can not be merged till main branch is for 3.1

[potiuk]

This looks good to me - but I think it might be worth to raise a devlist discussion for it @VladaZakharova -> there were past discussions about changing the sequence of resolving configurations, and I know people have strong opinion about "fixed" vs. "confifurable" sequence - and there are arguments pro / against each of those options.

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

[eladkal]

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

I agree. I am not comfortable with making this change without the UI indication / other mechanisem that allows dag authors to see the cluster admin setup for backend order

[Crowiant]

Hello @amoghrajesh @uranusjr I responded to your comments in the PR. Could you please help with the review?

[potiuk]

@amoghrajesh @uranusjr -> any comments here? I think there were some important questions/comments from you and I would love to merge that one to make sure it makes it for 3.2 (it looks good in general).

[eladkal]

I don't have time to re-review but I still have some concerns. I advise to tag this as experimental feature so community will use this feature with judgment.
I will however ask to please present a UI image of how the indication of backend order is shown to users.

[potiuk]

I advise to tag this as experimental feature so community will use this feature with judgment.

Good idea @eladkal

[Crowiant]

Hello @potiuk @eladkal Thank you for your comments. I added experimental tag to the feature.
Here are screenshots of how backends order looks on UI:

[potiuk]

Can you please resolve conflicts and fix the remaining static check?

[Crowiant]

Hello @potiuk I updated the PR. Since checks are passing, can we merge if there are no objections?

[potiuk]

Sorry for that - I've been a bit busy - can you please rebase one more time as it needs more conflict resolution - also the UI part is somethign maybe @jason810496 can you re-review?

[potiuk]

@eladkal - looks like your concerns are alleviated - can you please remove "request changes"?

[amoghrajesh]

@eladkal do you mind taking a look at this one again?

@Crowiant would you resolve the merge conflicts too please?

It doesn't seem like we will make this one in time for 3.2

[Crowiant]

Hello @amoghrajesh I rebased and set the backends_order feature for Airflow 3.2.1

@eladkal could you please re-review this PR?

[potiuk]

@moiseenkov This PR has a few issues that need to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 35 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.
• ❌ Other failing CI checks: Failing: Additional PROD image tests / Test e2e integration tests with PROD image / Regular e2e test. Run prek run --from-ref main locally to reproduce. See static checks docs.
• ⚠️ Unresolved review comments: This PR has 10 unresolved review threads from maintainers: @eladkal (MEMBER): 1 unresolved threads; @amoghrajesh (MEMBER): 8 unresolved threads; @uranusjr (MEMBER): 1 unresolved threads. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See pull request guidelines.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[potiuk]

@eladkal could you please re-review this PR?

@moiseenkov @chrisnordqvist -> cna you please rebase and make the PR green again. This can happen multple times, I know, but keeping the PR to be green for review is quite important as otherwise we do not knowwhatis really not working and what needs conflict resolution. We had a lot of troubles recently with ***lots of ** PRS and bad ones and also with lots of flaky tests- they are mitigated now so it should be easier.

[Crowiant]

Hello @potiuk @eladkal
I rebased this PR again. CI fails not because of my changes. Could we please move forward with this PR? It's been a year since it was created...

[potiuk]

Yes we certainly lost track of it

[potiuk]

As soon as we deal with the current overload we shall come back to it.

[potiuk]

@amoghrajesh -> you had comments on it, can you re-review as well? I thnk this one is indeed long overdue.