Repository navigation
PR #18772 breaks sql_alchemy_conn_cmd config #20092
Description
Apache Airflow version
2.2.1
What happened
#18772 added two options to tmp_configuration_copy() and defaults tasks run as the same user to generate the temp configuration file for the task runner with the options include_env=False and include_cmds=False. This is a change from the previous defaults which materialized the values of *_cmd configs into the JSON dump.
This presents a problem because when using sql_alchemy_conn_cmd to set the database connection and running as the same user the temporary config JSON dump now includes both sql_alchemy_conn, set to the Airflow distribution default as well as the user set sql_alchemy_conn_cmd. And because bare settings take precedence over _cmd versions while the Airflow worker can connect to the configured DB the task runner itself will instead use a non-existent SQLite DB causing all tasks to fail.
TLDR; The temp JSON config dump used to look something like:
{
"core": {
"sql_alchemy_conn": "mysql://...",
...
}
}
Now it looks something like:
{
"core": {
"sql_alchemy_conn": "sqlite:////var/opt/airflow/airflow.db",
"sql_alchemy_conn_cmd": "/etc/airflow/get-config 'core/sql_alchemy_conn'"
...
}
}
But because sql_alchemy_conn is set sql_alchemy_conn_cmd never gets called and tasks are unable to access the Airflow DB.
What you expected to happen
I'm not quite sure what is the preferred method to fix this issue. I guess there are a couple options:
- Remove the bare
sensitive_config_valuesif either_cmdor_secretversions exist - Ignore the Airflow default value in addition to empty values for bare
sensitive_config_valuesduring parsing - Go back to materializing the sensitive configs
How to reproduce
With either Airflow 2.2.1 or 2.2.2 configure the DB with sql_alchemy_conn_cmd and remove the sql_alchemy_conn config from your config file. Then run the Airflow worker and task runner as the same user. Try running any task and see that the task tries to access the default SQLite store instead of the configured one.
Operating System
Debian Bullseye
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
xyu commentedon Dec 7, 2021
I guess @karry-li-lego opened #19833 for this same issue, I'm not sure why they closed it because this problem is still valid on the main trunk branch.
eladkal commentedon Dec 19, 2021
cc @ashb
ashb commentedon Dec 19, 2021
This isn't meant to be the case
potiuk commentedon Dec 19, 2021
Actually it does, and it's even documented:
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html
This is also main reason why we are implementing this one: #18974
sydykov commentedon Jan 20, 2022
Hi! Is someone working on this?
xyu commentedon Jan 28, 2022
@ashb do you have a preferred method to resolve this of the three I mentioned above or perhaps some other idea?
Filter out default configs when overrides exist.
2ps commentedon Feb 18, 2022
Hello, we submitted a PR that works for us on this issue.
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)
Filter out default configs when overrides exist. (#21539)