Repository navigation
from 'apache-airflow-providers-google' package: No module named 'airflow.providers.postgres' #14286
Description
Apache Airflow version: 2.0.1
Environment:
- Cloud provider or hardware configuration: MacBook Pro
- OS (e.g. from /etc/os-release): Catalina 10.15.7
- Kernel (e.g.
uname -a): Darwin CSchillebeeckx-0589.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64 - Install tools: pip
requirements.txt
apache-airflow[crypto,celery,amazon,mysql,jdbc,password,redis,slack,snowflake,ssh,google,databricks,mongo,zendesk,papermill,salesforce]==2.0.1
pyarrow==0.17.1
iniconfig==1.1.1
sqlparse==0.4.1
google-api-python-client===1.12.8
google-auth==1.27.0
google-api-core==1.26.0
avro-python3==1.10.0
databricks-connect==7.3.8
matplotlib==3.3.4
scikit-learn==0.24.1
ipykernel==5.4.3
flower==0.9.7
What happened:
I'm using MySQL as a backend and am using the GoogleBaseHook; when I run the webserver I see:
webserver_1 | [2021-02-16 23:39:13,008] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.postgres'
webserver_1 | [2021-02-16 23:39:13,009] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.postgres'
webserver_1 | [2021-02-16 23:39:13,066] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.postgres'
webserver_1 | [2021-02-16 23:39:13,067] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.postgres'
If I add the extra package postgres the warnings are not present anymore.
What you expected to happen:
No warnings
How to reproduce it:
Anything else we need to know:
Activity
boring-cyborg commentedon Feb 17, 2021
Thanks for opening your first issue here! Be sure to follow the issue template!
potiuk commentedon Feb 17, 2021
Yep. We could remove the warnings indeed.
OBITORASU commentedon Feb 18, 2021
Can I work on this particular issue? I am assuming addding
postgresto theapache-airflow[crypto,celery,amazon,mysql,jdbc,password,redis,slack,snowflake,ssh,google,databricks,mongo,zendesk,papermill,salesforce]==2.0.1line should cover the warnings (correct me if I am wrong).potiuk commentedon Feb 18, 2021
Sure. Feel free. I think the gist is that we should log those warnings at info level not warning :)
potiuk commentedon Feb 18, 2021
Those warnings here are not really warnings - they will happen if you do not have all providers installed, but they should not be treated as warnings, this is expected behaviour, so it should be INFO.
OBITORASU commentedon Feb 18, 2021
Checking the error log and your suggestions above, should my approach be to introduce changes to
providers_manager.pyfile and change thelog.warningtolog.infoon line 295 to fix this issue? Hoping to hear back from you, thanks for your time and patience!ConstantinoSchillebeeckx commentedon Feb 18, 2021
Even at
INFO, isn't this going to be really chatty? I think the webserver will just keep emitting those statements; I don't know this codebase well at all, but wouldn't a better fix be not to emit that message if it's not needed?potiuk commentedon Feb 18, 2021
This should be a one-time only message per webserver process only at the start of the process. Do you see it more often/repeated @ConstantinoSchillebeeckx ?
16 remaining items
ConstantinoSchillebeeckx commentedon Mar 19, 2021
I can pick this up if no one minds; if someone can nudge me in the right direction that would be great.
potiuk commentedon Mar 19, 2021
@ConstantinoSchillebeeckx - I think the main task is to diagnose it to see what is really the issue here. I expected that those warnings will only be printed once (which would be acceptable) but seems in the webserver they continue spamming the logs - I will have time to take a look tomorrow, but if you want, you can pick it before.
ashb commentedon Mar 19, 2021
On webserver boot up, the app is loaded once before starting gunicorn, then again for each of the (by default 4) workers, and then by default one webserver worker is replaced every 30s. Each time a web process starts up it will load the providers and produce these logs.
potiuk commentedon Mar 19, 2021
Yep. Makes sense. Just started looking at it :)
potiuk commentedon Mar 19, 2021
Yep. Just reproduced it. The warnings come every 30 s
potiuk commentedon Mar 19, 2021
@OBITORASU @ConstantinoSchillebeeckx -> #14903 to address it.
Turn provider's import warnings into debug logs (#14903)