Skip to content

Cannot retrieve config from alternative secrets backend. #26754

New issue
New issue
Closed
#26784
Closed
Cannot retrieve config from alternative secrets backend.#26754
#26784
Labels
area:providerskind:bugThis is a clearly a bug

Description

@Flogue
Flogue
opened on Sep 28, 2022

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

Providers included in apache/airflow:2.4.0 docker image:

apache-airflow==2.4.0
apache-airflow-providers-amazon==5.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-mysql==3.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0

Apache Airflow version

2.4

Operating System

AWS Fargate

Deployment

Docker-Compose

Deployment details

We have configure the alternative secrets backend to use AWS SMP:

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"config_prefix": "/airflow2/config", "connections_prefix": "/airflow2/connections", "variables_prefix": "/airflow2/variables"}

What happened

All processes fail with:

`Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 178, in _get_secret
    response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
    from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 122, in _get_config_value_from_secret_backend
    return secrets_client.get_config(config_key)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 167, in get_config
    return self._get_secret(self.config_prefix, key)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 180, in _get_secret
    except self.client.exceptions.ParameterNotFound:
  File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
    from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 178, in _get_secret
    response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
    from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 49, in <module>
    from airflow.models.connection import Connection
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 32, in <module>
    from airflow.models.base import ID_LEN, Base
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line 76, in <module>
    COLLATION_ARGS = get_id_collation_args()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line 70, in get_id_collation_args
    conn = conf.get('database', 'sql_alchemy_conn', fallback='')
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 557, in get
    option = self._get_option_from_secrets(deprecated_key, deprecated_section, key, section)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 577, in _get_option_from_secrets
    option = self._get_secret_option(section, key)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 502, in _get_secret_option
    return _get_config_value_from_secret_backend(secrets_path)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 125, in _get_config_value_from_secret_backend
    'Cannot retrieve config from alternative secrets backend. '
airflow.exceptions.AirflowConfigException: Cannot retrieve config from alternative secrets backend. Make sure it is configured properly and that the Backend is accessible.
cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)

`

What you think should happen instead

Airflow 2.3.4 was using amazon provider 5.0.0 and everything was working fine. Looking at the SystemsManagerParameterStoreBackend class, the client method changed in amazon 5.1.0 (coming with AF 2.4). There use to be a simple boto3.session call. The new code calls for an import of SessionFactory. I do not understand why this import fails though.

How to reproduce

I assume anyone that sets up the parameter store as backend and try to use the docker image (FROM apache/airflow:2.4.0) will run into this issue.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. Flogue
    added
    area:providers
    kind:bugThis is a clearly a bug
    on Sep 28, 2022

  2. boring-cyborg commented on Sep 28, 2022

    @boring-cyborg
    boring-cyborgBot
    on Sep 28, 2022

    Thanks for opening your first issue here! Be sure to follow the issue template!

  3. Taragolis commented on Sep 29, 2022

    @Taragolis
    Taragolis
    on Sep 29, 2022
    Contributor

    Looking at the SystemsManagerParameterStoreBackend class, the client method changed in amazon 5.1.0 (coming with AF 2.4). There use to be a simple boto3.session call. The new code calls for an import of SessionFactory. I do not understand why this import fails though.

    @Flogue Yeah, this added in #25628 for allow switch roles and use additional configuration for create boto3.session.Session in secrets backend.

    However we have again chicken-egg problem 😠 - secrets backend tried to get configuration from itself

    airflow/airflow/providers/amazon/aws/hooks/base_aws.py

    Lines 663 to 676 in c94f978

    def resolve_session_factory() -> type[BaseSessionFactory]:
    """Resolves custom SessionFactory class"""
    clazz = conf.getimport("aws", "session_factory", fallback=None)
    if not clazz:
    return BaseSessionFactory
    if not issubclass(clazz, BaseSessionFactory):
    raise TypeError(
    f"Your custom AWS SessionFactory class `{clazz.__name__}` is not a subclass "
    f"of `{BaseSessionFactory.__name__}`."
    )
    return clazz
    SessionFactory = resolve_session_factory()

    Recommendation for now downgrade to apache-airflow-providers-amazon==5.0.0

  4. Flogue commented on Sep 29, 2022

    @Flogue
    Flogue
    on Sep 29, 2022
    Author

    Already did :) Thank you. Hopefully this is not too big of a headache to fix!

  5. Taragolis commented on Sep 29, 2022

    @Taragolis
    Taragolis
    on Sep 29, 2022
    Contributor

    Hopefully this is not too big of a headache to fix!

    With Circular Import Error it is not straightforward in different cases. Seems like it fail even earlier than I expected initially.
    Right now I think about disable SessionFactory in case if config_prefix set and switch back to previous behaviour. For both SSM and SM backends

  6. potiuk
    closed this as completedin #26784on Oct 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:providerskind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions