Repository navigation
AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure #34736
Description
Apache Airflow version
2.7.1
What happened
We have to hardcode our version of apache-airflow-providers-microsoft-azure to 6.2.4 in order for AzureContainerInstancesOperator to work as expected. If we bump the version to 6.3.0 or higher we get this error during execution:
[2023-10-03, 19:07:37 EEST] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 196, in execute
if self._ci_hook.exists(self.resource_group, self.name):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 198, in exists
for container in self.connection.container_groups.list_by_resource_group(resource_group):
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
return next(self._page_iterator)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
self._response = self._get_next(self.continuation_token)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 182, in get_next
pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 230, in run
return first_node.send(pipeline_request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
response = self.next.send(request)
[Previous line repeated 2 more times]
File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
response = self.next.send(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
self.on_request(request)
File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
self._token = self._credential.get_token(*self._scopes)
AttributeError: 'ServicePrincipalCredentials' object has no attribute 'get_token'What you think should happen instead
I would expect the operator to be able to connect to Azure without an issue.
How to reproduce
Install airflow 2.7.1 and try running AzureContainerInstancesOperator with some image and run it.
Operating System
Windows 10
Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure=6.3.0 or 7.0.0
Deployment
Docker-Compose
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Oct 3, 2023
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Taragolis commentedon Oct 3, 2023
I not familiar with internals of Azure libraries but guess it somehow related to:
eladkal commentedon Oct 3, 2023
Note that version 6.3.0 updated upstream library:
Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 (#33696)https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/6.3.0/changelog.html#misc
Airflow did not break the interface of the operator (this is why this is not a breaking change release from Airlfow side) but the upstream library did change logic and bumped major release.
You will need to review your code and make sure it's up to date with the newer version of
azure-mgmt-containerinstanceFor the moment - it does not seem like a bug in the provider code.