AWS hook should automatically refresh credentials when using temporary credentials

Apache Airflow version: 1.10.8 (Patched with latest AWS Hook)

Environment:

Cloud provider or hardware configuration: 4 VCPU 8GB RAM VM
OS (e.g. from /etc/os-release): RHEL 7.7
Kernel (e.g. uname -a): Linux 3.10.0-957.el7.x86_64
Install tools:
Others:
The AWS Hook functionality for AssumeRoleWithSAML is not available in this version, we manually added it via patching the hook file.

What happened:
We've been using this hook for a while now with this issue, basically sts.assume_role and sts.assume_role_with_saml will return temporary credentials that are only valid for eg 1 hour by default. Eventually with long running operators / hooks / sensors some of them fail because the credentials have expired.

Example error messages
An error occurred (ExpiredTokenException) when calling the AssumeRole operation: Response has expired
An error occurred (ExpiredTokenException) when calling the AssumeRoleWithSAML operation: Response has expired
botocore.exceptions.ClientError: An error occurred (ExpiredTokenException) when calling the operation: The security token included in the request is expired

What you expected to happen:

AWS hook should be updated to use boto3 RefreshableCredentials when temporary credentials are in use.

How to reproduce it:

Use any of the assume role methods with the AWS Hook, create a session, wait 1 hour (or whatever expiry period applies to your role), and try and use the hook again.

Anything else we need to know:

I have a solution, please self-assign this.

Update AWS Base hook to use refreshable credentials (#16770) (#16771)

Hi there If you are using an "assume_role"/similar with the aws hook then the credentials _should_ automatically refresh. I tested it using normal boto3 clients but have not tested using a waiter, like what the ECS operator uses. Maybe try add some logging lines to the aws base hook if you can, to see if the expected assume role and refresh credentials methods are being called. Regards

…

On Fri, 15 Oct 2021, 15:22 ss-nef, ***@***.***> wrote: Hello, I am not able to understand how to enable the automatic refresh when using the ECSOperator to orchestrate Fargate Containers. The container I am running is taking more than one hour to complete its task, which is why I am getting the following error when using the ECSOperator: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 220, in execute self._wait_for_task_ended() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 293, in _wait_for_task_ended waiter.wait(cluster=self.cluster, tasks=[self.arn]) File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait Waiter.wait(self, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 334, in wait raise WaiterError( botocore.exceptions.WaiterError: Waiter TasksStopped failed: An error occurred (ExpiredTokenException): The security token included in the request is expired``` what setting do I need to set to enable automtic refreshing, so that the waiter can wait for longer than one hour? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#16770 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGLEIY2ML7DTDURJDTVAH5LUHATKDANCNFSM47WJHO5A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

This is not tied to the Airflow release but to Amazon provider package release. See: http://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html
This change has been releasesd in apache-airflow-providers-amazon v2.1.0

Docker images for Airflow 2.1.0 has been shipped with apache-airflow-providers-amazon==2.0.0,
https://github.com/apache/airflow/blob/constraints-2.1.2/constraints-3.9.txt#L49
so if you are using this image then you need to update the package.
For details, see: http://airflow.apache.org/docs/docker-stack/build.html#example-of-upgrading-airflow-provider-packages

Navigation Menu

Repository navigation

AWS hook should automatically refresh credentials when using temporary credentials #16770

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AWS hook should automatically refresh credentials when using temporary credentials #16770

Description

Activity

baolsen commented on Oct 15, 2021

mik-laj commented on Oct 19, 2021

mik-laj commented on Oct 19, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions