Repository navigation
AWS - GlueJobOperator - stop job on task stop #32137
Description
Description
if you stop a running GlueJobOperator task ( mark fail ) the operator do not call aws glue to stop the runnning task
so the airflow task stop but the aws glue job continue to run
Use case/motivation
Like the KPO would be nice to stop the aws glue job in case the operator is stop
currently the logs of operator are
[2023-06-26, 10:07:32 UTC] {glue.py:149} INFO - Initializing AWS Glue Job: XXXXXXXX. Wait for completion: True
[2023-06-26, 10:07:32 UTC] {glue.py:290} INFO - Checking if job already exists: XXXXXXXXXXX
[2023-06-26, 10:07:32 UTC] {base.py:73} INFO - Using connection ID 'XXXXXXXX' for task execution.
[2023-06-26, 10:07:32 UTC] {credentials.py:1051} INFO - Found credentials from IAM Role: XXXXXXXXXXXX
[2023-06-26, 10:07:33 UTC] {glue.py:169} INFO - You can monitor this Glue Job run at: https://console.aws.amazon.com/gluestudio/home?region=eu-west-1#/job/test_ip/run/XXXXXXXXXXXXXXX
[2023-06-26, 10:07:33 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:39 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:45 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:51 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:57 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:08:02 UTC] {local_task_job.py:276} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2023-06-26, 10:08:02 UTC] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 13175. PIDs of all processes in the group: [13175]
[2023-06-26, 10:08:02 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 13175
[2023-06-26, 10:08:02 UTC] {taskinstance.py:1488} ERROR - Received SIGTERM. Terminating subprocesses.
[2023-06-26, 10:08:02 UTC] {process_utils.py:79} INFO - Process psutil.Process(pid=13175, status='terminated', exitcode=0, started='10:07:31') (13175) terminated with exit code 0
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity