AWS Batch Operator ignores job definition retry strategy  #39607

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

There's been 2 recent changes to the AWS BatchOperator:

• Added Feature: retry strategy parameter to Amazon AWS Batch Operator  #35789
• Fix recent change to Batch operator's retry_strategy #35808

They've added the option to specify the retry strategy for the job.

Unfortunately for someone who relies on the default retry strategy, which can be set/customized in the AWS Batch job definition, this value is now ignored

What you think should happen instead?

When the user does not specify a retry_strategy in constructor BatchOperator it should default to None. Then it will automatically fall back to the job definition retry strategy.

It should NOT be set to {"attempt": 1} as it is currently the case.

How to reproduce

• Create an aws job definition with a retry strategy, for example:

  "retryStrategy": {
    "attempts": 5,
    "evaluateOnExit": [
      {
        "onStatusReason": "Essential container in task exited*",
        "action": "exit"
      },
      {
        "onStatusReason": "*",
        "action": "retry"
      }
    ]
  }

• submit an aws BatchOperator with no retry_strategy or retry_strategy=None
• The retry strategy has been overwritten to this:

  "retryStrategy": {
    "attempts": 1,
    "evaluateOnExit": []
  },

Operating System

MWAA

Versions of Apache Airflow Providers

Using https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt

So:

• apache-airflow-providers-amazon==8.16.0
• airflow=2.8.1

But it is still happening with the latest version as far as I can tell:

Deployment

Official Apache Airflow Helm Chart

Deployment details

MWAA

Anything else?

Here's a quick workaround that should work even with older version of the BatchOperator that don't have retry_strategy:

    if hasattr(operator, "retry_strategy"):
        operator.retry_strategy = None

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct