Redshift task in running state but SQL execution does not start when OpenLineage is enabled #39969

Apache Airflow Provider(s)

openlineage

Versions of Apache Airflow Providers

apache-airflow-providers-openlineage==1.8.0
apache-airflow-providers-common-sql==1.12.0
apache-airflow-providers-amazon==8.23.0

Apache Airflow version

2.9

Operating System

GNU Linux

Deployment

Astronomer

Deployment details

No response

What happened

A SQLExecuteQueryOperator in an environment with apache-airflow-providers-common-sql==1.12.0 ran into a problem where the SQL execution never starts but the task is running. This issue only presented itself in a specific task in the DAG and not other tasks executing different SQLs.

Once we disabled OpenLineage, the task instance ran successfully. We enabled it again to see if it would fail, and it did.

This issue was observed both when using Snowflake and Redfshift.

It seems #39113 #39890 solves the issue for Snowflake, but we're still missing some work to avoid the issue happening for Redshift.

What you think should happen instead

Enabling OpenLineage should not affect task execution, even if it parsing, processing and emitting OL events fail.

How to reproduce

.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct