Error description cannot be shown

Apache Airflow version

2.3.3 (latest released)

What happened

Unfortunately, I cannot get further information about my actual error because of the following KeyError

Traceback (most recent call last):
  File "/usr/local/airflow/dags/common/databricks/operator.py", line 59, in execute
    _handle_databricks_operator_execution(self, hook, self.log, context)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/databricks/operators/databricks.py", line 64, in _handle_databricks_operator_execution
    notebook_error = run_output['error']
KeyError: 'error'

What you think should happen instead

No response

How to reproduce

No response

Operating System

I Assume some Linux distribution

Versions of Apache Airflow Providers

Astronomer Certified: v2.3.3.post1 based on Apache Airflow v2.3.3
Git Version: .release:2.3.3+astro.1+4446ad3e6781ad048c8342993f7c1418db225b25

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template!

@alexott

Hey @alexott - maybe you or someone from your team can help ?

What is the version of the Databricks provider?

What is the version of the Databricks provider? Can you also provide an example of the code - what operator are you using, etc.?

Here the code that run the deployment:

from airflow import AirflowException
from airflow.models import Variable
from airflow.providers.databricks.operators.databricks import (
    DatabricksRunNowOperator,
    XCOM_RUN_ID_KEY,
    XCOM_RUN_PAGE_URL_KEY,
    _handle_databricks_operator_execution
)

class DatabricksRunNowAttachOperator(DatabricksRunNowOperator):
    """
    Custom operator extending the existing DatabricksRunNowOperator.

    The difference is, that it stores the the run_id in a airflow variable and
    uses this to attach to the run if the operator is restarted

    This pattern avoids newly started jobs in case the operator is restarted, e.g. because of a scheduler restart.
    """

    def execute(self, context):
        variable_key = f"{context['task_instance_key_str']}_{XCOM_RUN_ID_KEY}"
        existing_job_run = Variable.get(key=variable_key, default_var=None)
        hook = self._get_hook()
        if existing_job_run is None:
            self.log.info("No job run found for this task, starting new run in Databricks.")
            self.run_id = hook.run_now(self.json)
            self.log.debug(f"Job started. Writing run id to job variable {variable_key}")
            Variable.set(key=variable_key, value=self.run_id)
        else:
            self.log.info(f"Found existing job run {existing_job_run} for this task. Attaching to this run.")
            self.run_id = existing_job_run

        try:
            _handle_databricks_operator_execution(self, hook, self.log, context)
        except AirflowException as e:
            if "failed with terminal state" in str(e):
                self.log.debug(f"Databricks job failed. Cleaning job variable {variable_key}")
                Variable.delete(key=variable_key)
            raise e
        self.log.debug(f"Databricks job terminated. Cleaning job variable {variable_key}")
        Variable.delete(key=variable_key)

DatabricksRunNowAttachOperator(
                task_id=job_config.name,
                job_id=job_config.id,
                polling_period_seconds=job_config.polling_period_seconds,
                retries=job_config.retries
            )

Regarding the Deployment. I currently do not find the concrete version but it is a legacy version (not E2)

what is the version of apache-airflow-providers-databricks ? latest?

it is based on quay.io/astronomer/ap-airflow:2.3.3-onbuild

Does this help?

Before we used quay.io/astronomer/ap-airflow:2.2.5-onbuild. After the update we got this error

it looks like it's caused by this PR: https://github.com/apache/airflow/pull/21709/files - I'll investigate it

@alexott

@alexott Could you already find something out?

I know the culprit (linked PR), but I need to get time to fix it (most probably around weekend...)

assigned you then :)

Correctly handle output of the failed tasks

Correctly handle output of the failed tasks (#25427)

Navigation Menu

Repository navigation

Error description cannot be shown #25286

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error description cannot be shown #25286

Description

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Jul 25, 2022

potiuk commented on Jul 25, 2022

alexott commented on Jul 25, 2022

alexott commented on Jul 25, 2022

Lazloo commented on Jul 25, 2022

Lazloo commented on Jul 25, 2022

alexott commented on Jul 25, 2022

Lazloo commented on Jul 25, 2022

Lazloo commented on Jul 25, 2022

alexott commented on Jul 25, 2022

Lazloo commented on Jul 26, 2022

alexott commented on Jul 26, 2022

potiuk commented on Jul 28, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions