Skip to content

Running airflow dags backfill --reset-dagruns <dag_id> -s <execution_start_dt> -e <execution_end_dt> results in error when run twice.  #21058

Closed

Description

@uplsh580

Apache Airflow version

2.2.3 (latest released)

What happened

It's the same situation as #21023.
Only change to airflow dags backfill from airflow dags test.

(airflow) [www@np-data-eng-airflow-sync001-lde-jp2v-prod ~]$ airflow dags backfill tutorial --reset-dagruns -s 2022-01-20 -e 2022-01-23
You are about to delete these 9 tasks:
<TaskInstance: tutorial.print_date scheduled__2022-01-20T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.print_date scheduled__2022-01-21T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.print_date scheduled__2022-01-22T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.sleep scheduled__2022-01-20T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.sleep scheduled__2022-01-21T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.sleep scheduled__2022-01-22T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.templated scheduled__2022-01-20T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.templated scheduled__2022-01-21T07:48:54.720148+00:00 [success]>
<TaskInstance: tutorial.templated scheduled__2022-01-22T07:48:54.720148+00:00 [success]>

Are you sure? (yes/no):
y
Traceback (most recent call last):
  File "/home1/www/venv3/airflow/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
    args.func(args)
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 108, in dag_backfill
    dag_run_state=State.NONE,
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/models/dag.py", line 1948, in clear_dags
    dry_run=False,
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/models/dag.py", line 1887, in clear
    dag_run_state=dag_run_state,
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 270, in clear_task_instances
    dr.state = dag_run_state
  File "<string>", line 1, in __set__
  File "/home1/www/venv3/airflow/lib/python3.7/site-packages/airflow/models/dagrun.py", line 194, in set_state
    raise ValueError(f"invalid DagRun state: {state}")
ValueError: invalid DagRun state: None

What you expected to happen

No response

How to reproduce

  1. Setup Airflow 2.2.3
  2. Run any dag (in my case, using tutorial dag file).
  3. Run again same dag by airflow dags backfill command.

Operating System

CentOS Linux 7.9

Versions of Apache Airflow Providers

Providers info
apache-airflow-providers-celery | 2.1.0
apache-airflow-providers-cncf-kubernetes | 3.0.1
apache-airflow-providers-ftp | 2.0.1
apache-airflow-providers-http | 2.0.2
apache-airflow-providers-imap | 2.1.0
apache-airflow-providers-mysql | 2.1.1
apache-airflow-providers-redis | 2.0.1
apache-airflow-providers-slack | 4.1.0
apache-airflow-providers-sqlite | 2.0.1
apache-airflow-providers-ssh | 2.3.0

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. potiuk commented on Jan 24, 2022

    @potiuk
    Member

    Feel free to fix ! If you are fast, we can even cherry-pick to 2.2.4 :)

  2. uranusjr commented on Jan 24, 2022

    @uranusjr
    Member

    We should also add a check in clear_task_instances to catch this mistake, otherwise future regressions are bound to happen.

  3. potiuk commented on Jan 24, 2022

    @potiuk
    Member

    We should also add a check in clear_task_instances to catch this mistake, otherwise future regressions are bound to happen.

    Good point!

  4. uplsh580 commented on Jan 27, 2022

    @uplsh580
    ContributorAuthor

    I create the other PR because the documents(CONTRIBUTORS_QUICK_START.rst, TESTING.rst) also seemed to need an update.
    #21140

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions