Flaky “Requested bundle is not configured” error during 2 to 3.1+ upgrade with custom Dag bundle names

Body

related: make bundle_name not nullable #47592

Why

Some of the users encounter the following flaky error for 2 to 3.1+ upgrade.

If user configure the Dag Bundle with other name than dags-folder when running the Make bundle_name not nullable migration, which might raise the following "Requested bundle '{name}' is not configured." error when triggering DagRun right after the migration.

airflow/airflow-core/src/airflow/dag_processing/bundles/manager.py

Lines 333 to 345 in 450a08e

    
               def get_bundle(self, name: str, version: str | None = None) -> BaseDagBundle: 
        
                   """ 
        
                   Get a DAG bundle by name. 
        
                   :param name: The name of the DAG bundle. 
        
                   :param version: The version of the DAG bundle you need (optional). If not provided, ``tracking_ref`` will be used instead. 
        
                   :return: The DAG bundle. 
        
                   """ 
        
                   cfg_bundle = self._bundle_config.get(name) 
        
                   if not cfg_bundle: 
        
                       raise ValueError(f"Requested bundle '{name}' is not configured.") 
        
                   return cfg_bundle.bundle_class(name=name, version=version, **cfg_bundle.kwargs)

Here're some other edge case that will cause the above error:

case 1: Dag Bundle name not dags-folder

AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[{"name": "main", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/files/dags", "refresh_interval": 1}}]'

case 2: Multiple Custom Dag Bundle

AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[{"name": "main1", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/files/dags", "refresh_interval": 1}},  {"name": "main2", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/files/dags", "refresh_interval": 1}}]'

Root Cause

The migration script will only insert dag-folder bundle instead of the correct Dag bundle before the fix for all the above cases.

airflow/airflow-core/src/airflow/migrations/versions/0082_3_1_0_make_bundle_name_not_nullable.py

Lines 76 to 88 in da09404

    
                   conn.execute( 
        
                       text( 
        
                           """ 
        
                           UPDATE dag 
        
                           SET bundle_name = 
        
                               CASE 
        
                                   WHEN fileloc LIKE '%/airflow/example_dags/%' THEN 'example_dags' 
        
                                   ELSE 'dags-folder' 
        
                               END 
        
                           WHERE bundle_name IS NULL 
        
                           """ 
        
                       ) 
        
                   )

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Reassign Dags from unconfigured bundles at DFP startup

Guard 2 to 3+ migration against custom Dag bundle configs using start…

[v3-3-test] Guard 2 to 3+ migration against custom Dag bundle configs…

Guard 2 to 3+ migration against custom Dag bundle configs using start…

Navigation Menu

Repository navigation

Flaky “Requested bundle is not configured” error during 2 to 3.1+ upgrade with custom Dag bundle names #63323

Body

Why

Root Cause

Committer

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def get_bundle(self, name: str, version: str \| None = None) -> BaseDagBundle:
	"""
	Get a DAG bundle by name.

	:param name: The name of the DAG bundle.
	:param version: The version of the DAG bundle you need (optional). If not provided, ``tracking_ref`` will be used instead.

	:return: The DAG bundle.
	"""
	cfg_bundle = self._bundle_config.get(name)
	if not cfg_bundle:
	raise ValueError(f"Requested bundle '{name}' is not configured.")
	return cfg_bundle.bundle_class(name=name, version=version, **cfg_bundle.kwargs)

	conn.execute(
	text(
	"""
	UPDATE dag
	SET bundle_name =
	CASE
	WHEN fileloc LIKE '%/airflow/example_dags/%' THEN 'example_dags'
	ELSE 'dags-folder'
	END
	WHERE bundle_name IS NULL
	"""
	)
	)

Flaky “Requested bundle is not configured” error during 2 to 3.1+ upgrade with custom Dag bundle names #63323

Description

Body

Why

Root Cause

Committer

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions