Repository navigation
Closed
Labels
area:corearea:data-aware-schedulingassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitionsassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitionskind:bugThis is a clearly a bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet
Description
Apache Airflow version
main (development)
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When testing asset scheduling on Airflow 3, i encountered some errors. DAGs with multiple datasets like:
from airflow.providers.standard.operators.bash import BashOperator
from airflow.datasets import Dataset
from airflow import DAG
import datetime as dt
with DAG(
dag_id='dataset_consumer_two_datasets_as_list',
start_date=dt.datetime(2024, 7, 3),
schedule=[Dataset("ds1", {"some_extra": 1}), Dataset("ds2")],
) as dag2:
task_2 = BashOperator(
task_id="task_2",
bash_command="exit 0;",
)
display this error in the UI:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/urllib/parse.py", line 121, in _decode_args
return tuple(x.decode(encoding, errors) if x else '' for x in args)
File "/usr/local/lib/python3.9/urllib/parse.py", line 121, in <genexpr>
return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'dict' object has no attribute 'decode'
The same happens for any schedule using more than one dataset:
with DAG(
dag_id='dataset_consumer_two_datasets_as_list',
start_date=dt.datetime(2024, 7, 3),
schedule=[Dataset("ds1", {"some_extra": 1}), Dataset("ds2")],
) as dag2:
task_2 = BashOperator(
task_id="task_2",
bash_command="exit 0;",
)
with DAG(
dag_id='dataset_consumer_two_datasets_with_or',
start_date=dt.datetime(2024, 7, 3),
schedule=(Dataset("ds1", {"some_extra": 1}) | Dataset("ds2")),
) as dag3:
task_3 = BashOperator(
task_id="task_3",
bash_command="exit 0;",
)
with DAG(
dag_id='dataset_consumer_complex_logical_dependency',
start_date=dt.datetime(2024, 7, 3),
schedule=(
(Dataset("ds1", {"some_extra": 1}) | Dataset("ds2"))
& (Dataset("ds3") | Dataset("ds4"))
),
) as dag4:
task_4 = BashOperator(
task_id="task_4",
bash_command="exit 0;",
)
with DAG(
dag_id='dataset_consumer_dataset_or_time_schedule',
start_date=dt.datetime(2024, 7, 3),
schedule=AssetOrTimeSchedule(
timetable=CronTriggerTimetable("0 0 * 4 *", timezone="UTC"),
# datasets=(
assets=(
(Dataset("ds1", {"some_extra": 1}) | Dataset("ds2"))
& (Dataset("ds3") | Dataset("ds4", {"another_extra": 345}))
)
),
) as dag5:
task_5 = BashOperator(
task_id="task_5",
bash_command="exit 0;",
)
The interesting part is that single Dataset seems to be working just fine:
with DAG(
dag_id='dataset_consumer_single_dataset',
start_date=dt.datetime(2024, 7, 3),
schedule=Dataset("ds1"),
) as dag0:
task_0 = BashOperator(
task_id="task_0",
bash_command="exit 0;",
)
with DAG(
dag_id='dataset_consumer_single_dataset_as_list',
start_date=dt.datetime(2024, 7, 3),
schedule=[Dataset("ds1")],
) as dag1:
task_1 = BashOperator(
task_id="task_1",
bash_command="exit 0;",
)
with DAG(
dag_id='dataset_consumer_dataset_or_time_schedule',
start_date=dt.datetime(2024, 7, 3),
schedule=AssetOrTimeSchedule(
timetable=CronTriggerTimetable("0 0 * 4 *", timezone="UTC"),
assets=Dataset("ds1")
),
) as dag5:
task_5 = BashOperator(
task_id="task_5",
bash_command="exit 0;",
)
What you think should happen instead?
No response
How to reproduce
Run breeze on main and try running any of the above DAGs.
Operating System
MacOs 15.3.1 (24D70)
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
Breeze
Anything else?
Maybe I'm doing something wrong or my installation is somehow broken, let me know.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
area:corearea:data-aware-schedulingassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitionsassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitionskind:bugThis is a clearly a bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet
Activity
tirkarthi commentedon Mar 11, 2025
Dataset has the following signature. I guess the positional arguments are being passed as name and uri with the extra dict validated against uri. I just checked the docs and they have examples of
Datasetwith uri passed as first argument which might need to be updated.https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html#quickstart
cc: @uranusjr @Lee-W
kacpermuda commentedon Mar 11, 2025
Yes, exactly, I did some more tests and the my passing of
extrais breaking the parsing. Maybe we should try making the stacktrace more descriptive? We can also close the issue as it's obviously my mistake. I was testing this both on Airflow 2 and 3 and did not realize the change in signature.uranusjr commentedon Mar 11, 2025
You’re supposed to pass
extraas a keywoard argument, not positional. We didn’t enforce that well in 2.x. @Lee-W Let’s add a ruff rule to lint this.kacpermuda commentedon Mar 11, 2025
I'll close it now, thanks !
Lee-W commentedon Mar 12, 2025
Would like to confirm what we want to check. if a
dictis passed intoAsset, it needs to be pass toextra=?uranusjr commentedon Mar 13, 2025
Flag if the second positional parameter passed to Asset is a dict.
(To be fair Mypy already catches this, but I am very doubtful any data engineers bother with type checkers.)