Repository navigation
Update code examples from "classic" operators to TaskFlow decorators in docs #25319
Description
Body
Related to #9415, which updates the code example DAGs to use the TaskFlow API, the documentation throughout should also reflect the same functional style for overall documentation alignment as well as encourage users and bring more visibility to these TaskFlow decorators.
Currently there are 3 core TaskFlow decorators can be used instead of their "classic" operator counterparts.
| Classic Operator | TaskFlow Decorator |
|---|---|
PythonOperator |
@task |
PythonVirtualenvOperator |
@task.virtualenv |
BranchPythonOperator |
@task.branch |
There are a number of doc files that could be updated to the TaskFlow equivalents of the "classic" operators:
- docs/apache-airflow/best-practices.rst
- docs/apache-airflow/faq.rst
- docs/apache-airflow/index.rst
- docs/apache-airflow/tutorial.rst
- I'll be bold and say maybe there shouldn't be a "classic" tutorial and a TaskFlow API tutorial especially since the desire is to encourage TaskFlow and it's been 1.5 years since its release. Open to discuss this of course and what the community thinks here.
- docs/apache-airflow/concepts/dags.rst
- docs/apache-airflow/concepts/operators.rst
- docs/apache-airflow/concepts/params.rst
- docs/apache-airflow/howto/operator/python.rst
- This should be refactored to
_docs/apache-airflow/howto/decorators/python.rst. Add@task.short_circuitTaskFlow decorator #25752 is the last decorator implementation for Python operators. Once that is merged, we should refactor this page and fully sunset the operator doc.
- This should be refactored to
- docs/apache-airflow/upgrading-from-1-10/index.rst
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
[-]Update code examples from "classic" operators to TaskFlow decorators[/-][+]Update code examples from "classic" operators to TaskFlow decorators in docs[/+]hankehly commentedon Jul 27, 2022
Please assign this to me!
josh-fell commentedon Jul 27, 2022
All yours! FYI, if you feel that these updates would be best done in multiple PRs that's totally fine.
Bowrna commentedon Jul 30, 2022
@hankehly Could i take part of the issue if you don't mind? :) I am happy to help with this.
hankehly commentedon Jul 31, 2022
Absolutely. Let's create a PR per file so we don't overlap
Bowrna commentedon Jul 31, 2022
yeah sure @hankehly Please choose the files that you are taking and share it here. I will choose the ones that i will pick and share it too.
hankehly commentedon Aug 11, 2022
#25657 updates the following
Bowrna commentedon Aug 11, 2022
@hankehly I am picking up all the .rst files under folder
docs/apache-airflow/concepts/and making changes in the classic operators. Is that ok?uranusjr commentedon Aug 11, 2022
Copying my comment from the PR for discussion:
Hmm, the static check failures raise a point for discussion. XComArg has an (unfortunate) API design
xcom_arg[key]that can be used to pull an XCom under a different key, but I’m not sure I like it being used withoutput.task.output["foo"]feel to me more like accessing a dict, not pulling another XCom. The API design can be discussed/changed later, but here I think we should decide whether that is a usage we want to encourage or not (personally I’m against).hankehly commentedon Aug 20, 2022
I will finish up the remaining files
hankehly commentedon Aug 20, 2022
I agree with this. How about combining the two? Right now the TaskFlow API tutorial "builds on" the existing tutorial. I say we..
PythonOperatoretc.. to (1) but describe it as "discouraged"If others are ok with this I'll go ahead and create a draft.
josh-fell commentedon Sep 6, 2022
Thinking about this item again, the how-to should be refactored to
_docs/apache-airflow/howto/decorators/python.rstnow that based on #25752 being the last decorator to be added for Python operators. I updated the list in the description.josh-fell commentedon Sep 6, 2022
Yeah go ahead and create a draft. That's going to be the best place for discussion.
hankehly commentedon Sep 8, 2022
Great, I'll get started on that.
eladkal commentedon Oct 6, 2022
We need to be carful with
upgrading-from-1-10/index.rstThis doc initial goal was to upgrade to 2.0 (to assist with breaking changes) so we need to make sure not to introduce syntax that can't work on 2.0
ferruzzi commentedon Jun 27, 2023
If this project is still of interest, it looks like #25711 got
docs/apache-airflow/concepts/dags.rstanddocs/apache-airflow/concepts/operators.rst