Rewrite Breeze in Python

This is follow up and discussion space for the discussion started in https://lists.apache.org/thread.html/ra7e0629922436607de72fa3ea124aa4336df033163f4af0ebcee7b98%40%3Cdev.airflow.apache.org%3E

While I was opposing this while we were focusing on Airflow 2.0, and there are some good reasons why initially I started Breeze in Bash, I think with the current state of Airflow 2.0 betas, with Airflow 2.0 fully based on Python 3.6 and with some "stability" and "good set of features" we have in Breeze and a good level of modularisation we achieved - it's the right time to think about a rewrite.

I did not raise this subject to add a distraction on top of what is already a lot of work for 2.0, but I think having Breeze rewritten in Python could be the "one more thing" that we could do - as a community to make 2.0 experience even better, and one that can make the community even closer.

I was thinking that Breeze is perfect to be split into separate smaller pieces, describe some assumptions that we will have for its use, and turn it into a true community effort where a lot of people will contribute and where we will be able to simplify some of the stuff, and - most importantly - make more people from the community know about how our CI and development environment works and be able to solve any problems there.

Breeze (and underlying bash libraries) are crucial, to get our CI working and I am mostly the single point of contact (and failure!) when it comes to that - I would love to not be one :) and I think with most of the core committers busy with 2.0, this is also an opportunity for more of the contributors to take their part in it (and eventually earn their rank to become committers!). For the core committers, this is an extra opportunity to learn how the system works, influence its design, and possibly simplify some parts of it - even if they will be mostly focused on 2.0.

I would like to do it well - write some assumptions in a design doc, plan the work and split it into separate issues, and lead the effort - but I would love if most of the work is done by others, who would then become familiar with the whole of it.

THIS IS NOT YET A DESIGN PHASE. I am merely trying to see whether there is an interest in taking part in it and helping in the project. Design discussions will follow, and proper project plan will be developed for that, I am looking for people who would like to help with that and become part of the "Breeze 2" team to design and develop it.

Anyone interested - please comment here.

And just another comment here. I would love to have a good mixture of people participating:

core commiters
"fresh" commiters
frequent contributors
casual contributors
users who would like to contribute

Raising my hands up here. Would be interested to hear and help. ✋

Hi! I would like to help with this task. ✋

I would like that! sign me up

I'm interested.

I would like to contribute.

I'm interested too.

Cool! I think the critical mass has been reached :), I am writing a short design doc to start with :)

I would like to contribute if I have something to offer. I do most of the grunt work of keeping Composer and local Airflow in our company running, so I hope finally my experience will come to good open source use.

I'm interested as well.

I am interested.

FINALLY FINALLY IT'S THERE.

Yeah. Finally the time had come to start work on the rewrite:

First, founding PR is here: #19867 - those who are interested will find there some ADR (Architecture Decision Records) which we will keep throughout the whole project to capture all important decisions we've made and once we merge it, more stuff will come.

We have interns working on it - and help of anyone here to comment, test any anything (or even do some stuff !) is MORE than welcome!

@uranusjr

Hey everyone. We are progressing with Breeze2 implementation with Edith and Bowrna and I think we are getting close to start thinking about switching for some users, however I just proposed (based on earlier discussions and proposals with @uranusjr ) a change in the approach of breeze command line that I wanted to get opinion of the current users.

You might lack some context, but I am happy to answer questions here and discuss it here as well or in slack - I described the change proposla in #20921 and I would love some comments of everyone who uses Breeze currently.

Also we are quickly approaching the time when all important decisions are made, some simplifications (#20664 especially) are made in the current Breeze to make it "far less complex" and some basic "working" scaffolding is usable, and I think we might to reach out to some of the people who were happy to help to help us with some tasks.

I think it would be great to engage more people to help @Bowrna and @edithturn to complete their tasks

Just react with hands up if you want to help and I will post here some more details on how you can join the effort in couple of days.

✋

@bhavaniravi

Hey @bhavaniravi -> if you want to help you can take a look at https://github.com/apache/airflow/projects/13 - there some issues in progress but there are some "ready to pick" and you can take a look at the history of those that we've already implemented and possibly pick one of the tasks there :). Happy to explain more if you might have questions.

@Bowrna

Hello here!

Feel the (new) breeze - written in Python.

I have just merged a PR (#22880) that marks a completion of the most important stage of the work that @Bowrna and @edithturn (our interns from Outreachy) have completed - with myself, @eladkal and @xurror mentoring them. Big claps to them!! That was quite an internship :).

It also marks the beginning of a new era of a Bash-less (or Bash-low) dev environment for Airflow. If you’ve used Breeze before, you will get mostly the same things you are used to - but it is written in Python rather than in Bash.

There are many reasons we moved, I will not get into details (see below if you want to know all the context). Vast majority of “user” functionality is ported to new Breeze (with small differences but you will likely find it easy to switch). The UI is soooo much nicer (see screenshots in https://github.com/apache/airflow/blob/main/BREEZE.rst). Also we kept the ./breeze-legacy script and for now it continues to work the same as old Breeze - so you can use it for a while if things do not work (./breeze-legacy and breeze work in parallel).

We will gradually remove Bash code as we get confirmation that Python works for you.

There are some prerequisites that you need to follow (install pipx and setup breeze + autocomplete). But there are generally less than the old breeze, really.

Important - after you install the new breeze completely, you no longer need to run ./breeze. Just breeze is enough. The ./breeze script should continue to work for a while, but we will eventually remove it too.
Please let us know at the #airflow-breeze channel in slack if you have any problems. Any questions/comments are welcome. There will likely be some teething problems.... But remember ./breeze-legacy is there for you just in case something does not work (and it should work seamlessly alongside the new breeze).

Also there are quite a few tasks left in the project https://github.com/apache/airflow/projects/13 that we will be completing - feel free to comment and take any of the tasks left. We will be happy to help to jump-start it.
For those who are interested - the reasoning and all design decisions are kept here: https://github.com/apache/airflow/tree/main/dev/breeze/doc/adr in the form of Architecture Decision Records.
Once again - great job @Bowrna and @edithturn !

Some status with so-far bash remoal for Breeze:

Before:

python:      288754 (96.35%)
sh:           10950 (3.65%)

now:

python:      348937 (97.73%)
sh:            8095 (2.27%)

We have not yet removed some of the legacy code so the numbers are a little misleading, but I hope soon we will be able to remove all of them, we are getting closer to it

https://github.com/apache/airflow/projects/13

And today:

python:      371161 (97.77%)
sh:            5128 (1.35%)
javascript:      3204 (0.84%)
xml:            132 (0.03%)

@Bowrna

After those two PRs: #26612 and #26828 we can officially close this task with Shell code getting to less than

We go from here:

sh:           10945 (3.51%)

to here (less than 1%):

sh:            3536 (0.92%)

@Bowrna @edithturn @eladkal

@Bowrna

Done. Closed. Complete. @Bowrna @edithturn

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Wow Jarek!
Congratulations 👏🏽👏🏽

Congrats Jarek :) I am happy to be part of this

Navigation Menu

Repository navigation

Rewrite Breeze in Python #12282

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Rewrite Breeze in Python #12282

Description

Activity

potiuk commented on Nov 11, 2020

davido912 commented on Nov 11, 2020

pranavssh commented on Nov 11, 2020

Zedmor commented on Nov 12, 2020

caddac commented on Nov 12, 2020

yangrong688 commented on Nov 12, 2020

mik-laj commented on Nov 12, 2020

potiuk commented on Nov 12, 2020

judoole commented on Nov 13, 2020

TobKed commented on Nov 13, 2020

turbaszek commented on Nov 13, 2020

13 remaining items

mrbazzan commented on Oct 10, 2021

potiuk commented on Nov 28, 2021

potiuk commented on Jan 18, 2022

potiuk commented on Jan 18, 2022

edithturn commented on Jan 18, 2022

bhavaniravi commented on Feb 11, 2022

potiuk commented on Feb 11, 2022

potiuk commented on Apr 10, 2022

potiuk commented on Jun 25, 2022

potiuk commented on Sep 3, 2022

potiuk commented on Oct 2, 2022

potiuk commented on Oct 3, 2022

edithturn commented on Oct 3, 2022

Bowrna commented on Oct 4, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions