Skip to content

Reorganise issue template for providers wrt to testing process link #33439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Aug 17, 2023
Merged

Reorganise issue template for providers wrt to testing process link #33439

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

[Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors)

@eladkal eladkal Aug 16, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW the doc says:
NOTE! You should Ctrl-C and restart the connections to restart airflow components and make sure new provider packages is used.

I think we should further clarify also on how to do it rather than say what needs to be done.
(explain what command to run to restart and explain how to do the verify)

@potiuk potiuk Aug 16, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eladkal -> What would you suggest ? This is unrelated and should be separate PR but I agree 👍

NOTE! You should Ctrl-C and restart the connections to restart airflow components and make sure new provider packages is used.

is quite wrong .

The idea is that you need to restart the components to get the new providers loaded by new interpreters. Maybe you can phrase it well?

@pankajkoti pankajkoti Aug 16, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, can do it in a separate PR :)

Do we mean to elaborate on the steps, like in each of the pane in Tmux session, kill the current process and restart using the corresponding commands like airflow webserver, airflow scheduler, airflow triggerer, airflow celery worker (if using CeleryExecutor), etc.?

@eladkal eladkal Aug 16, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah totally separate PR.

yes, we should have explnation of what we want to do, why we are doing it and how to actually do it.

I think ideally breeze should handle it by it's own (thus making the paragraph we want to introduce now redundant)
Something like:

breeze start-airflow --use-airflow-version 2.2.4 --python 3.8 --backend postgres \
    --load-example-dags --load-default-connections --override-default-provider-versions apache-airflow-providers-amazon==10.10.0rc1

but that is a feature request for later.

@pankajkoti pankajkoti Aug 16, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, Breeze can already handle it when specifying it at startup. We have the section Manually testing release candidate packages linked in the same document

and @potiuk recently enhanced that section a lot by giving numerous examples in PR #32948.

@potiuk potiuk Aug 17, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah totally separate PR.

yes, we should have explnation of what we want to do, why we are doing it and how to actually do it.

I think ideally breeze should handle it by it's own (thus making the paragraph we want to introduce now redundant) Something like:

breeze start-airflow --use-airflow-version 2.2.4 --python 3.8 --backend postgres \
    --load-example-dags --load-default-connections --override-default-provider-versions apache-airflow-providers-amazon==10.10.0rc1

but that is a feature request for later.

Indeed - this precisely what I improved for Airflow recently (and I think it has all the aspects of what you are talking about @eladkal). It would be great if someone takes it as an example and distills that into better instructions if the current provider's ones are not good enough. Other than correcting the strange sentence I am not sure how to enhance current instructions, though?

Maybe someone could create a PR with better description and we could iterate on it to make it clearer ? I am not sure how to more describe the whys and whether the current provider instructions fail to deliver on that @eladkal ?

For the reference this is the current description for Airflow.

Verify the release candidate by Contributors
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the actual users of Apache Airflow test it in their own staging/test installations. Each release candidate is available on PyPI apart from SVN packages, so everyone should be able to install the release candidate version.

But you can use any of the installation methods you prefer (you can even install it via the binary wheels downloaded from the SVN).

Installing release candidate in your local virtual environment
pip install apache-airflow==rc
Optionally it can be followed with constraints

pip install apache-airflow==rc
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-/constraints-3.8.txt"`
Note that the constraints contain python version that you are installing it with.

You can use any of the installation methods you prefer (you can even install it via the binary wheel downloaded from the SVN).

There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow. Running the following command will use tmux inside breeze, create admin user and run Webserver & Scheduler:

breeze start-airflow --use-airflow-version 2.7.0rc1 --python 3.8 --backend postgres
You can also choose different executors and extras to install when you are installing airflow this way. For example in > order to run Airflow with CeleryExecutor and install celery, google and amazon provider (as of Airflow 2.7.0, you need > to have celery provider installed to run Airflow with CeleryExecutor) you can run:

breeze start-airflow --use-airflow-version 2.7.0rc1 --python 3.8 --backend postgres
--executor CeleryExecutor --airflow-extras "celery,google,amazon"
Once you install and run Airflow, you should perform any verification you see as necessary to check that the Airflow works as you expected.


Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:
Expand All @@ -12,10 +16,6 @@ Those are providers that require testing as there were some substantial changes
{%- endfor %}
{%- endfor %}

The guidelines on how to test providers can be found in

[Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors)

<!--

NOTE TO RELEASE MANAGER:
Expand Down