Status of testing of Apache Airflow 2.2.1rc2

@Jorricks

Airflow 2.2.1rc2 testing

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.1rc2.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

Document changes related to PR #16634 (#18644): @Jorricks
Don't install SQLAlchemy/Pendulum adapters for other DBs (#18745): @JavierLopezT @ashb
Linked issues:
- ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block (#17546)
Don't bake ENV and _cmd into tmp config for non-sudo (#18772): @ashb @khalidmammadov
Linked issues:
- Remove duplicate line of code for configuration copy (#18723)
Add Callbacks Section to Logging & Monitoring (#18842): @john-jac
Add information about keepalives for managed Postgres (#18850): @hpatel-higi @potiuk
Linked issues:
- Scheduler stops running: psycopg2.OperationalError: SSL SYSCALL error: EOF detected (#18846)
Fix occassional deadlock on MSSQL test DagMaker cleanup (#18857): @potiuk
Group PATCH DAGrun together with other DAGRun endpoints (#18885): @BasPH
Fix catchup by limiting queued dagrun creation using max_active_runs (#18897): @ephraimbuddy
Check python version before starting triggerer (#18926): @jedcunningham
Try to move "dangling" rows in upgradedb (#18953): @uranusjr @sbialkowski-pixel
Linked issues:
- Migrate from 2.1.4 to 2.2.0 (#18894)
Fix XCom.delete error in Airflow 2.2.0 (#18956): @jordanjeremy
Row lock TI query in SchedulerJob._process_executor_events (#18975): @ephraimbuddy
Fixes #18943 null execution date on insert to task_fail violating NOT NULL (#18979): @robinedwards
Linked issues:
- IntegrityError inserting into task_fail table with null execution_date from TI.handle_failure_with_callback (#18943)
Sentry before send fallback (#18980): @leonsmith @joshzana
Linked issues:
- Missing value for new before_send config causes airflow.exceptions.AirflowConfigException: section/key [sentry/before_send] not found in config (#18963)
Upgrade old DAG/task param format when deserializing from the DB (#18986): @ashb
fail backfill command before loading DAGs if missing args (#18994): @SamWheating
Workaround libstdcpp TLS error (#19010): @potiuk
Allow Param to support a default value of None (#19034): @jedcunningham
Warn about unsupported Python 3.10 (#19060): @mik-laj
Rename trigger page label to Logical Date (#19061): @uranusjr
Remove incorrect type comment in Swagger2Specification._set_defaults classmethod (#19065): @brain-buster
Relax packaging requirement (#19087): @uranusjr @andrewgodwin
Linked issues:
- Add "packaging" to core requirements (#18122)
Change ds, ts, etc. back to use logical date (#19088): @uranusjr
Ensure task state doesn't change when marked as failed/success/skipped (#19095): @ephraimbuddy
add detailed information to logging when a dag or a task finishes. (#19097): @steveyz-astro
Hide tooltip when next run is none (#19112): @bbovenzi
Prevent scheduler crash when serialized dag is missing (#19113): @ephraimbuddy
Fix queued dag runs changes catchup=False behaviour (#19130): @robinedwards
Add test for interval timetable catchup=False (#19145): @uranusjr @robinedwards
Linked issues:
- Fix queued dag runs changes catchup=False behaviour (#19130)
Crerate TI context with data interval compat layer (#19148): @uranusjr @andreychernih
Linked issues:
- TypeError: unsupported operand type(s) for +=: 'NoneType' and 'datetime.timedelta' (#19143)
Faster db migration to Airflow 2.2 (#19166): @jedcunningham
Add TriggererJob to jobs check command (#19179): @mik-laj
Clearing ti.next_method and ti.next_kwargs on task finish. (#19183): @ReadytoRocc @jedcunningham
Linked issues:
- Deferred TI's next_method is not cleared when Clearing a Successful Task (#19120)
- Deferred TI's next_method and next_kwargs not cleared on retries (#18146)
Fix airflow jobs check cmd for TriggererJob (#19185): @mik-laj
Add DagRun.logical_date as a property (#19198): @uranusjr
Fix Unexpected commit error in schedulerjob (#19213): @ephraimbuddy
Linked issues:
- Ensure task state doesn't change when marked as failed/success/skipped (#19095)

Thanks to all who contributed to the release (probably not a complete list!):
@ephraimbuddy @JavierLopezT @jedcunningham @sbialkowski-pixel @Jorricks @khalidmammadov @john-jac @brain-buster @andrewgodwin @joshzana @SamWheating @mik-laj @leonsmith @ReadytoRocc @ashb @robinedwards @potiuk @bbovenzi @jordanjeremy @steveyz-astro @BasPH @hpatel-higi @uranusjr @andreychernih

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

python3 -m pip install apache-airflow[postgres]==2.2.1rc2 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc2/constraints-3.9.txt"

log

ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    flask-appbuilder 3.3.2 depends on Flask-OpenID<2 and >=1.2.5
    The user requested (constraint) flask-openid==1.2.5

@uranusjr

python3 -m pip install apache-airflow[postgres]==2.2.1rc2 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc2/constraints-3.9.txt"

log

ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    flask-appbuilder 3.3.2 depends on Flask-OpenID<2 and >=1.2.5
    The user requested (constraint) flask-openid==1.2.5

This error makes no sense >=1.2.5 is ok for 1.2.5.

Are you sjure you used the right Python version?
Can you upgrade PIP to latest version

@uranusjr - I have no time to take a look at it soon, can you please take a look? I think we had similar case before that wrong conflict was reported

Yeah the error is weird !!

using python 3.9 ( pip version is 21.3.1 )

but for 2.2.1rc1 I do not have the problem

python3 -m pip install apache-airflow[postgres]==2.2.1rc1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.1rc1/constraints-3.9.txt"

The final error message is bad, but an error is emitted that indicates the real issue:

Collecting Flask-OpenID==1.2.5
  Downloading Flask-OpenID-1.2.5.tar.gz (43 kB)
     |████████████████████████████████| 43 kB 699 kB/s
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/uranusjr/Documents/play/airflow-221rc2-flask-openid/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ydevayvc
       cwd: /tmp/pip-install-iqrcgtl5/flask-openid_b6dd0035aaa24038a0cbc5a5b9b113c7/
  Complete output (1 lines):
  error in Flask-OpenID setup command: use_2to3 is invalid.
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d1/a2/9d1fba3287a65f81b9d1c09c4f7cb16f8ea4988b1bc97ffea0d60983338f/Flask-OpenID-1.2.5.tar.gz#sha256=5a8ffe1c8c0ad1cc1f5030e1223ea27f8861ee0215a2a58a528cc61379e5ccab (from https://pypi.org/simple/flask-openid/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

So this is the same use_2to3 issue we had previously. The constraints file should pin Flask-OpenID to 1.3.0 (which has a pre-built wheel available) instead of 1.2.5.

So this is the same use_2to3 issue we had previously. The constraints file should pin Flask-OpenID to 1.3.0 (which has a pre-built wheel available) instead of 1.2.5.

Fixing it!

@uranusjr

Thanks @uranusjr I knew I saw it before!

@raphaelauv

Resolved - the constraint tag was wrongly set (it was set to be the same as 2.1.0b1. It should work now @raphaelauv !

@potiuk

@potiuk yes now it work , sorry for not pointing to the reel error , thanks @uranusjr

Cool! Also Docker Images are rebuilt and pushed now with the updated constraints. All should be good for testing 🚀

Just tested - the #19010 - works fine - LD_PRELOAD is propely set in the image 👍

#18943 is fixed

I just discovered #19252 not sure if its a blocker though

@robinedwards

Yeah that sounds like a regression unfortunately :( -- was it working in 2.2.0 @robinedwards ?

No it doesn't work in 2.2.0 either

Aah ok -- in that case it is not a regression so not a blocker. Let's fix it for 2.2.2

#18980 looks good 👍

confirmed that #18956 is fixed

Tested #18926, #19034, #19166, #19183, #19179, #19185.

I also partially tested #19088, though I didn't tick this one off. I only verified manual runs logged into their own files which only uses ts.

I also ticked off all the "docs/test only" changes (should have been removed from the issue in the first place which we can do in the future).

I guess we have few more hours to test the remaining ones (if possible) :)

Thanks everyone for testing! Apache Airflow 2.2.1 (based on RC2) has been accepted.

I'll continue with the release process, and the release announcement will follow shortly.

Navigation Menu

Repository navigation

Status of testing of Apache Airflow 2.2.1rc2 #19246

Airflow 2.2.1rc2 testing

Committer

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Status of testing of Apache Airflow 2.2.1rc2 #19246

Description

Airflow 2.2.1rc2 testing

Committer

Activity

raphaelauv commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

kaxil commented on Oct 27, 2021

raphaelauv commented on Oct 27, 2021

uranusjr commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

raphaelauv commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

potiuk commented on Oct 27, 2021

robinedwards commented on Oct 27, 2021

robinedwards commented on Oct 27, 2021

kaxil commented on Oct 27, 2021

robinedwards commented on Oct 27, 2021

kaxil commented on Oct 27, 2021

leonsmith commented on Oct 27, 2021

jordanjeremy commented on Oct 28, 2021

jedcunningham commented on Oct 29, 2021

potiuk commented on Oct 29, 2021

jedcunningham commented on Oct 29, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions