Skip to content

Fix serialization of datetimes with a fixed UTC offset - #74922

Merged
shahar1 merged 2 commits into
apache:mainfrom
justinpakzad:fix/datetime-serialization-fixed-offset
Oct 24, 2026
Merged

shahar1 merged 2 commits into
apache:mainfrom
justinpakzad:fix/datetime-serialization-fixed-offset

Conversation

@justinpakzad

@justinpakzad justinpakzad commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

The serde timezone serializer can't handle timezones that are just a fixed offset with no name. The datetime serializer saves an empty place holder and reading the value back (e.g., via Xcom pull) fails. Take the following Dag:

from datetime import datetime
from airflow.sdk import dag, task

@dag
def serde_offset_tz():
    @task
    def push_offset():
        return datetime.fromisoformat("2024-01-01T00:00:00+02:00")
    @task
    def do_something(value):
        print(repr(value), value.utcoffset())

    do_something(push_offset())
serde_offset_tz()

On main, do_something would fail with: ValueError: ZoneInfo keys must be normalized relative paths, got:

This PR adds support for timezones with a fixed, nonzero UTC offset and no name, by encoding them as a pendulum FixedTimezone. Additional tests were added to cover the new behavior. The NoNameTZ helper now returns None from utcoffset so that test_timezone_serialize_no_name still covers the fallback path (no name or fixed offset).

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Sonnet 5.5)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@justinpakzad
justinpakzad force-pushed the fix/datetime-serialization-fixed-offset branch from 2a28470 to b22fb93 Compare October 18, 2026 00:39
@justinpakzad
justinpakzad marked this pull request as ready for review October 18, 2026 00:40

@fat-catTW fat-catTW left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Look good overall!
Non-blocking: could we also assert that the fixed offset itself survives the datetime round-trip?

test_datetime currently compares only timestamp(), so a value restored as 07:00+00:00 would still pass for an original 09:00+02:00.
An assertion such as the following would make the intended guarantee explicit:

assert deserialized_obj.utcoffset() == datetime.timedelta(hours=2)

@justinpakzad
justinpakzad force-pushed the fix/datetime-serialization-fixed-offset branch from b22fb93 to 5511a30 Compare October 19, 2026 22:27
@justinpakzad

justinpakzad commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the PR. Look good overall! Non-blocking: could we also assert that the fixed offset itself survives the datetime round-trip?

test_datetime currently compares only timestamp(), so a value restored as 07:00+00:00 would still pass for an original 09:00+02:00. An assertion such as the following would make the intended guarantee explicit:

Yep! Good call. The else branch actually never even ran here because they all matched the first isinstance check, so the .timestamp() comparison never happened. I've updated the test to check utcoffset() for datetimes and == for everything else.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnamed fixed-offset tzinfo (e.g. datetime.timezone(timedelta(hours=2))) now round-trips through serde as a pendulum FixedTimezone, matching how the existing UTC special case already worked, and the new parametrized case plus test_timezone_serialize_fixed_offset both fail on main without the fix. The extra utcoffset() assertion in test_datetime is a good catch too, since == on aware datetimes alone would not notice a lost offset.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Airflow handles maintainer review:
Contributing guide.

@shahar1
shahar1 merged commit ad3ecb8 into apache:main Oct 24, 2026
95 checks passed
@vatsrahul1001 vatsrahul1001 added the backport-to-v3-4-test Backport to v3-4-test label Oct 26, 2026
@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.4.0 milestone Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

vatsrahul1001 pushed a commit that referenced this pull request Oct 26, 2026
…74922) (#75647)

(cherry picked from commit ad3ecb8)

Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants