Skip to content

lib/python/pygrass: fix Python 3.14 pickling error in grid tests#7257

Open
SyedAhad01 wants to merge 24 commits intoOSGeo:mainfrom
SyedAhad01:fix-python314-pickling
Open

lib/python/pygrass: fix Python 3.14 pickling error in grid tests#7257
SyedAhad01 wants to merge 24 commits intoOSGeo:mainfrom
SyedAhad01:fix-python314-pickling

Conversation

@SyedAhad01
Copy link
Copy Markdown
Contributor

Fixes PicklingError failures when running tests with Python 3.14.

Python 3.14 cannot pickle local functions defined inside other functions
when using multiprocessing. This fix moves all run_grid_module inner
functions to module-level helper functions and uses functools.partial
to pass parameters.

Related to #6104

@github-actions github-actions bot added Python Related code is in Python libraries tests Related to Test Suite labels Apr 1, 2026
@SyedAhad01 SyedAhad01 changed the title pygrass: fix Python 3.14 pickling error in grid tests pygrass: Fix Python 3.14 pickling error in grid tests Apr 1, 2026
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
SyedAhad01 and others added 4 commits April 2, 2026 04:00
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@echoix
Copy link
Copy Markdown
Member

echoix commented Apr 1, 2026

Read our contribution guide, you'll see that most of the back and forth of precious shared CI time can be avoided by using precommit (or prek) to have most simple formatting checks already made.

And I assume you're using an AI or agent or something like that, and I saw you were in VS Code in another PR, so you can reread and stage each line to make sure you're only committing what is relevant to the PR. Touching unrelated lines only makes reviewing harder: the last PR together, it started with many many changes, whilst it ended up being less than 10 lines, two parts. Way easier when concise like that.

@SyedAhad01
Copy link
Copy Markdown
Contributor Author

Thank you for the feedback and the guidance. I will read the contribution guide and set up pre-commit before pushing further changes. I will also make sure to carefully review each line and only commit what is directly relevant to the fix

@echoix
Copy link
Copy Markdown
Member

echoix commented Apr 1, 2026

Don't worry, your general approach is fine. There's already some examples in the repo of the pattern needed, that I've fixed as a proof of concept a while ago. And it looked a bit like that.

@SyedAhad01 SyedAhad01 changed the title pygrass: Fix Python 3.14 pickling error in grid tests pygrass.modules: fix Python 3.14 pickling error in grid tests Apr 2, 2026
@SyedAhad01 SyedAhad01 changed the title pygrass.modules: fix Python 3.14 pickling error in grid tests lib/python/pygrass: fix Python 3.14 pickling error in grid tests Apr 2, 2026
@echoix
Copy link
Copy Markdown
Member

echoix commented Apr 4, 2026

You don’t have to keep your branch (and draft) updated with main, after each of our commits. We can’t have all hundreds of PRs redo all CI after each commit, it would be exponential

@echoix
Copy link
Copy Markdown
Member

echoix commented Apr 4, 2026

Unless you’re pushing new changes and it will run again, then ya, it’s a good time to make it run on updated code and linters

@SyedAhad01
Copy link
Copy Markdown
Contributor Author

Thank you for clarifying. I understand now I should only update the branch when I am pushing my own new changes, not after every commit to main. I will keep that in mind for future PRs. Sorry for the unnecessary CI runs.

@SyedAhad01 SyedAhad01 marked this pull request as ready for review April 4, 2026 15:41
@SyedAhad01
Copy link
Copy Markdown
Contributor Author

Hi @echoix just a gentle heads-up that the PR is ready for review whenever you have a moment. All 26 checks are passing. I appreciate your time and continued guidance throughout this process.

Copy link
Copy Markdown
Member

@echoix echoix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it is working as you'd think yet. If you look at pytest test in macOS or Windows (Linux should work fine), let's take Windows, the tests of pygrass grid tests should not end up with expected failure anymore. At one point in your development, assuming everything went right, the tests should've been fixed and passing, and they would fail because of an unexpected success. That would have been because the custom attribute @xfail_mp_spawn that wasn't removed.

I see that that attribute's error has changed, but the tests that should've been fixed doesn't have that attribute removed. So the tests still fail, but we ignore it if the default start method is spawn, like Windows.

So, there's still a bit of work to do.

Copy link
Copy Markdown
Member

@echoix echoix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help you out, I identified the lines where the attributes should be removed, plus other second reading comments

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Comment on lines +185 to +187
),
check=multiprocessing.get_start_method() == "spawn"
or surface != "non_exist_surface",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That logic here is unexpected to me, could you explain a bit?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this condition to handle the case where the surface does not exist. On Windows/macOS (spawn), the subprocess always fails, so we let the error propagate. On Linux (fork), we skip the error so the test can still check if mapsets were cleaned. Let me know sir if there is a simpler way to do this

SyedAhad01 and others added 4 commits April 5, 2026 11:56
Co-authored-by: Edouard Choinière <27212526+echoix@users.noreply.github.com>
Co-authored-by: Edouard Choinière <27212526+echoix@users.noreply.github.com>
Co-authored-by: Edouard Choinière <27212526+echoix@users.noreply.github.com>
@SyedAhad01
Copy link
Copy Markdown
Contributor Author

After removing xfail_mp_spawn, all tests now fail with RuntimeError: Subprocess failed with exit code 1 on both macOS and Windows. It seems the issue is inside GridModule itself where the internal worker functions are not picklable under spawn. Could you point me to the example in the repo that shows the correct pattern for fixing this? I want to make sure I apply the right fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

libraries Python Related code is in Python tests Related to Test Suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants