Skip to content

Skip zero-sized parameters in HP fragment mapping - #8298

Merged
sfc-gh-truwase merged 3 commits into
deepspeedai:masterfrom
fwerkor:fix-zero-size-fragment-map
Aug 23, 2026
Merged

Skip zero-sized parameters in HP fragment mapping#8298
sfc-gh-truwase merged 3 commits into
deepspeedai:masterfrom
fwerkor:fix-zero-size-fragment-map

Conversation

@fwerkor

@fwerkor fwerkor commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • exclude zero-sized parameters from ZeRO-1/2 HP-fragment overlap mapping
  • avoid constructing an empty fragment when an empty parameter's offset lies inside a partition

Fixes #8297.

Testing

  • current master (edaa722), 2x A6000: minimized ZeRO-1 reproducer fails before the change with fragment start 8 should be < fragment_end 8 and completes after the change
  • 2-process CPU/Gloo: minimized reproducer completes with both ZeRO-1 and ZeRO-2 after the change
  • the trailing zero-sized-parameter reproducer from [BUG] ZeRO-1/2 fail on trailing zero-sized trainable parameters #8279 still completes
  • python3 -m py_compile deepspeed/utils/mixed_precision_linkage.py

@fwerkor
fwerkor force-pushed the fix-zero-size-fragment-map branch from 1143241 to fb759f4 Compare August 23, 2026 05:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1143241a49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deepspeed/utils/mixed_precision_linkage.py Outdated
Comment thread deepspeed/utils/mixed_precision_linkage.py Outdated
Signed-off-by: Cao Yuhang <caoyuhang@fwerkor.com>
@fwerkor
fwerkor force-pushed the fix-zero-size-fragment-map branch from fb759f4 to 5f1ab9b Compare August 23, 2026 06:06
@fwerkor

fwerkor commented Aug 23, 2026

Copy link
Copy Markdown
Contributor Author

@codex

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 5f1ab9b6d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@sfc-gh-truwase
sfc-gh-truwase added this pull request to the merge queue Aug 23, 2026
Merged via the queue into deepspeedai:master with commit 573ce48 Aug 23, 2026
13 checks passed
@fwerkor
fwerkor deleted the fix-zero-size-fragment-map branch August 24, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ZeRO-1/2 fail mapping zero-sized parameter inside a partition

2 participants