Skip to content

FIX: Keep combining marks inside words masked by SATAMaskingConverter - #4390

Merged
jbolor21 merged 1 commit into
microsoft:mainfrom
saquibjawedbit:fix/sata-mask-combining-marks
Nov 19, 2026
Merged

jbolor21 merged 1 commit into
microsoft:mainfrom
saquibjawedbit:fix/sata-mask-combining-marks

Conversation

@saquibjawedbit

Copy link
Copy Markdown
Contributor

Description

Fixes #4389. SATAMaskingConverter keeps the punctuation around a masked word and replaces the word itself, but its trailing-punctuation pattern counted combining marks (Unicode category M*: decomposed accents, Indic vowel signs) as punctuation, because they are not \w. So a word ending in one leaked that mark after the mask: "café" (decomposed) became "[MASK]́" and "नमस्ते" became "[MASK]े".

  • Combining marks at the start of the computed suffix are now moved back into the word core, so they are masked with it; real trailing punctuation ("café!" → "[MASK]!") is still kept.
  • Added a parametrized test with 3 cases (decomposed Latin accent, Devanagari vowel sign, accent followed by punctuation). It fails on main and passes with the fix; tests/unit/converter passes (3175 passed, 52 skipped), and pre-commit passes on the changed files.

Trailing accents and vowel signs are not \w, so they were split off as punctuation and left after the mask, leaking part of the hidden word ("café" -> "[MASK]́", "नमस्ते" -> "[MASK]े").
Copilot AI balanced review requested due to automatic review settings November 18, 2026 19:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@jbolor21 jbolor21 self-assigned this Nov 19, 2026

@jbolor21 jbolor21 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Running through our pipelines and we will merge in thanks!

@jbolor21
jbolor21 added this pull request to the merge queue Nov 19, 2026
Merged via the queue into microsoft:main with commit fd00f9d Nov 19, 2026
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: SATAMaskingConverter leaves a word's trailing accent or vowel sign after the mask ("café" -> "[MASK]́", "नमस्ते" -> "[MASK]े")

3 participants