Unicode 15.1 support

[stevengj]

Support for Unicode 15.1, which means updating the tables but also adding a new rule to the grapheme-break algorithm to account for the new Indic_Conjunct_Break property. Fixes #252

Currently a work-in-progress. To do:

• get the grapheme test passing

Update: should be ready now.

[PallHaraldsson]

Do you think this, i.e. the Julialang PR JuliaLang/julia#51799
should be backported to 1.10, in case it will become LTS, or at least by then?

I reviewed the code here, which is rather small (and that PR trivial), except for the Ruby generator that I think I don't need to scrutinize, and it seems safe/preferred to backport, though I did not look at utf8proc_data.c since it's quite large (and generated?).

I don't think your change is a breaking change, but I'm not sure.

The repertoire addition consists almost entirely of urgently needed CJK ideographs, synchronized with planned additions to the Chinese national standard, GB 18030. The remaining additions to the repertoire extend the set of ideographic description characters, to better enable description of unusual CJK ideographs.

Because of, at Wikipedia:

The updated standard GB18030-2022, is incompatible[how?], and it had an enforcement date of 1 August 2023.[3] It has been implemented ICU 73.2; and in Java 21,[4] and backported to older Java 8, 11, 17 (LTS releases) and 20.0.2.[5]

Also:

Major updates were made to UAX #9, Unicode Bidirectional Algorithm, UAX #31, Unicode Identifiers and Syntax, and UTS #39, Unicode Security Mechanisms, to coordinate with the publication of an important new Unicode Technical Standard: UTS #55, Unicode Source Code Handling.

* Segmentation rule changes, most notably:
a. Support was added to line breaking (UAX #14, Unicode Line Breaking Algorithm) for orthographic syllables in a number of South and Southeast Asian writing systems.
b. Grapheme cluster breaking (UAX #29, Unicode Text Segmentation) has adopted the aksara cluster behavior for six scripts. That cluster breaking behavior had previously been widely available via CLDR and ICU.
c. These changes involved significant character property updates.

What I find likely breaking about regarding GB18030-2022 and thus I think Unicode 15.1 (but not at the level of utf8proc?)::

CJK/Unihan Changes

• Seven old provisional properties have been removed.
• Six new provisional properties have been added.

[stevengj]

It's not a breaking change, I think (mainly just adding new characters, and tweaking some grapheme-break rules), but it's a new feature and thus probably not eligible for backport.