[sbulen]

Spun off from #9271

Mid-upgrade, $db_character_set was getting erroneously set to 'utf8' in step 2 under some circumstances. So, if there were non-utf8 characters, updates to that data could mangle or delete the data. This is mostly visible during the json conversion, where lots of latin1 characters may be encountered. Both MySQL & MariaDB are affected.

In 2.0, $db_character_set was not set by the SMF installer. So it's blank for a lot of forums. The 2.1 upgrader, when the 'Migrate to new Settings.php' box is checked, detects the blank $db_character_set and defaults it to 'utf8'. This causes the upgrader to issue a SET NAMES for 'utf8'. Database updates prior to the utf8 conversion could cause issues.

So, this PR detects what $db_character_set should be, then sets it in Settings.php. That way the Settings.php migration leaves it alone. It is then properly set to 'utf8' in a later step, after the utf8 conversion completes.

If approved, I'll work on a 3.0 version.

Note there is still one odd wrinkle related to charsets/collations I haven't figured out... During the utf8 conversion, MariaDB is setting all the tables to utf8mb3_uca1400_ai_ci instead of utf8mb3_general_ci, the SMF 2.1 default. They are not synonymous. Haven't figured out why yet.

[sbulen]

Some screenshots....

This is a screenshot of the serialized strings in log_actions before the upgrade:

Log actions after the upgrade... Items with non-utf8 chars were lost:

With this PR, the characters are maintained:

[Sesquipedalian]

Very nice!

[Sesquipedalian]

Note there is still one odd wrinkle related to charsets/collations I haven't figured out... During the utf8 conversion, MariaDB is setting all the tables to utf8mb3_uca1400_ai_ci instead of utf8mb3_general_ci, the SMF 2.1 default. They are not synonymous. Haven't figured out why yet.

I wouldn't worry too much about that. The difference between utf8mb3_uca1400_ai_ci and utf8mb3_general_ci lies only in how they collate (i.e. sort) strings:

• The utf8mb3_general_ci collation uses simple sorting rules and ignores case when sorting strings. For example, it will not recognize that ß is the same as ss in German and therefore will not sort them correctly when the language is set to German.
• The utf8mb3_uca1400_ai_ci collation uses the Unicode 14 sorting rules and ignores both case and accents when sorting strings. It will correctly sort strings containing ß and ss when the language is set to German.

Both collations will read and write the data as 3-byte UTF-8 strings, which is what matters for present purposes.

[sbulen]

The difference between utf8mb3_uca1400_ai_ci and utf8mb3_general_ci lies only in how they collate (i.e. sort) strings

Yep, understood. The inconsistency bugs me though. Especially because MariaDB installs on one but upgrades to the other. It's not even consistent within MariaDB.

I'll research a bit more & write up a new issue for that (if only so I can move on...).

I believe MariaDB is at times falling back to defaults. E.g., uca1400_ai_ci is its default collation. Also I read somewhere that 'utf8' doesn't default to 'utf8mb3' (needs to be verified)... And we have a mix of 'utf8' and 'utf8mb3' throughout our various sources. Since utf8 => utf8mb3 in MySQL, that makes no difference. In MariaDB, though...

Possibly low priority, the collation by itself doesn't bug me too much, but I'm afraid that falling back to defaults may bite us in other ways here.

[Sesquipedalian]

Well, if you want to spend time looking into why MariaDB is switching to its default collation in certain circumstances, you are obviously free to do so. But I can tell you that we will be unlikely to accept a PR that tries to fight MariaDB about mere collations. As long as we are sure that it maintains the correct character set (i.e. the utf8mb3 part rather than the _general_ai_ci part), changes to the collation are not worth the effort of testing against the upgrader.

[sbulen]

Note this PR was also spawned from #9259