[albertlast]

Description

The upgrader has to cope with two things that happen to it constantly and that nothing tests: being run more than once, and being cut off half way. This adds a check for each, to the Docker development environment, alongside the existing install-forum.sh and reset.sh.

BASE=../SMF-2.1/.docker/baseline/artifacts/2.1.7-1/small/mysql.sql

.docker/rerun-upgrade.sh     --engine mysql --baseline "$BASE"
.docker/interrupt-upgrade.sh --engine mysql --baseline "$BASE"

rerun-upgrade.sh upgrades a 2.1 baseline, upgrades it again, and reports what the second run changed. It should change nothing. Running the upgrader twice is not the exotic case: it is what an admin does when a page times out, and it is what every 3.0 patch upgrade does, since VERSION_MAP keys on an upper bound and 3.0.99 selects the v3_0 migrations for any 3.0.x forum.

interrupt-upgrade.sh kills the upgrader at a chosen substep, starts it again, and reports whether the forum it ends up with is the one an uninterrupted upgrade would have built. --at N picks one substep; --points 10,50 a set; the default sweeps five points. Kill points are given as a percentage of an uninterrupted run's substeps, so the same numbers mean the same places whatever the baseline is and however many migrations the version grows.

Recovery here means starting again from the top, over a database in neither the old shape nor the new one. The step, substep and start position live in $_GET and nowhere else; maintenance_tool_progress in Settings.php is the only thing written to disk, it records the version the run started from rather than where it had reached, and it is written by preExit(), which a killed process never runs. Every migration therefore has to cope with a half-migrated database, which is a stronger requirement than merely being safe to repeat over a finished one.

--backup turns on the backup step, which the command line otherwise skips. It is worth turning on precisely because a retry is what endangers what it produces: backup_table() opens with DROP TABLE IF EXISTS, so a second pass replaces a good pre-upgrade copy with whatever the database holds by then.

The readings

Four, each written so the same database always produces the same bytes, since anything varying on its own would be reported as a change the upgrade made. Between them they separate the ways this goes wrong:

reading	catches
the DDL of every table	an ALTER that ran when it should not have, a table left half altered
the row count of each	an INSERT with no guard on it, run twice
the settings table	a setting rewritten in place, which a row count cannot see
the columns of the backup_ tables	a good backup replaced by a retry — one taken before the migrations still carries the columns 3.0 removes

Timestamps that move on their own are filtered, as are the \restrict lines pg_dump 17 brackets its output with, which carry a key regenerated on every run.

What it found

Each of these has its own pull request:

• an interrupted upgrade cannot be started again once HolidaysToEvents has dropped calendar_holidays, on both engines
• PostgreSQL loses six indexes on upgrade, all on columns widened for 2038
• PostgreSQL ends up with 80 duplicate index pairs out of 259
• normalize() compares a column property, generated_expression, that exists under neither that name in the schema classes nor in what the database reports

It also needs #9302, without which a 2.1 → 3.0 upgrade does not complete at all on the current release-3.0 and neither script has anything to measure.

Not reachable from the unit suite

These drive upgrade.php against real MySQL and PostgreSQL databases in the Docker environment. tests/bootstrap.php provides no database and no request, so none of this is expressible as a PHPUnit test; it is a pair of scripts a person or CI runs, in the same shape as .docker/ci.sh. Nothing here is part of the shipped forum — it lives in .docker/, which the check-smf-index.php and check-smf-license.php checks skip.

Issues References (Fixes|Related|Closes)

1. Related: [3.0] Lets an upgrade start again after the holidays table has gone #9626, [3.0] Rebuilds an index whose column the same pass has just changed #9623, [3.0] Recognises a normal index that is only under the wrong name #9624, [3.0] Compares the generation expression a column actually has #9625 -- everything below was found with these scripts
2. Related: [3.0] Misc fixes #9302 -- needed before either script has anything to measure
3. Related: [3.0][Testing] Check that an upgraded database matches a fresh install #9531 -- the other half of the question, whether an upgraded database matches a fresh install