[albertlast]

Description

The templates are going to move from SMF's own template layer to Plates. That touches almost every page, and almost none of those pages is covered by a test, so there is no way today to tell whether a converted page still renders what it used to.

This adds a snapshot suite that records what 234 pages render now, and fails when one renders differently:

• guest: board index, message index, topic display, print page, recent posts, login, registration, reminder, search, stats, help, error pages
• member: the same browsing pages with a member's controls, the posting forms, personal messages, every profile area, the member list, the calendar
• admin: the moderation centre, about 130 admin pages, the topic tools (split, merge, move, poll edit), a member's profile as an administrator
• chrome: the head, header, menu and footer, once per audience, plus the admin and moderation menus

.dev/test.sh --engine mysql --testsuite snapshot
SMF_UPDATE_SNAPSHOTS=1 .dev/test.sh --engine mysql --testsuite snapshot   # re-record

A run takes about five and a half minutes in the Docker environment on Windows, so it has a testsuite of its own and --testsuite http stays as quick as it was. Nearly all of that is the bind mount: every request stats a few hundred source files across it, at about 2 ms a file against 4 µs on the container's own disk, so any page costs a second or so. It should be much quicker on Linux, though I have not measured that.

What is compared

Not the HTML. tests/Support/DomSnapshot.php reduces each page to its elements, attributes and text, one node per line, so a diff reads like a tree:

• whitespace, comments, attribute order and class order are dropped, because a template rewritten in another engine reflows all of them
• the board URL, board directory, forum name, session, security tokens, hashes, email addresses and a search's encoded parameters become placeholders
• dates become {date}, lengths of time {duration}, ids and counts # (decimals and thousands included), and the word after a count is written the same in the singular and the plural (# reply(s))
• the text of a link to a profile, a board or a topic becomes {member}, {board} or {subject}, wherever else that text appears on the page too
• a sibling, or a group of up to four siblings, that repeats the one before it is recorded once, so a list reads the same with two rows or two hundred

What is left is the structure, the attributes and the text the templates and language files put there. The unit test for it pins both halves: the variations it has to ignore, and the changes it must still catch (a class dropped, an attribute dropped, a string changed or missing, escaping lost, an element moved, a column lost from every row).

Each snapshot also records the page's HTTP status and the errors the page logged. A converted template that renders the same page while logging an undefined index still shows up as a diff, and a page that fatals today is pinned as doing so.

Stable when the forum changes

The snapshots must not change when somebody posts, registers or adds a board. Two things make that hold:

• fixture-forum.php builds the content the pages are pointed at, once per forum, through SMF's own API: a category with boards and child boards, topics (one sticky, locked and with a poll), replies from members, an administrator and a guest, three members, a personal message and two calendar events. Every date is pinned to January 2020, there are at least two of everything a template loops over, and everything is marked read. Nothing is posted by the forum's own administrator, whose profile is not ours to rely on; a fixture administrator does it. The fixtures carry a version, and a forum holding an older one has them removed and rebuilt.
• Pages that list the whole forum are narrowed to those fixtures: the board index is compared for the fixture category only, the member list through a search for the fixture members, and the board pickers (search, move, merge, topic maintenance, membergroup access, board permissions, board management) only for the entries that mention the fixtures. Log rows, who is online, unread counts and page links are left out.

perturb-forum.php tests exactly this. It registers members with awkward names, creates categories, boards and child boards (including one under an existing board), posts topics and replies from the administrator, the new members and a guest, some sticky, some locked, sends PMs to the administrator, and gives the administrator a signature, personal text and website. It is for a forum you are going to throw away.

Configuration is not masked. The admin pages show the forum's settings, so the snapshots are recorded on, and meant to be compared against, a freshly installed forum, as CI installs. Against a dev forum that had been used for weeks, the only differences left are the three pages that show securityDisable, which that forum has switched on.

Validation

On a freshly installed forum in an isolated compose project, for each engine:

step	MySQL	PostgreSQL
record on a fresh forum	234	(uses the MySQL recording)
verify on the fresh forum	234 / 234	234 / 234
perturb (+40 members, 4 boards, 40 topics), verify	234 / 234	234 / 234
perturb again, verify	234 / 234	234 / 234

One set of snapshots serves both engines. Flood control is cleared through log_floodcontrol before each login or form post rather than waited out, as suggested in review. Earlier rounds of this failed, and each failure was fixed at the cause rather than re-recorded. They are listed because they are the reasons behind rules that could look arbitrary:

• the fixture member showing as online to whichever class ran after the member tests
• <strong>Today</strong> at 10:00 instead of a date once somebody had signed in that day
• the unread PM badge disappearing once the member tests had opened the PM
• the verification honeypot, whose field name is random per request
• "1 view" becoming "2 views" as the topics were looked at
• the calendar test leaving the calendar switched on, because IntegrationTestCase restores $modSettings in memory after each test and updateModSettings() then saw nothing to change
• after perturbing, 25 failures from board pickers, the member list, the stats top tens and page links, which is where the narrowing above comes from
• against a used dev forum: the administrator's signature on the fixture posts, the hash in minified file names (� never matched after minified_), averages like 2.5, ;boardseen, and fixture ids in the selectors
• in the recording itself: a Settings_bak.php the forum could not write, which put a warning on every settings page, and the password prompt in front of a member's account settings, recorded in place of the page
• on PostgreSQL, lists whose order the query leaves to the database (the permissions on a profile, the permission profiles, the scheduled tasks, the posts on the split topic page). These are compared sorted; two of them are bugs, below

What is deliberately not covered

• pages that fetch from simplemachines.org or a package server
• the theme editor's view of a file, which shows the template source and so changes with every conversion
• the integration hooks list, which reads every source file and times out over a Windows bind mount
• phpinfo(), and the parts of the credits and cache settings that describe the machine (web server, database version, PHP extensions)
• the error log, whose filters are built from whatever has been logged

Not in CI yet

--testsuite integration in .github/workflows/phpunit.yml excludes tests/Integration/Http, so neither the existing HTTP tests nor these run in CI. CI serves the forum from PHP's built-in server with a different set of extensions from the Docker image, and some pages (the attachment settings with the php.ini upload limits, for one) may well render differently there. Wiring that up is a follow-up worth doing on its own, once it has been run against CI's environment.

Found along the way

None of these is fixed here; each has its own pull request. The snapshots work around them and are recorded as the pages behave today.

• ?action=recent;boards=1,2 returns a 500: explode(): Argument #2 ($string) must be of type string, array given. The recent posts snapshot uses ;c= instead. Fixed in [3.0] Lets recent posts be narrowed to a list of boards again #9713.
• On PostgreSQL the member list search finds nobody when the search has a capital letter in it. Memberlist::search() wraps the columns in LOWER() but no longer folds the search term; [3.0][Testing] Case insensitive comparisons (part 2 of 2) — guard the convention in the unit suite #9597 folds it with {string_ci:search}, but was written against the code before the email search changes and needs rebasing onto it. The member list snapshot searches in lower case. Fixed in [3.0] Finds members on PostgreSQL whatever case the search is typed in #9716.
• TopicSplit has no ORDER BY at all unless the member views newest first, so on PostgreSQL the split page lists the posts in whatever order the table gives them. Fixed in [3.0] Lists the posts to split off a topic in a defined order #9714.
• PermissionProfile loads the profiles with no ORDER BY, so they come out in a different order on each engine. Fixed in [3.0] Loads the permission profiles in the order they were made again #9715.
• Installed by the same script, a fresh MySQL forum ended up with webmaster_email at the default noreply@myserver.com and a fresh PostgreSQL one at the address the installer was given. Not looked into; email addresses are masked, so it does not reach the snapshots.

Issues References (Fixes|Related|Closes)

Related: the planned conversion of the templates to Plates (no issue yet), #9713, #9714, #9715, #9716.

🤖 Generated with Claude Code

[albertlast]

@Sesquipedalian here you test env for plate migration