Skip to content

test(migration): guard against reserved column names on MariaDB and MySQL - #9991

Merged
rubenvdlinde merged 10 commits into
mainfrom
test/4909-reserved-column-names
Nov 12, 2026
Merged

rubenvdlinde merged 10 commits into
mainfrom
test/4909-reserved-column-names

Conversation

@WilcoLouwerse

@WilcoLouwerse WilcoLouwerse commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Forward-port of the two tests that #5957 added to the release line, so main is guarded too. The fix itself is already on main via #5879; these tests were only on beta / stable2.1.

  • tests/Unit/Migration/ReservedColumnNamesTest.php checks every column the migrations define through the schema API against the words each supported MariaDB and MySQL series reserves. Doctrine quotes a column name only when its own keyword list for the platform has the word, and its MariaDB list misses row_number, which broke MariaDB installs (Migration Version1Date20260915050000 fails on MariaDB: column name row_number is a reserved word #4909). CI does not run the migrations on MariaDB, so this test is the guard. A short allowance lists the reserved words already in use that Doctrine does quote; one test keeps that allowance from going stale and another checks that Doctrine really quotes each of them. Columns created by raw SQL are outside the check; the class docblock names the two migrations that do so.
  • The lists are read from the upstream server sources by tests/Support/ReservedWordSource.php: the keywords in sql/lex.h that the grammar's identifier rule in sql/sql_yacc.yy does not allow as an identifier. The parser fails loudly when that rule is missing or the result is implausible. Released series: MariaDB 10.5, 10.6, 10.11, 11.3, 11.4, 11.8 and MySQL 8.0, 8.4. Development branches (MariaDB main, MySQL trunk) are reported as an incomplete test, not a failure, so a word a coming series reserves is visible without turning unrelated PRs red.
  • A released series' committed snapshot in tests/Unit/Migration/fixtures/ is the union over its branch head and every release tag (159 tags in total), because a branch head is only the newest point release: MariaDB 10.11.x, 11.3.x, 11.4.x and 11.8.x reserve ROW_NUMBER (from 10.7 on), while the 10.11 and 11.4 heads no longer reserve it. At run time the test checks the live branch head merged with the snapshot; when a download fails (connection error, 429, 5xx) the snapshot alone is used, so the test never needs network. A 404 or redirect fails the test on purpose: the source moved. One line on STDERR says where each list came from. php tests/Unit/Migration/fixtures/update-reserved-words.php regenerates the snapshots (optional GITHUB_TOKEN).
  • tests/Unit/Db/ImportPreviewRowMapperMigratedTest.php runs the mapper's queries as SQL against the migrated table: the source row lives in source_row, not row_number, and findByPreview / findPendingWrites return rows in source order.

Only change against the beta version of the mapper test: findByPreview is called with ListQuery::unbounded(), the signature on main.

Checks

  • Ran: vendor/bin/phpunit -c phpunit-unit-local.xml tests/Unit/Migration/ReservedColumnNamesTest.php on the host (PHP 8.4, no Nextcloud server), exit 0, OK (5 tests, 93 assertions, none skipped or incomplete), once online and once with RESERVED_WORDS_BASE_URL=https://invalid.invalid (every list from its snapshot). Also exit 0 with -c phpunit.xml.
  • Ran: the parser over all 159 release tags of the eight released series; each snapshot equals the union over its tags.
  • Ran: mutation check, one migration column temporarily renamed to row_number: the test fails (exit 1) and names the column with mariadb 10.11, mariadb 11.3, mariadb 11.4, mariadb 11.8, mysql 8.0, mysql 8.4; reverted.
  • Ran: the parser against real grammar files with the identifier rule renamed: it throws, naming the rule.
  • Ran: php -l on the changed PHP files, no syntax errors.
  • Not run: ImportPreviewRowMapperMigratedTest (needs Nextcloud's Doctrine classes and errors on the host), the full unit suite, Psalm, and a real MariaDB 10.6 server; CI runs the first three.
  • phpcs.xml covers lib/ only, so these files are outside it, as on beta.

Test plan

Refs #4909

🤖 Generated with Claude Code

…ySQL

Forward-port of the two tests that #5957 added on the release line.

- ReservedColumnNamesTest checks every column the migrations define
  against the reserved words measured on MariaDB 11.3 and MySQL 8.0
  (fixtures/). Doctrine quotes a column name only when its own keyword
  list has the word, and its MariaDB list misses row_number (#4909).
- ImportPreviewRowMapperMigratedTest runs the mapper's queries against
  the migrated table: the source row lives in source_row, and both
  readers return rows in source order.

The mapper test calls findByPreview with ListQuery::unbounded(), the
signature on main.

Refs #4909

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WilcoLouwerse and others added 9 commits November 12, 2026 17:48
…am sources

ReservedColumnNamesTest used two hand-committed lists, one for MariaDB
11.3 and one for MySQL 8.0, that had to be edited whenever a version
moved. It now derives the list per server series from the upstream
source: the keywords in the lexer's symbol table (sql/lex.h) that the
grammar's identifier rule (MariaDB keyword_ident, MySQL ident_keyword in
sql/sql_yacc.yy) does not allow as an identifier. The series are branches
(MariaDB 10.5, 10.6, 10.11, 11.3, 11.4, 11.8 and main; MySQL 8.0, 8.4 and
trunk), so new point releases and the words a coming series reserves are
picked up without an edit. 11.3 stays because MariaDB 11.0 to 11.3
reserve ROW_NUMBER, the word behind #4909, and 11.4 does not.

The parser lives in tests/Support/ReservedWordSource.php. For the
versions measured earlier on live servers (MariaDB 11.3, MySQL 8.0) it
returns exactly the words that were measured.

The committed snapshots stay as the offline fallback, one per series,
and are regenerated by fixtures/update-reserved-words.php instead of
being edited by hand. Per series the test downloads with a 5 second
connect timeout, falls back to the snapshot when the network does not
answer (one failure ends the downloads for the rest of the run), and
checks the union of both. A 404 or a layout the parser no longer reads
fails the test instead of falling back, so a renamed branch or a changed
upstream file cannot leave it checking stale lists unnoticed.

Refs #4909

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e is not found

The parser followed the identifier rule (keyword_ident / ident_keyword) and,
when upstream renamed it, found nothing reachable and counted every lexer
keyword as reserved (688 words for MariaDB 11.3, `status` and `date`
included). That passed the 200-word lower bound and surfaced as an unrelated
column failure in a migration.

parse() now throws a RuntimeException naming the engine, the series and the
rule when the start rule is missing, or when a rule it reaches has no
definition. The plausibility check on a list gained an upper bound and a
sentinel (`status` and `date` must not be reserved), and it also covers the
committed snapshots.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…iling on them

MariaDB `main` and MySQL `trunk` were merged into the same lookup as the
released series, so the day either branch reserved a word an existing column
uses, every open PR would fail without a change of its own.

SOURCES now marks which series are development branches. Released series
keep failing testNoMigrationColumnIsAReservedWord; a word only a development
branch reserves is reported by a separate test through markTestIncomplete(),
which phpunit.xml and the CI workflow do not turn into a failure. Both
messages name, per column, the engine and series that reserve the word. The
schema walk is shared by the three tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… download fallback

ReservedWordSource::words() reads the snapshot on every run and merges it with
the online list, so a word upstream stops reserving stays flagged until the
snapshot is regenerated. The class docblocks, the generator's docblock and the
header it writes said the snapshot is read only when the download fails.

The generator's header text is corrected and the ten snapshots regenerated;
their word lists are unchanged, only the header lines differ.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…wance exempts

ALREADY_IN_USE exempts a word on every engine and series because Doctrine
quotes it wherever it is reserved. Nothing asserted that: if an engine later
reserved e.g. `groups` on MariaDB, where Doctrine's list lacks it, the
allowance would hide exactly the #4909 failure.

A new test checks, per engine, that each allowed word an engine reserves is in
every Doctrine keyword list that engine's platforms can use (MariaDBKeywords,
MariaDb102Keywords, MariaDb117Keywords; MySQL80Keywords, MySQL84Keywords: the
classes of doctrine/dbal 3.10, which Nextcloud 32 to 35 all ship). It skips
with a message naming the classes when they cannot be loaded. The docblock
now says why one global list is acceptable: the premise is asserted per
engine.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…side the check

The test reads the schema built from the migrations' changeSchema() only, so
a column added with raw SQL or in preSchemaChange/postSchemaChange never
appears in it. The class docblock now says so and names the two migrations
that do this today (Version1Date20260322120000 and Version1Date20260706100000)
and why a source scan was not added: their column names are interpolated or
spread over concatenated strings, so a scan would miss or misread them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e each list came from

download() treated every non-200 as "offline" and latched it for the run: a
301 (redirects are not followed), a 429 or a 5xx ended the downloads silently,
and the origin of each list was only printed when the assertion failed.

Now 200 is used; a 404 or any 3xx throws with the URL, because the source
moved and no snapshot repairs that; 429, 5xx, other statuses and connection
errors fall back to the snapshot and keep the reason. The test writes one line
on STDERR after the run naming which series came from upstream and which from
the snapshot, with the download failure, so a snapshot-only run is visible
while the suite stays green; the failure message carries the same line. The
refresh script prints the reason too. curl_close() is gone (deprecated in
PHP 8.5, a no-op since 8.0).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…bare clone

ReservedColumnNamesTest runs under phpunit-unit-local.xml on a host without a
Nextcloud server; only the import-preview mapper test still needs the server's
classes, and its own docblock says so. The paragraph claiming otherwise is
removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s, not only the branch head

A series branch is only its newest point release, and upstream also
un-reserves words: the MariaDB 10.11 and 11.4 branch heads no longer list
ROW_NUMBER while every released 10.11.x and 11.4.x (and 11.8.x) does. The
snapshots held head-only words, so the word that broke the import preview
migration (#4909) was caught on MariaDB only through the 11.3 branch.

The refresh script now builds the snapshot of a released series as the union
over the branch head and every release tag of that series (tags listed through
the GitHub API, `<engine>-<series>.<n>` only; MySQL 8.0 starts at 8.0.11,
because 8.0.0 to 8.0.4 are development milestones with another lex.h layout).
A development branch has no tags: head only. It goes one request at a time,
takes an optional GITHUB_TOKEN for the rate limit, caches finished tags so an
interrupted run resumes, and exits 1 naming the cause. Each header states the
tag range, the tag count and the date.

The test still reads the live head and merges the snapshot, which is now also
the only source for the older point releases; the docblocks say so. A new
assertion pins the defect class: ROW_NUMBER must be in the MariaDB 10.11,
11.3, 11.4 and 11.8 lists and the MySQL 8.0 and 8.4 lists.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Comment thread tests/Support/ReservedWordSource.php Resolved
Comment thread tests/Support/ReservedWordSource.php Resolved
Comment thread tests/Support/ReservedWordSource.php Resolved

@WilcoLouwerse WilcoLouwerse left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (Standard · full)

Self-review by the author, posted as a comment because GitHub does not accept an approval on one's own PR; a colleague's review is still needed. Tests only, nothing under lib/. The review found one blocker and three concerns, plus four minor points; all eight were fixed on the branch before this verdict (commits 3e4991da95..e72c6065b8), and a separate check of the fix commits found no defect.

Connected issues

Issue State Effect of this PR
#4909 closed (fixed by #5879 / #5957) unaffected; this PR adds the guard on main

🔴 Blocker (fixed)

🟡 Concerns (fixed)

🟢 Minor

  • f4 — the allowance's premise (Doctrine quotes these words) was untested. Fixed in 7f7ab230de.
  • f5 — columns created by raw SQL are not seen by the test. Stated in the class docblock, with the two migrations, in 166af53cae; no scan added.
  • f6 — any status other than 200 or 404 fell back silently. Fixed in 93d0afc7fa: 404 and redirects fail, outages fall back, one STDERR line names each list's origin.
  • f7 — the docblock said the test cannot run in a bare clone. Fixed in 645a5953e2.
  • Open, from the check of the fix commits, none blocking: the refresh script's tag cache in the system temp directory is created world-writable and written non-atomically (ReservedWordSource::releaseWords()); when one series fails, the other snapshots are still rewritten (exit 1); two comments say the branch heads "no longer list" ROW_NUMBER where "no longer reserve" is meant; the snapshot headers do not say that pre-release tags (for example 11.4.0) are included.

Checks on the fixed head e72c606: ReservedColumnNamesTest on the host: exit 0, 5 tests, 93 assertions, online and with the download host unreachable, also with -c phpunit.xml; every snapshot equals the union over its release tags (159 tags, recomputed independently); mutation (a column renamed to row_number) fails the test. Not run locally: ImportPreviewRowMapperMigratedTest (needs a Nextcloud server), the full suite.
Gates: the CI checks on this head, the gate check included, had not finished when this was posted; no gate findings this round.

@rubenvdlinde
rubenvdlinde merged commit c457f76 into main Nov 12, 2026
11 of 17 checks passed
@WilcoLouwerse

Copy link
Copy Markdown
Contributor Author

/backport to stable2.1

WilcoLouwerse added a commit that referenced this pull request Nov 13, 2026
…on stable2.1

On this line MigratedSqliteDatabase has no fallback for a bare clone, so the
sentence #9991 dropped still holds here.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
rjzondervan added a commit that referenced this pull request Nov 13, 2026
…erved-words

test(migration): read reserved words per database version from upstream sources (stable2.1 backport of #9991)
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ cc0d124

Check PHP Vue Security License Tests
lint ✅
phpcs ❌
phpmd ❌
psalm ❌
phpstan ❌
phpmetrics ✅
eslint ❌
stylelint ✅
build ✅
check-specs ✅
test-l10n ❌
test-l10n-parity ❌
format ❌
check-schema-l10n ✅
test-newman-route-ratchet ✅
composer ✅ ✅ 169/169
npm ✅ ✅ 639/639
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌ 56767/56972 · 27 skipped (counts from 1 of 8 legs)
Newman ❌
Playwright ❌ 252/338 · 5 skipped · 150 did not run
Hydra gates ❌

Quality workflow — 2026-11-14 22:55 UTC

Download the full PDF report from the workflow artifacts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants