Skip to content

fix(db): normalise date/date-time properties in bulk writes - #4773

Merged
rubenvdlinde merged 7 commits into
mainfrom
fix/magicbulkhandler-date-normalization
Nov 12, 2026
Merged

rubenvdlinde merged 7 commits into
mainfrom
fix/magicbulkhandler-date-normalization

Conversation

@bbrands02

Copy link
Copy Markdown
Contributor

Summary

  • MagicBulkHandler wrote a schema's date/date-time property straight through, without the schema-aware normalization MagicMapper::prepareObjectDataForTable() already applies on the non-bulk path.
  • An ISO-8601-with-offset string (DateTime::ATOM, the convention every ObjectService::saveObjects() caller uses) reached MySQL's DATETIME column unconverted and failed with SQLSTATE[22007].
  • date and date-time are handled distinctly: a date-time is an instant, so its offset is applied before storing; a date names a calendar day and is never shifted through a timezone.

Test plan

  • New MagicBulkHandlerDateNormalizationTest (6 tests)

🤖 Generated with Claude Code

MagicBulkHandler wrote a schema's date-time property straight through
without the schema-aware normalization MagicMapper::prepareObjectDataForTable()
already applies on the non-bulk path. An ISO-8601-with-offset string (the
DateTime::ATOM convention ObjectService::saveObjects() callers use) reached
MySQL's DATETIME column unconverted and failed outright (SQLSTATE[22007]).
date and date-time are handled distinctly: a date-time is an instant and
gets its offset applied before storing; a date names a calendar day and is
never shifted through a timezone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ 78736d7

Check PHP Vue Security License Tests
lint ✅
phpcs ❌
phpmd ❌
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-specs ✅
test-l10n ✅
test-l10n-parity ✅
format ✅
check-schema-l10n ✅
composer ✅ ✅ 173/173
npm ✅ ✅ 648/648
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌
Newman ❌
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ❌

Quality workflow — 2026-10-22 10:11 UTC

Download the full PDF report from the workflow artifacts.

…malizer

Review feedback on #4773:

1. [Investigated, not a bug] format: date and format: date-time map to the
   SAME `datetime` column (MagicMapper::mapStringProperty()), and
   DateTimeNormalizer::formatDateForDatabase()'s own docblock documents the
   "Y-m-d H:i:s"-shaped representation it deliberately keeps for a calendar
   day. The bulk path already matched the non-bulk path's established,
   intentional behaviour -- confirmed by comparing against the identical
   inline logic in MagicMapper::prepareObjectDataForTable(). Pinned with an
   explicit comment and a new direct-parity test rather than changed.

2. [Fixed] MagicBulkHandler hand-rolled DateTimeInterface handling
   (DateTimeImmutable::createFromInterface + manual timezone conversion)
   alongside delegating strings to DateTimeNormalizer -- duplicating a
   contract DateTimeNormalizer::normalize() already implements for
   DateTimeInterface input. Both branches now call
   formatForDatabase()/formatDateForDatabase() directly; removed the
   now-unused DateTimeImmutable/DateTimeZone imports. Verified byte-for-byte
   identical output for every existing test case, including the
   DateTimeInterface one, before and after.

3. [Addressed] Added a real integration test
   (BulkAndHandlersIntegrationTest::
   testBulkDateTimeWriteMatchesNonBulkWriteAndPersistsWithoutSqlError)
   against an actual database: a bulk write and a non-bulk write
   (MagicMapper::insert()) of the identical ISO-8601-with-offset date/
   date-time values must both persist without a SQL error and read back
   identically. Verified passing against a real Nextcloud install (not
   runnable in this sandbox's bare source tree, same pre-existing limitation
   as every other test in that file).

Also added: a direct-parity unit test calling DateTimeNormalizer's own
methods and comparing against the bulk handler's output, and a test pinning
that a non-string/non-DateTimeInterface value on a date property is still
left untouched after the simplification.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@bbrands02

Copy link
Copy Markdown
Contributor Author

Investigated both, fixed what was real. Pushed e53ccd4.

1. [Medium] Calendar dates with a time component — investigated, not a bug.

  • Column type: MagicMapper::mapStringProperty() maps BOTH format: date and format: date-time to the same 'type' => 'datetime' column. There is no separate DATE-only column.
  • DateTimeNormalizer::formatDateForDatabase()'s own docblock documents this explicitly: "@return string|null Y-m-d H:i:s-formatted string preserving the given calendar day" — the time component is deliberate, not a leftover.
  • Parity confirmed by direct comparison: MagicMapper::prepareObjectDataForTable() (non-bulk) runs the exact same in_array($propertyFormat, ['date-time','date']) branch, calling the same normalizer methods. The bulk path already matched.
  • No change made here, other than pinning it: added an explicit comment at the call site, a new unit test asserting byte-for-byte parity against calling DateTimeNormalizer directly, and a real-DB integration test (see [Changelog CI] Add Changelog for Version 0.1.4 #3).

2. [Medium] DateTimeInterface bypassing DateTimeNormalizer — confirmed and fixed.

  • DateTimeNormalizer::normalize() already accepts DateTimeInterface input directly (never re-zoning it) — both formatForDatabase() and formatDateForDatabase() already produce exactly what the manual DateTimeImmutable::createFromInterface() + conditional setTimezone() branch was reproducing by hand.
  • Simplified MagicBulkHandler.php to route both string AND DateTimeInterface input through the same two normalizer calls; removed the now-dead DateTimeImmutable/DateTimeZone imports. Non-string/non-DateTimeInterface values (a schema violation) are still left untouched, unchanged from before.
  • Verified identical output for every existing test, including the DateTimeInterface-specific one, before and after the change.
  • Did not touch the equivalent duplicated logic in MagicMapper::prepareObjectDataForTable() (non-bulk, pre-existing, unrelated to this PR) — flagging that the same simplification would apply there too, as a separate follow-up, per "avoid unnecessary refactoring."

3. [Low] Missing integration test — added.
BulkAndHandlersIntegrationTest::testBulkDateTimeWriteMatchesNonBulkWriteAndPersistsWithoutSqlError: creates schema properties with date/date-time format, bulk-writes AND non-bulk-writes (MagicMapper::insert()) the identical ISO-8601-with-offset values against a real database, and asserts both read back identically. Couldn't run it in this sandbox (bare source tree, no NC install — same pre-existing limitation as all 87 other tests in that file), so I ran it against a real local Nextcloud instance instead: passes, 87/87 in the file green.

4. Verified behavior — all five points confirmed unchanged, covered by the existing + new tests.

Full tests/Unit/Db/ + tests/Unit/Service/ suite still green locally (19859 tests, same 8 pre-existing unrelated failures); BulkAndHandlersIntegrationTest green against a real NC install (87 tests).

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ 4d35dc6

Check PHP Vue Security License Tests
lint ✅
phpcs ❌
phpmd ❌
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-specs ✅
test-l10n ✅
test-l10n-parity ✅
format ✅
check-schema-l10n ✅
composer ✅ ✅ 173/173
npm ✅ ✅ 648/648
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌
Newman ❌
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ❌

Quality workflow — 2026-10-22 11:46 UTC

Download the full PDF report from the workflow artifacts.

Comment thread lib/Db/MagicMapper/MagicBulkHandler.php Resolved

@WilcoLouwerse WilcoLouwerse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES (Standard · full)

The bulk path now normalises dates exactly as the non-bulk path does; the only thing blocking is the inline IF that turns the required quality / PHP Quality (phpcs) check red, with a ready suggestion block.

🔴 Blockers (1)

  • phpcs: inline IF not allowed — lib/Db/MagicMapper/MagicBulkHandler.php:264. The ternary added in cb3c357 fails phpcs (reproduced locally on e53ccd4; green on development); the comment carries the if/else rewrite as a suggestion.

🟢 Minor (1)

Verified along the way: parity with MagicMapper::prepareObjectDataForTable() for null, empty, unparseable and DateTimeInterface input; prepareObjectsForDynamicTable() is the only builder of column values, so insert, update, MySQL and PostgreSQL all get the normalisation; a date keeps its wall-clock day.
Scope decision: full PR diff, 3 files, 340 lines, read against a checkout of e53ccd4.
Gates: quality / Hydra Gates fails on gate-7 no-admin-idor and gate-49 controller-exception-translation, identically on development @ 424b31f — pre-existing, not this PR's finding.
CI: besides phpcs, quality / PHPUnit, quality / Integration Tests (Newman) and quality / PHP Quality (phpmd) are red; PHPUnit and Newman are red on development too.

…tions

Bot review on #4773:

[Blocker] phpcs: inline IF not allowed (MagicBulkHandler.php:264). Replaced
the ternary with a `match` expression -- not if/else, which would reintroduce
the ElseExpression phpmd finding the ternary itself was fixing. `match` is
already this file's own style for an adjacent line and satisfies both gates:
no ternary (phpcs), no `else` keyword (phpmd).

Caught while re-verifying: that same code had also pushed
prepareObjectsForDynamicTable() over phpmd's ExcessiveMethodLength threshold,
and the class itself over ExcessiveClassLength -- confirmed empty on
development's copy of this file. Fixed the method by extracting the
date-normalization logic into a concisely-documented normalizeDateProperty()
helper (trimmed from an initial 24-line docblock that made the class length
worse, not better, before landing on the current one). The class-length
finding is different in kind: measured development's class at 986/1000 lines
before this PR touched it -- genuinely pre-existing debt a date-normalization
fix should not have to absorb by splitting an unrelated 1000-line class.
Added one PHPMD\Rule\Design\LongClass baseline entry for this file, the same
mechanism and file this baseline already uses for its StaticAccess entry.

[Minor] Added two concrete expected-value assertions to the integration test
(confirmed against a real database) alongside the existing bulk-vs-non-bulk
parity check, so a regression shared by both paths (e.g. in
DateTimeNormalizer itself) cannot pass silently.

Verified with the exact CI invocations: phpcs 0 errors, phpmd 0 new findings
on the full lib/ tree. Full tests/Unit/Service/Flow/ suite green (1566
tests); BulkAndHandlersIntegrationTest green against a real Nextcloud
install (87 tests, including the strengthened date test with 4 assertions).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@bbrands02

Copy link
Copy Markdown
Contributor Author

Both addressed, pushed e53ccd4 → 805f1d9.

🔴 Blocker — inline IF: fixed, but not with the suggested if/else — used `match` instead. If/else would have reintroduced the `ElseExpression` phpmd finding that the original ternary was specifically written to avoid; `match` has neither a ternary nor an `else` keyword, and this file already uses it the same way two lines below. Verified with the exact CI invocation: `./vendor/bin/phpcs --standard=phpcs.xml --warning-severity=0` → 0 errors.

Caught while re-verifying (not in the review, but would have failed the same gate): that same code change had also pushed `prepareObjectsForDynamicTable()` over phpmd's method-length threshold, and the whole class over its class-length threshold. Confirmed both are new — zero findings on `development`'s copy of this file.

  • Method length: extracted the date-normalization logic into a `normalizeDateProperty()` helper. First attempt at its docblock was 24 lines and made things worse (moved lines into the class, didn't remove any) — trimmed to 5.
  • Class length: measured `development`'s version of this class at 986/1000 lines before this PR touched it. That's pre-existing debt a date-normalization bugfix shouldn't have to pay down by splitting an unrelated 1000-line class. Added one `PHPMD\Rule\Design\LongClass` baseline entry — same file, same mechanism this baseline already uses for its `StaticAccess` entry.

🟢 Minor — parity-only integration test: added two concrete expected-value assertions (confirmed against a real database) alongside the existing bulk-vs-non-bulk equality check: `2026-10-13T13:50:03+00:00` for the date-time (offset converted to UTC) and `2026-10-20` for the date (wall-clock day kept, read back bare at this API boundary). A regression shared by both paths now can't hide behind the parity check alone.

Verified with the exact CI commands: phpcs 0 errors, phpmd 0 new findings across the full `lib/` tree. `tests/Unit/Service/Flow/` suite green (1566 tests); `BulkAndHandlersIntegrationTest` green against a real Nextcloud install (87 tests, the date test now carries 4 assertions).

Comment thread phpmd.baseline.xml Resolved
Comment thread lib/Db/MagicMapper/MagicBulkHandler.php Resolved
WilcoLouwerse
WilcoLouwerse previously approved these changes Oct 22, 2026

@WilcoLouwerse WilcoLouwerse left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (Standard · delta)

Every finding from round 1 is fixed. Two non-blocking 🟢 threads from this round are open, and development requires every review thread resolved before merge, so those two still stand between the PR and a merge, along with this head's CI, which was still running at the time of this review.

🔧 Fix PR: #4783 — 1 of 2 findings, one commit each

Merge it into fix/magicbulkhandler-date-normalization, then re-request the review. The other finding stays on its thread.

Round Found Status
1 (e53ccd4) 🔴 phpcs inline IF · 🟢 parity-only integration test both fixed in 805f1d9
2 (805f1d9) — this review 🟢 new phpmd baseline entry · 🟢 ?string TypeError open
Finding Was Now Where
🔴 phpcs: inline IF not allowed round 1 resolved in 805f1d9 thread
🟢 Integration test asserts parity only round 1 resolved in 805f1d9 thread
🟢 New LongClass baseline entry; the helper could live in DateTimeNormalizer new new — 🟢 phpmd.baseline.xml:124
🟢 ?string $propertyFormat can throw a TypeError (fix proposed in #4783) new new — 🟢 MagicBulkHandler.php:1021
Details of this round

Before the next round: fix rules on (fix-induced 2 + missed-sibling 0 ≥ new 0).

Verified: the match behaves exactly like the old ternary; its default arm covers date-time. normalizeDateProperty() handles exactly the cases the old inline code did (null, non-date formats and non-string values pass through; strings and DateTimeInterface are delegated), and empty strings are still turned into null before it is called. The concrete assertions are correct: 15:50:03+02:00 reads back as 2026-10-13T13:50:03+00:00 via formatDatabaseValueForIso8601, and 2026-10-20T23:30:00-05:00 keeps its day. MagicBulkHandlerDateNormalizationTest passes on 805f1d9 (8 tests); the integration test needs a Nextcloud DB and was not run here. The LongClass claim was measured: phpmd reports none on development and 1015 lines on this head.
Scope decision: delta only, e53ccd4...805f1d9, 1 commit, 3 files, 73 lines.
Gates: quality / Hydra Gates had not reported on 805f1d9 at the time of this review.
CI: phpcs and phpmd are clean locally on MagicBulkHandler.php; the PR's own checks were still queued.

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ 1428eb7

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-specs ✅
test-l10n ✅
test-l10n-parity ✅
format ✅
check-schema-l10n ✅
composer ✅ ✅ 173/173
npm ✅ ✅ 648/648
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌
Newman ❌
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ❌

Quality workflow — 2026-10-22 15:10 UTC

Download the full PDF report from the workflow artifacts.

…izeDateProperty()

Under strict_types the `?string $propertyFormat` parameter turned a
malformed schema `format` (e.g. an integer) into a TypeError, where the
previous inline strict in_array() check simply skipped the property.
Type the parameter `mixed`: the strict in_array() guard already rejects
every non-matching value, so no other change is needed. Adds a unit test
pinning that a non-string format passes the value through untouched.

Review thread: #4773 (comment)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ 9c32569

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ❌
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-specs ✅
test-l10n ✅
test-l10n-parity ✅
format ✅
check-schema-l10n ✅
composer ✅ ✅ 173/173
npm ✅ ✅ 640/640
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌ cancelled — no verdict
Newman ✅ 714/714
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ✅

Quality workflow — 2026-10-30 16:20 UTC

Download the full PDF report from the workflow artifacts.

…lisation test

Main's MagicBulkHandler takes (MagicTableSql, IEventDispatcher, DateTimeNormalizerInterface); the test used the old five-argument constructor. Also restores a blank line the branch dropped by accident.

Assisted-by: ClaudeCode:claude-opus-5-5
@rubenvdlinde
rubenvdlinde merged commit cdbdcdd into main Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants