Skip to content

feat(integriq): build mapping-formats-and-lookups - #3574

Merged
rubenvdlinde merged 21 commits into
developmentfrom
quality/build-mapping-formats-and-lookups
Nov 17, 2026
Merged

rubenvdlinde merged 21 commits into
developmentfrom
quality/build-mapping-formats-and-lookups

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Builds part of openspec/changes/mapping-formats-and-lookups (result: partial).

Tasks done

  • 1: lib/Util/CsvCodec.php with decode(), headers() and encode(). It reads through fgetcsv() on a php://temp stream, so a quoted field with a line break stays one field. Delimiter, enclosure and header row are configurable, and a UTF-8 BOM is ignored. FileMigrationSource::parse() and headers() now call it, and the existing migration source tests pass unchanged.
  • 2 (implementation): a synchronization with sourceConfig.format: csv decodes each page with csvDelimiter, csvEnclosure and csvHeader and hands one array per row to the mapping. The file format field in the synchronization settings is now a select of json, jsonl, xml and csv. For csv it shows the delimiter, enclosure and header-row options.

Tasks left

  • The Playwright half of task 2's test, and tasks 3 (output format and XML writer), 4 (sending XML or CSV to an API target) and 5 (the lookup function and its allowlist).

Notes

  • The three new strings went in through scripts/l10n-ai.js (en and nl). That script writes the whole of l10n/en.js and l10n/nl.js in its own layout, so those two files show a large diff. The content is the old keys plus three new ones.
  • The codec reads with an empty escape character (RFC 4180). The old migration parser used PHP's default backslash escape.

Tests

  • 11 new unit tests: 9 on CsvCodec, and 2 on the CSV fetch path in SynchronizationFetchSinglePageTest. A broken codec line was seen failing them.
  • Unit suite: 8504 tests. The 5 failing tests also fail on development. phpstan is clean on the touched PHP files, phpcs reports no new findings, and eslint is clean on the touched Vue file. test:l10n passes.

Generated by Claude Code

Adds CsvCodec, moves the file migration source onto it, reads a CSV
source response per synchronization, and turns the file format field
into a select with the CSV options.

Assisted-by: ClaudeCode:claude-opus-5-5
…p ci]

Assisted-by: ClaudeCode:claude-opus-5-5
…get golden [skip ci]

SyncConfigWidget now renders a CSV header switch and a format select, so
syncMessageValidation.spec.js's @nextcloud/vue mock needs the switch, and
the type-file golden records the select and the CSV fields. Also puts the
validation-mode docblock back above its own computed.

Assisted-by: ClaudeCode:claude-opus-5-5
…p ci]

l10n: en/nl .js regenerated from the merged .json; the branch's three CSV keys added to en.json and nl.json, where they were missing.

Assisted-by: ClaudeCode:claude-opus-5-5
Task 3: a mapping declares its output format. XMLResponse's writer moves
to ArrayToXml (XMLResponse calls it; its tests pass unchanged),
MappingOutput writes a result as XML or CSV, MappingService::renderOutput
reads outputFormat/outputOptions from the new register fragment (which
seeds person-to-xml), the mapping test returns renderedOutput for XML and
CSV only, and the preview shows it read-only.

Assisted-by: ClaudeCode:claude-opus-5-5
Task 4: a synchronization sends an XML or CSV mapping's output to an API
target as that text with its content type, and no JSON. The mapping is
the synchronization's sourceTargetMapping, or the updateMapping on an
update, read once per engine; a mapping without an output format sends
JSON exactly as before.

Assisted-by: ClaudeCode:claude-opus-5-5
Task 5: lookup(schema, filters, field, default) in both mapping engines,
behind an administered allowlist. A schema off the list fails the
mapping and reads nothing; several matches return the default with a
warning; answers are cached per run; no outbound call. Administrators
keep the list on the admin page through /api/admin/mapping-lookups, and
every change is logged with who made it.

Assisted-by: ClaudeCode:claude-opus-5-5
rubenvdlinde added a commit that referenced this pull request Nov 14, 2026
l10n: kept both sides' new keys.

Assisted-by: ClaudeCode:claude-opus-5-5
…p ci]

Both sides appended keys to l10n/en.js and l10n/nl.js; all are kept.

Assisted-by: ClaudeCode:claude-opus-5-5
…ip ci]

Keep development's backend catalogue order and add this change's
strings as one block after Edit mapping rule, so sibling changes adding
strings at the start merge cleanly. No key or value changes.

Assisted-by: ClaudeCode:claude-opus-5-5
…tail [skip ci]

Same keys and values, added after Edit mapping rule instead of at the
end, so sibling changes appending at the tail merge cleanly.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…p ci]

Assisted-by: ClaudeCode:claude-opus-5-5
…e component [skip ci]

ArrayToXml, CsvCodec and MappingOutput move from lib/Util to
lib/Service/Format and LookupAllowlist from lib/Twig to lib/Service, so the
controllers and the migration source no longer reach two extra components.
ArrayToXml keeps convert() and hands the element building to
XmlElementBuilder. No behaviour change; tests follow the moved classes.

Assisted-by: ClaudeCode:claude-opus-5-5
…ent merge [skip ci]

Assisted-by: ClaudeCode:claude-opus-5-5
…p ci]

Resolve the AdminSettings.vue conflict by keeping both the mapping lookup
and the IdP registration settings panels.

Assisted-by: ClaudeCode:claude-opus-5-5
…wlists [skip ci]

The lookup allowlist section repeated the environment allowlist section
line for line, which lowered the duplication rating. Both now render
AllowlistEditor and take their loading and change handling from the
allowlistEditor mixin; endpoints, messages and test ids are unchanged.

Assisted-by: ClaudeCode:claude-opus-5-5
…es [skip ci]

The lookup allowlist, the register lookup and the XML encoder were called
across components as concrete classes. The controllers and the Twig runtime
now reach them through small contracts bound in ServiceBindings, and the
register lookup moves into the service component it reads from. Restores
the php component independence share to its merge-base level.

Assisted-by: ClaudeCode:claude-opus-5-5
…ers [skip ci]

Completes the previous commit: the callers, the bindings and the tests now
use the new contracts, and the register lookup leaves the Twig component.

Assisted-by: ClaudeCode:claude-opus-5-5
@rubenvdlinde
rubenvdlinde merged commit d6d2947 into development Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant