Skip to content

feat(scripts): snapshot pinned contract details schemas and flag breaking changes - #1580

Merged
gabrielseco merged 12 commits into
mainfrom
feat/schema-snapshot-guard
Oct 16, 2026
Merged

gabrielseco merged 12 commits into
mainfrom
feat/schema-snapshot-guard

Conversation

@gabrielseco

@gabrielseco gabrielseco commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a nightly check that the contract details schema served for each version this library pins still behaves the same as when we last committed it, and reports a breaking difference if not.

Why

A pinned schema version is meant to be frozen, but the backend can change what it serves for a version without publishing a new one. That happened when a backend flag was removed: three fields became required for Spain on an already-published version, and we only found out through a failing e2e test on unrelated PRs.

The existing nightly canary didn't catch it, because it auto-fills whatever the schema asks for and then submits. It proves a form can still be saved, not that the schema is the same.

Comparing the schema text isn't enough either: conditions and component references expand on the server, so the served JSON has to be built into a form and compared by what a user would see.

What changed

Toggle details
  • scripts/schema-snapshot/behaviour.ts: builds the served schema the way the Onboarding flow does and records a fingerprint: which fields are visible and required with an empty form, what each single radio, select or checkbox choice adds or removes, and each field's input type, allowed values and length/number limits. Titles, descriptions and ordering are not recorded.
  • Diff rules: a field becoming required or new-and-required, a field disappearing, an input type change, a removed allowed value, a tightened limit, a choice that now requires more fields, or a choice that starts hiding a visible field is breaking. A field becoming optional, a new optional field or a new allowed value is informational.
  • scripts/schema-snapshot.ts (npm run schema-snapshot): fetches the pinned version for every EOR onboarding country from the sandbox gateway and compares against snapshots/contract-details/<COUNTRY>.v<N>.json. --update rewrites the baselines, --country=XXX limits the run, --create-missing only adds baselines that don't exist yet. Breaking differences, a missing baseline or a fetch error exit with code 1. Files it writes are formatted with oxfmt, so they pass check-format.
  • snapshots/contract-details/: baselines for the 92 countries at their current pins (about 930 KB). The version is in the file name, so a bumped pin is never compared against the old version's baseline.
  • Bumping a pin: the PR job runs --create-missing, which writes the baseline for the new version, keeps the older ones so a reverted pin still has its baseline, commits it to the PR branch as a bot, and comments on the PR with how the new version differs from the previous one. It never overwrites an existing baseline, so a pinned version that changed still fails the nightly check. It does not run for PRs from forks, which have no access to the secrets.
  • scripts/schema-snapshot/baselines.ts: small helpers to list baselines and find the previous one.
  • .github/workflows/schema-snapshots.yml: the nightly job compares every country and opens or updates a schema-snapshot issue, closing it when the check passes again. The PR job above reuses the existing canary GitHub App and credentials. Cancellation of in-progress runs is off so the bot's own push can't cancel the run that is still posting the comment.
  • scripts/schema-snapshot/behaviour.test.ts and baselines.test.ts: cover a newly required field, an optional field that becomes required, a conditional that starts requiring another field, a tightened limit, and description-only changes producing no diff, for both useHeadlessForm strategies.

Known limits: it only sees what the sandbox serves, uses fixed scenario values (choices that depend on employer data, dates or salary bands can go unseen), and tries one choice at a time, not combinations. Pins that point at a version the backend still edits in place are held to the same rules.

No public API change. Everything lives under scripts/, snapshots/ and .github/.

Screenshots

N/A

Related Resources

Testing

  • Tested against the example/ app in a browser: N/A, no app code changed.
  • Feature flag: N/A

Ran npm run schema-snapshot -- --update against the sandbox for all 92 countries, then a second comparison run: 89 matched and 3 differed because of a NaN limit that doesn't survive JSON. That is fixed, and those 3 now match on re-run.

Replaying the Spain change (the three fields removed from required in the baseline, then compared with what the sandbox serves now) reports all three as "now required".

Simulated a bump locally by renaming ESP's baseline to an older version and running --create-missing: it wrote the new file and reported that the new version behaves the same as the old one; a second run found nothing to do. The bump PR job itself has not run in CI yet, so its first real run is on the first PR that bumps a pin.

Mutation checks: with the choice-by-choice scenarios disabled, the two conditional tests fail; with required-field detection disabled, the two required-field tests fail.

🤖 Generated with Claude Code


Note

Medium Risk
New CI and bot commits guard onboarding contract-details pins and depend on sandbox secrets; no runtime app API change, but false negatives/positives could affect release confidence for form behavior.

Overview
Adds contract details schema snapshotting so pinned JSON schema versions are compared to committed baselines by form behavior (visibility, required fields, conditionals, constraints)—not raw JSON—catching silent backend changes the submit canary would miss.

Introduces npm run schema-snapshot and scripts/schema-snapshot/ (fingerprint/diff via the same headless form path as Onboarding, baseline helpers, tests), stores per-country/version fingerprints under snapshots/contract-details/, and marks snapshots/** as linguist-generated.

CI: a scheduled workflow runs the compare against sandbox, opens/updates a schema-snapshot issue on breaking drift, and closes it when green; on PRs that touch pins, a bot job runs --create-missing, commits new baselines, and comments with diffs vs the previous version (reuses existing canary sandbox credentials and GitHub App).

Minor: oxlint copy now refers to schema scripts under scripts/ generally.

Reviewed by Cursor Bugbot for commit 5694ec0. Bugbot is set up for automated code reviews on this repo. Configure here.

…king changes

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle Size Report

Metric Current Previous Change Status
Total (gzip) 230.8 kB 230.8 kB 0 B (0%) 🟢
Total (raw) 642.63 kB 642.63 kB 0 B (0%) 🟢
CSS (gzip) 21.95 kB 21.95 kB 0 B (0%) 🟢
CSS (raw) 114.51 kB 114.51 kB 0 B (0%) 🟢

Size Limits

  • ✅ Total gzipped: 230.8 kB / 350 kB (65.9%)
  • ✅ Total raw: 642.63 kB / 850 kB (75.6%)
  • ✅ CSS gzipped: 21.95 kB / 25 kB (87.8%)

Largest Files (Top 5)

  1. index.esm-wIkXrqU6.js - 11.4 kB (0 B (0%))
  2. styles.css - 10.97 kB (0 B (0%))
  3. index.css - 10.97 kB (0 B (0%))
  4. internals-BOZceErx.js - 6.13 kB (0 B (0%))
  5. hooks-ThU7CDLI.js - 5.85 kB (0 B (0%))
View All Files (291 total)
File Size (gzip) Change
index.esm-wIkXrqU6.js 11.4 kB 0 B (0%)
styles.css 10.97 kB 0 B (0%)
index.css 10.97 kB 0 B (0%)
internals-BOZceErx.js 6.13 kB 0 B (0%)
hooks-ThU7CDLI.js 5.85 kB 0 B (0%)
sdk.gen-BRPiw4M1.js 5.58 kB 0 B (0%)
index.js 5.51 kB 0 B (0%)
flows/Onboarding/hooks.js 4.67 kB 0 B (0%)
utils-CUgvJHl4.js 4.07 kB 0 B (0%)
FieldSetField-D0HdSLTs.js 4.04 kB 0 B (0%)

✅ Bundle size check passed

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Deploy preview for remote-flows ready!

Project:remote-flows
Status: ✅  Deploy successful!
Preview URL:https://remote-flows-8iezjupg7-remotecom.vercel.app
Latest Commit:5694ec0

Deployed with vercel-action

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Deploy preview for adp-cost-calculator ready!

Project:adp-cost-calculator
Status: ✅  Deploy successful!
Preview URL:https://adp-cost-calculator-b5andcujt-remotecom.vercel.app
Latest Commit:5694ec0

Deployed with vercel-action

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📊 Coverage Report

✅ Coverage increased! 🎉

Metric Current Previous Change Status
Lines 89.61% 89.59% +0.02% 🟢
Statements 89.48% 89.47% +0.02% 🟢
Functions 88.48% 88.48% 0% ⚪
Branches 81.28% 81.26% +0.02% 🟢

Detailed Breakdown

Lines Coverage
  • Covered: 5398 / 6024
  • Coverage: 89.61%
  • Change: +0.02% (1 lines)
Statements Coverage
  • Covered: 5463 / 6105
  • Coverage: 89.48%
  • Change: +0.02% (1 statements)
Functions Coverage
  • Covered: 1428 / 1614
  • Coverage: 88.48%
  • Change: 0% (0 functions)
Branches Coverage
  • Covered: 3353 / 4125
  • Coverage: 81.28%
  • Change: +0.02% (1 branches)

✅ Coverage check passed

@gabrielseco gabrielseco self-assigned this Oct 15, 2026
gabrielseco and others added 4 commits October 15, 2026 16:14
The baselines are behaviour fingerprints, not scripts, and a top-level
snapshots folder leaves room for other flows to add their own.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Changes under scripts/schema-canary/ trigger the Schema Canary workflow on
PRs. The fingerprint and baseline helpers belong to the snapshot check, so
they now live in scripts/schema-snapshot/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tests

A new optional field with a limit is no longer reported as breaking, and a
choice that stops showing the fields it used to show is breaking instead of
informational. The nightly issue text no longer blames the backend alone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gabrielseco
gabrielseco marked this pull request as ready for review October 16, 2026 06:58

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d215739. Configure here.

Comment thread scripts/schema-snapshot/behaviour.ts Resolved
gabrielseco and others added 2 commits October 16, 2026 09:21
…reaking

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…jsf 1.2.19

The fieldset fixture declared the conditional field as `false` in the base
schema and tried to reveal it in `then`. jsf 1.2.19 (pulled in by kit 1.1.4)
keeps a `false` property subschema when a branch adds a constraint to it, so
the field never became visible. Declare the real field in the base and hide it
in `else`, like the other fixtures.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gabrielseco
gabrielseco requested a review from cammellos October 16, 2026 07:31
@gabrielseco
gabrielseco merged commit 29e9b7f into main Oct 16, 2026
18 checks passed
@gabrielseco
gabrielseco deleted the feat/schema-snapshot-guard branch October 16, 2026 09:02
@gabrielseco gabrielseco mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants