Skip to content

fix(tokens): the nightly token sync finishes inside its hour again - #1713

Merged
rubenvdlinde merged 2 commits into
developmentfrom
fix/token-sync-narrow-retry
Nov 17, 2026
Merged

rubenvdlinde merged 2 commits into
developmentfrom
fix/token-sync-narrow-retry

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Since 10 October the nightly Sync NL Design System Tokens run was cancelled at its 60 minute timeout every night (36 of the last 40 runs cancelled, 3 failed, 1 passed). Nothing from upstream has landed since, and #1139 has been open since 7 October.

Cause. One upstream set, noaberkracht, now fails DenhaagContrastPairsTest::testEveryShippedSetPassesEveryDenhaagPair: its primary colour moved from #947200 to #8e6d00, and the step marker's current state reads 4.34:1. The gate is right to reject it. But a failed first pass sends the sync into a per-set retry that runs the full gate for each of the ~40 sets, at about 3.7 minutes each in CI, which can never fit in the hour.

Change

  • The retry runs only the tests that failed with every set in: scripts/token-set-gate.sh takes TOKEN_SET_GATE_PHPUNIT_FILTER and TOKEN_SET_GATE_VITEST, and narrowGate() builds them from the failure names. A failure that is not a test name (a fatal, say) falls back to the full gate, and the full gate still runs once on the result.
  • The job re-records the golden fixtures and the shell catalogue after converting, and verifies them. They are recorded over the shipped sets, so every sync PR that changed a set turned six unit tests red (checked on this run's output: 5 goldens and ShellCatalogueTest). The job changes sets and never code, so re-recording all 35 goldens only moves the ones that read the sets. Measured: on unchanged sets the re-record writes identical files.
  • tests/Unit/fixtures/golden/ joins the paths the job commits, and the timeout comment states the measured cost.

Verified locally

  • Full sync against nl-design-system/themes@8d49f299 with the default gate: 14m44s, exit 0, noaberkracht the only rejection (it keeps its shipped version). Before: the same run reproduced the cancel path.
  • The re-record and its verify step, on that run's output: all goldens and ShellCatalogueTest pass.
  • hydra/scripts/diff-check.sh with hydra gates: green. vitest syncUpstreamTokens.spec.js: 26 passed (2 new for narrowGate). shellcheck on the gate script: clean.
  • Unit suite (phpunit-unit.xml, no coverage): 2900 tests OK. npm run lint: exit 0.
  • composer check:strict once: only inherited failures, all in lib/ PHP this PR does not touch (cs:check, phpmd, psalm on CustomTokenSetsBundleSection, DesignTokensMapper:430, LoginCopyService, one phpstan error).

Not in this PR. The three upstream sets thematiq lacks (voorne-aan-zee, sittard-geleen, vidar) are skipped by the reader, not by the timeout; they follow in their own PRs. scripts/sync-upstream-tokens.mjs gains /* global console, process */: the eslint config does not declare Node globals for scripts/, which the diff check lints.

Assisted-by: ClaudeCode:claude-opus-5-5

Since 10 October every nightly sync was cancelled at the 60 minute timeout.
One upstream set (noaberkracht, Den Haag step marker at 4.34:1) failed the
first gate pass, and the per-set retry then ran the full gate for each of
the ~40 sets, at about 3.7 minutes each.

The retry now runs only the tests that failed with every set in (a PHPUnit
--filter and/or the failing vitest specs, through token-set-gate.sh), and
the full gate still runs once on the result. Measured locally: the whole
sync takes 14m44s and rejects noaberkracht alone.

The job also re-records the golden fixtures and the shell catalogue, which
are recorded over the shipped sets: without that, every sync PR that
changed a set turned six unit tests red.

Assisted-by: ClaudeCode:claude-opus-5-5
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/thematiq @ befe26a

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ❌
psalm ❌
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-manifest ✅
test-l10n ✅
format ❌
test-fonts ✅
test-token-set-coverage ✅
composer ❌ ✅ 111/111
npm ✅ ✅ 2/2
app:check-code ⏭️
info.xml ✅
REUSE ✅
lockfile sync ✅
PHPUnit ❌ 2905/2912
Newman ✅ 182/182
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ❌

Quality workflow — 2026-11-17 11:54 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
rubenvdlinde merged commit 81cfd54 into development Nov 17, 2026
38 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant