Skip to content

The quoll's phantom card was a fixture choosing an arbitrary source, not an unboxed test - #161

Merged
holden merged 1 commit into
mainfrom
claude/razrfly-dictionary-flaky-tests-bae26d
Sep 23, 2026
Merged

holden merged 1 commit into
mainfrom
claude/razrfly-dictionary-flaky-tests-bae26d

Conversation

@holden

@holden holden commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #160. The brief for this work said the flakes came from an unboxed test (@tag :unboxed) running inside an async: true module. There is no such test. At 4ede8b6 and at main, word_page_test.exs and word_live_test.exs mention @tag :unboxed only in comments. runner.ex:122 in ExUnit 1.19 waits for every async module before it starts any sync one. #160 had already made the rest of the requested changes: setup_sandbox/1 refuses unboxed-in-async, unboxed tests truncate on setup, claim_database!/0 locks and empties the database at startup, and the query_canceled came from the 15 s DBConnection default (now 60 s). This PR fixes the one flake that survived all of that.

The unboxed explanation, and why the flake was something else

If you searched "unboxed" to get here: an unboxed test commits real rows, so it's the obvious suspect when a sandboxed test sees rows it never made. For word_page_test.exs:324 it wasn't the cause. WordFixtures.link!/3 makes a naming sense when it's given none. any_source_id/0 picked that sense's source with an unordered limit: 1, so it got whichever row comes first in the heap. Running alone, that's WordNet, the first source seeded, and the sense folds into the WordNet card. Under the async suite, rows from concurrent sandboxes and the holes their rollbacks leave reorder the heap. The naming sense then belongs to poetrydb, artsy or wiktionary and becomes a card of its own. order_by: s.id makes the fixture pick WordNet every time, which is what the test always got alone. No test changes.

Reproduction

  • Main, as it is: 1 of the first 5 full runs failed. The log is below.
    test/devils_dictionary/lexicon/word_page_test.exs:324
    left:  ["wordnet", "poetrydb"]
    right: ["wordnet"]
    
    This was a freshly created partition, with no other process on it, about 200 tests into the async phase.
  • Not a committed row: during a full run, a separate psql polled sources/lexemes/senses twice a second. The counts stayed at 0 throughout the async phase, and the first committed rows appeared with the sync deferred_constraints_test. Replaying seed 205211 four times did not fail again, so the failure depends on timing, not on test order.
  • Organic evidence: I temporarily logged any_source_id/0 against min(id) for one full run. 9 of 68 calls returned bierce ×7, poetrydb or wikipedia.
  • Deterministic reproduction (scratch test, not committed): the body of :324, preceded by update sources set name = name where slug = 'wordnet'. That moves the wordnet tuple to the end of the heap inside the sandbox. Before the fix: unordered limit 1 now returns: [["wiktionary"]], left: ["wordnet", "wiktionary"], which is the second example in the original report. After the fix: the same unordered query still says wiktionary, and the test passes.

Run log

Main, before the fix (MIX_TEST_PARTITION=_flaky7c mix precommit):

run load (1/5/15) result
1 5.1 5.0 4.4 1 failure (:324, poetrydb)
2 12.8 7.2 5.3 0 / 1,736
3 11.4 8.0 5.8 0 / 1,736
4 8.9 8.1 6.0 0 / 1,736
5 11.7 9.1 6.6 0 / 1,736

This branch:

run time load at start → end result
1 85 s 15.2 → 11.4 0 / 1,736
2 87 s 11.4 → 16.0 0 / 1,736
3 86 s 16.0 → 14.0 0 / 1,736
4 90 s 15.6 → 18.0 0 / 1,736, with a second full suite on _flaky7c_repro running beside it (0 / 1,736)
5 90 s 17.4 → 24.7 0 / 1,736, with a second full suite running beside it (0 / 1,736)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Made test fixture source selection deterministic.

…not a committed row

After #160 the suite still failed, once in the first five runs on main, on
word_page_test.exs:324 with `["wordnet", "poetrydb"]`. The async phase
commits nothing (watched from a second connection, and ExUnit 1.19's
runner waits for every async module before it starts a sync one), and
replaying the failing seed four times did not bring it back. The row was
the test's own. `link!/3` makes a naming sense when it is given none, and
`any_source_id/0` chose that sense's source with a bare `limit: 1`: the
first row in the heap. On a table only one run writes to, that is WordNet,
the first source seeded, and the sense folds into the WordNet card. Under
the async suite, other sandboxes' rows and the holes their rollbacks leave
reorder the heap. A probe counted 9 of 68 calls in one full run returning
`bierce`, `poetrydb` or `wikipedia`, and the naming sense became a card of
its own. That is the `artsy` and `wiktionary` of the original report as well.

`order_by: s.id` makes it WordNet every time, which is what the test
always got alone. Reproduced deterministically by rewriting the wordnet
row inside the sandbox so it is no longer first in the heap: `["wordnet",
"wiktionary"]` before, green after. Five full precommit runs at load 11–25,
the last two beside a second full suite on another partition: 1,736 tests,
0 failures each, 85–90 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Essentials

Run ID: e3bb0501-c4c2-431d-ab80-4e4503611f7d

📥 Commits

Reviewing files that changed from the base of the PR and between 6e68bea and 5b66ad7.

📒 Files selected for processing (1)
  • test/support/word_fixtures.ex

Included review availability: 0 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.


📝 Walkthrough

Walkthrough

The any_source_id/0 fixture helper now orders sources by ascending ID before selecting one.

Changes

Fixture source selection

Layer / File(s) Summary
Order source selection
test/support/word_fixtures.ex
any_source_id/0 orders sources by ascending ID before applying the limit. A comment describes the selection.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 5b66a

This fixture change makes source selection deterministic; no remaining merge-blocking risk is evident.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title identifies the fixture’s arbitrary source selection as the issue and distinguishes it from an unboxed test. It is lengthy, but it clearly describes the main change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@holden

holden commented Sep 23, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant