test_dogfood_whitelist is flaky when the whole file runs #228

Activity

1. 0xKT

   added a commit that references this issue on Jul 29, 2026

   test: clear the audit debt in the test suite (#230)

   ...

   a740baa
2. 0xKT commented on Jul 29, 2026

   

   0xKT

   on Jul 29, 2026

   MemberAuthor

   More actions

   Diagnosed. The "Where the evidence points" section above is wrong -- there is no interference between the parametrised sessions. Three separate defects stack up, and each one hides the next.

   1. The exit sequence sends one Ctrl+C, but Ctrl+C is a three-step ladder

   Raven/ui-tui/src/app/useInputHandlers.ts

   Lines 434 to 479 in a740baa

   	if (key.ctrl && ch.toLowerCase() === 'c') {
   	if (live.busy && live.sid) {
   	// Phase 6 typed cancel path (per design.md §D5). When the chat
   	// stream is attached AND a typed turn is in flight, route Ctrl+C
   	// into `turn.cancel`; the server's `error(reason=cancelled_by_client)`
   	// event drives the actual UI reset via `chatStream.dispatch`. When
   	// chatStream is detached or no typed turn is active, fall back to
   	// the legacy `session.interrupt` path so existing CLI/agent
   	// streaming behavior (still consumed by 169 .tsx) is preserved.
   	const chat = chatStreamRef?.current ?? null
   	if (chat?.isTurnActive()) {
   	if (live.escapeArmed) {
   	// Second Ctrl+C while the turn is still in flight — the first
   	// cancel produced no terminal event (events lost / server
   	// wedged). Hard-reset the prompt locally so the user is never
   	// stuck waiting on a response that can't arrive. This
   	// does NOT route through the legacy session.interrupt path.
   	patchUiState({ escapeArmed: false })
   	chat.forceReset()
   	return
   	}
   	// First Ctrl+C: issue the normal server cancel and arm the escape
   	// hatch — the busy placeholder flips to the force-quit hint and a
   	// second Ctrl+C falls back to the local reset above.
   	patchUiState({ escapeArmed: true })
   	chat.cancel().catch((err: Error) => actions.sys(`cancel failed: ${err.message}`))
   	return
   	}
   	return turnController.interruptTurn({
   	appendMessage: actions.appendMessage,
   	gw: gateway.gw,
   	sid: live.sid,
   	sys: actions.sys
   	})
   	}
   	if (cState.input || cState.inputBuf.length) {
   	return cActions.clearIn()
   	}
   	return actions.die()

   A single Ctrl+C only reaches actions.die() when the UI is not busy and the composer is empty. Otherwise it is consumed as a turn cancel or as clear-input. The test presses escape then exactly one ctrl+c, even though the file's own docstring describes the pattern as "Ctrl+C x2".

   Changing the teardown to press Ctrl+C until the session exits (bounded at 4) and running the whole file:

   before: 3 failed, 4 passed, 4 xfailed, 3 xpassed
   after:  0 failed, 7 passed, 7 xpassed
   

   Instrumented press counts show channels list, skill get, cron show and sentinel status each needed two.

   All seven xfail(strict=False) cases XPASS once the exit sequence is right, so the marker reason ("Raven TUI overlay exit UX inconsistent") describes a harness bug, not a Raven UX defect. The markers should go.

   2. Every one of the 14 output assertions is vacuous

   The expected-output regex is matched against the whole screen, and the screen includes the composer line echoing the command that was just typed. 13 of the 14 patterns are satisfied by that echo alone -- skill list matches on skill, cron show on cron, sandbox list on sandbox, and so on. The fourteenth, status (OpenRouter|Model:|provider), is satisfied by the welcome frame's own status line (claude-opus-4.8 - OpenRouter - tui:...).

   This is a pure string property, no timing involved: comparing each pattern against the string > /<command> reproduces it. So the file never verified that any slash command produced output; it only proved that the TUI starts and eventually exits.

   That also retracts this issue's closing claim that "the individual cases genuinely exercise the slash routing path".

   3. Readiness keys off the banner, so input can be dropped silently

   The file waits for r"Raven", which paints before the app accepts input. Typing immediately after it, the composer was observed still showing its placeholder (Ask me anything...) with the command nowhere on screen -- the keystrokes were dropped entirely. Defect 2 is what makes this invisible: a dropped command still satisfies the output assertion.

   Why the earlier readiness experiment made skill_list fail deterministically

   That result is recorded above as a reason to revert the readiness fix. It is actually the three defects interacting: fixing readiness means the command really submits and really runs a turn, so the UI is busy when the single Ctrl+C arrives, and defect 1 turns from intermittent into certain.

   Ruled out, with evidence

   • Session interference. A pytest plugin logging tui-use list at every setup and teardown shows each case's session killed before the next one starts; no case ever ran with a sibling session alive.
   • Daemon reaping the exit code. tui-use info keeps Status: exited and Exit Code: 0 indefinitely (polled for 14s after exit), so expect_exit cannot lose a code it already had.

   Fix in progress

   All three, on test/dogfood_ctrl_c_exit_ladder: wait for the status bar to report ready, assert command output outside the composer line using patterns taken from real output, press Ctrl+C until exit, and drop the seven bogus xfail markers.
3. 0xKT

   closed this as completedin #236on Jul 30, 2026
4. KevinHuSh

   added 2 commits that reference this issue on Sep 14, 2026

   test: clear the audit debt in the test suite (EverMind-AI#230)

   ...

   1a35bc5

   test(tui): fix the dogfood e2e suite that never asserted anything (Ev…

   ...

   caec60f