[0xKT]

Summary

Follow-up to #413, from a measured look at what was left: a CI shard now spends 72 s on fixed cost (22 s environment, 48 s collection and worker start) against 82 s of tests, the four shards are within 15 s of each other, and 70 unmarked tests were still over the 2 s wall-clock ceiling on CI against 42 on a laptop because CPU-bound work runs three to four times slower there. This PR makes the ceiling a gate that is portable across machines, takes the remaining waits out, precompiles bytecode so the workers stop doing it four times, and fixes two product waits the same investigation surfaced.

• test(*): the ceiling judges idle time instead of wall clock, where idle is the wall clock minus every CPU second spent on the test's behalf, the reaped children's included (os.times). A test waiting on a production sleep or a process that answers nothing waits the same seconds on every machine; a test that computes for 2 s here and 7 s on CI, or drives a browser or eight spawned workers, is not the problem. The numbers ride on the teardown report so the xdist controller can judge them; options are --idle-ceiling / --idle-ceiling-strict. The fixture that shortens production ladders takes in the ACP cancel settle (5 s) and the gateway delivery grace (2 s); the CLI agent tests stub the one-shot command's unconditional 2 s hand-off grace; the rpc subagent tests no longer spawn a login shell or probe a real endpoint over the network; three literal timeouts shrink where the stub never answers anyway. What is left over the ceiling carries a marker: seven real LibreOffice renders and two tests that wait on a process answering nothing carry slow, two whose assertion is the wait itself carry production_timing.
• ci(*): every shard runs with --idle-ceiling-strict (on the make line, not in addopts: the Windows job runs --noconftest). Bytecode precompilation was tried and dropped, the numbers are below; the workflow is otherwise unchanged from ci(*): run unit tests as four shards and speed up the suite #413.
• test(agent): a heartbeat test asked for five beats of the fifteen its intervals should produce in a 0.6 s window, and a loaded runner delivered three. It reads the two gaps instead, which the load cannot invert: the wait before the first beat is one interval, the wait after it is the other. Older than the gate that surfaced it; nothing here touches the registry.
• perf(ppt-engine): MeasuredWidth.width remembers measured widths (bounded, oldest key out); a 15-slide template's QA gates made 1848 freetype calls over 162 distinct keys. Importing text_metrics first raised on a circular import through template.capacity; capacity now imports the module and resolves the name at call time.
• perf(*): raven gateway and raven web start the litellm import on a daemon thread at boot, so saving a key from the settings page no longer waits three to five seconds on it.
• fix(providers): one thread at a time initialises litellm. It raises three logger levels for the duration of the import and puts the captured values back after, which holds only while one thread is doing it; a second caller arriving inside that window captures WARNING as the level to restore, and litellm's DEBUG and INFO stop reaching the file sink until a restart. The window was narrow while every import sat on a startup path and is ordinary now that a server warms it in the background, so the two callers share a module lock. The strict-mode warm-up also runs before the fixtures that redirect the home, and publishing the OAuth token directories created them, so it warms under a temporary home.

Type

• Fix
• Feature
• Docs
• CI / tooling
• Refactor
• Other

Verification

Whole suite, uv run --frozen --all-extras pytest -q (macOS, 4 workers), before and after the waits came out and with the ceiling strict:

run	result	idle list
main after #413	22052 passed, 43 skipped in 219.41 s	39 over 2 s wall clock
this branch, --idle-ceiling warning	22057 passed, 43 skipped in 210.20 s	14 over 2 s idle, all real render or subprocess, now marked
this branch, --idle-ceiling-strict	22058 passed, 43 skipped in 204.89 s, exit 0	none

The ceiling is three seconds, placed from the suite's own distribution rather than guessed. Unmarked tests over one second of idle, measured across the whole suite:

idle	tests	what they are
1.00-1.19 s	19	a test that sets a one second timeout and lets it expire
1.2-1.3 s	3	a subprocess handshake
1.5-1.6 s	3	an ACP turn interrupted after 0.8 s, and two like it
above 1.6 s	0	the renders above this line all carry a marker

A runner stretches that 1.6 s to about 2.2 s, which is how a two second line failed two shards over tests waiting exactly as long as they meant to. Three seconds sits in the gap the population leaves, and stays under every ladder this guards against: the shortest constant the audit found was a three second grace, the rest run 5, 15, 30 and 60 seconds.

What the metric separates, measured on this suite: a test that sleeps 5 s reports 4.96s idle of 5.02s; a test that drives a headless browser for 13 s reports under 0.8 s; six raven doctor subprocesses report 1.2 s where wall clock said 4.2 s; eight spawned processes contending for a lock report under 0.8 s where wall clock said 3.6 s.

Files touched by the product changes and the new markers (uv run --frozen --all-extras pytest -q tests/test_litellm_setup.py tests/test_cli_gateway_commands.py tests/test_cli_serve_commands.py tests/test_ppt_engine_assets_fonts.py tests/test_ppt_engine_template_capacity.py tests/test_cli_doctor_commands.py tests/test_ppt_engine_end_to_end.py tests/test_ppt_engine_render_service.py tests/test_rpc_model.py): 434 passed, 2 skipped in 33.39s, none over 1 s idle. python -c "import raven_ppt.services.assets.text_metrics" in a fresh interpreter: ImportError before, imports after.

Test-side speedups, measured per test: the ten CLI agent tests 2.0-4.4 s to 0.05-0.36 s; the three rpc subagent tests 1.4-2.6 s to 0.01-0.02 s; the rpc save-key test 2.2-2.8 s to 0.03 s; the two ACP probe tests 2.1 s to 0.2 s.

Bytecode precompilation, seconds from the start of the pytest step to the first test, per shard, over three runs of this workflow:

what was compiled	shard 1	shard 2	shard 3	shard 4	mean	install step
nothing (#413)	52.7	46.5	45.9	46.0	47.8	3-4 s
site-packages and checkout	39.8	39.3	35.6	45.5	40.1	12-17 s
checkout only	55.5	50.1	44.4	53.8	51.0	3-4 s

Compiling site-packages is the half that moved the number and it cost more in the install step than it returned. Compiling the checkout alone is inside the noise of doing nothing: the four workers share one .pyc cache, so the first to import a module writes it for the rest, and most of what a laptop measures as cold compilation is pytest rewriting test files through a cache compileall cannot fill. Both are gone; the comment in the workflow records it.

The reworked heartbeat assertion was checked against the bug it guards by making the registry keep its first interval for every beat: one beat arrives in the window and the assertion fails on it.

The diff gate is reproduced locally against origin/main after a full make coverage: Diff coverage: 100.00% (12/12 executable changed lines). The two warm-up calls in the serving boot coroutines carry a no-cover pragma naming the test that pins them, because those coroutines bind a port and serve until they are killed.

The lock is checked against the race it closes: with it taken away, a test that holds the first caller inside the import and starts a second fails on [30, 30, 30] == [20, 20, 20]. A strict session under a clean HOME now leaves no .raven behind.

make lint-python clean.

• Relevant tests pass locally
• Relevant lint / type checks pass locally
• User-facing docs or screenshots are updated when needed (none)

Risk

• The ceiling is now a gate on CI. A new test that waits more than 2 s without slow or production_timing turns its shard red; the summary names the test and the marker to use. A local run stays warning-only.

• A subprocess that does real work is charged to the test that reaped it, so only a real wait trips the gate. A strict session imports litellm up front, because its 1.7 s of file opens would otherwise land on whichever test in the shard imports it first.

• The litellm warm-up runs on a daemon thread at server boot only; unit tests never reach either boot path (their source is pinned instead), and a failed import is logged, not raised.

• The width memo returns the same floats; the cap evicts the oldest key. The capacity import change resolves the same name one step later.

• Rollback: revert the PR; every change is additive or a marker.

• Security impact considered (no security surface touched)

• Backward compatibility considered

• Rollback path is clear for risky changes

Related Issues

N/A

[gloryfromca]

Blocking: fix the failing strict idle gate and serialize LiteLLM initialization before merge.

I found two blockers, marked inline.

I reviewed the full diff, the affected xdist/report and server/provider call paths, and the four-commit history. I also checked the applicable AGENTS.md, CLAUDE.md, and CONTEXT architecture rules, backward compatibility, and whether the test edits weakened behavior. The timeout reductions, mocks, and timing markers retain the assertions they cover; I found no additional blocker there.

Verification:

• uv run --frozen --python 3.12 --all-extras pytest tests/test_litellm_setup.py tests/test_ppt_engine_assets_fonts.py tests/test_cli_gateway_commands.py tests/test_cli_serve_commands.py -q --idle-ceiling-strict: 144 passed.
• Focused rerun of GitHub's over-ceiling tests with the same strict flag: 6 passed, 1 skipped, but the session failed on two over-ceiling tests (4.40 s and 2.40 s idle).
• Fresh-process two-thread import_litellm() reproduction: no exceptions, but all three logger levels changed from INFO (20) to WARNING (30).
• GitHub CI at this head: unit shards 1/4, 2/4, and 4/4 failed on seven over-ceiling tests; coverage gates were skipped.

[gloryfromca]

Blocking: make the remaining strict shard pass, serialize LiteLLM initialization, and keep strict test startup out of the real Raven home.

The previous idle-gate blocker is narrower but still open: shards 1/4, 3/4, and 4/4 now pass, while shard 2/4 still fails and leaves coverage skipped. The LiteLLM concurrency blocker is code-identical and still reproduces. I found one new blocker in the strict-mode warm-up, marked inline.

I re-read the full current diff and the two-commit delta, relevant pytest/xdist and provider call paths, and commit history. I also rechecked AGENTS.md, CLAUDE.md, CONTEXT architecture boundaries, backward compatibility, and whether the updated mocks and removed timing markers weakened their tests. The probe isolation remains covered by dedicated probe tests, and I found no further blocker in those changes.

Verification:

• Focused rerun of the seven formerly over-ceiling tests with --idle-ceiling-strict: 6 passed, 1 skipped locally; the skip is the unavailable Chromium system library.
• GitHub CI: unit shard 2/4 fails because the browser crop test records 9.99 seconds idle; coverage is skipped.
• Fresh-process concurrent import_litellm() reproduction still ends with logger levels [30, 30, 30].
• One strict test under a clean HOME passed but created $HOME/.raven/oauth before test fixtures ran.

[gloryfromca]

Blocking: serialize LiteLLM initialization before this can merge.

The current delta fixes the strict-idle-gate objection: the two real-browser tests are legitimately marked slow, that marker does not deselect them from the default suite, and the current GitHub run reported no idle-ceiling violation. Removing the measured-unhelpful bytecode precompile is also internally consistent.

The LiteLLM initialization race remains the sole blocker. It satisfies all three round-five requirements: this PR introduces the background import; ordinary first-request traffic can overlap its multi-second initialization; and after the overlap leaves LiteLLM logger levels at WARNING, lower-level file logging is lost until the operator restarts, with the same race exposed again. The background and on-demand paths still need one shared synchronization boundary.

Named nonblocking follow-ups: strict local pytest still creates or chmods the developer's real ~/.raven/oauth before isolation fixtures run, and the PR title still promises precompilation even though this revision removes it. I withdrew and resolved the home-isolation blocker under the late-round convergence rule.

Verification: uv run --frozen --python 3.12 --all-extras pytest -q --idle-ceiling-strict tests/test_design_engine_rendering_outcome.py completed with 22 passed and 2 skipped; both skips were because local Chromium lacks libatk-1.0.so.0. GitHub unit shards 1/4, 2/4, and 3/4 passed. Shard 4/4 had no idle-ceiling failure but finished 1 failed, 5522 passed, 5 skipped on the untouched timing-sensitive test_the_first_beat_is_late_and_the_rest_are_not, so the overall workflow is not green; I treated that as out-of-scope verification evidence rather than a finding introduced by this revision.

Coverage: I read the full github/main...HEAD diff and the delta from the prior reviewed head, checked the repository rules, relevant callers and commit history, backward compatibility, test-selection semantics, whether the new marks weakened coverage, and the warm-up architecture. No dependency, CLI, asset, or domain-term rules are implicated.

[gloryfromca]

Blocking: synchronize LiteLLM initialization and make the new warm-up calls satisfy diff coverage.

The new heartbeat-test delta is sound: it now checks the initial and recurring gaps that the implementation actually promises, without depending on how many callbacks a loaded runner schedules. It introduces no additional blocker.

The existing LiteLLM initialization race remains unchanged and open. The current CI run also surfaced the new inline blocker now that every strict-idle unit shard completes: PR diff coverage is 78.57%, below the required 90% gate.

Verification: uv run --frozen --python 3.12 --all-extras pytest -q tests/test_tool_registry_timeout.py passed all 14 tests. GitHub CI passed all four strict-idle unit shards and every other completed check except the required diff-coverage gate.

Coverage: I read the full github/main...HEAD diff and the delta from the prior reviewed head, checked the affected heartbeat implementation and test history, the unchanged LiteLLM callers and synchronization architecture, repository rules, compatibility, and whether the test rewrite weakened the behavior being guarded. No dependency, asset, CLI-naming, or domain-term rule is newly implicated.

[gloryfromca]

Blocking: stabilize the strict idle ceiling and serialize LiteLLM initialization before this can merge.

The current delta fixes the prior diff-coverage objection. Moving the serve import to module scope and documenting the two long-lived boot calls as coverage exclusions is supported here by source-presence assertions plus direct helper tests; isolated exact-head coverage confirms those are the intended exclusions. I replied and resolved that thread.

Two blockers remain. The LiteLLM logger-level race is unchanged. The strict gate is also not portable yet: exact-head shards 3/4 and 4/4 fail solely on new 2.19 s and 2.09 s idle hits, although the prior revision passed all four shards and both named tests pass alone under the same strict option. This branch introduces the required gate, normal PR execution reaches it, and there is no merge path when it fails and skips coverage, so all three late-round blocker conditions are met. I reopened the original gate thread rather than filing a duplicate.

Verification: four focused warm-up call-site/helper tests passed in an isolated exact-head snapshot; the two exact CI idle hits passed there under --idle-ceiling-strict; git diff --check github/main...HEAD passed. GitHub shards 1/4 and 2/4 passed, shards 3/4 and 4/4 failed only at the strict idle summary, and coverage was skipped.

Review coverage: I bound the full github/main...HEAD patch at base a5f84b7164fe and head 0008ff0813a8 (SHA-256 e678b5c7684c4d6cf7072d15b6d7ffbdc9520a26e41f1c3ec19c14d4cbaf4c2b), read the current delta, and checked repository rules, callers and history, compatibility, test weakening, concurrency, filesystem side effects, performance caching, CI rollout, and recoverability. Patch-risk assessment: revise; moderate impact, high regression likelihood, partial protection, easy recovery, high confidence.

[gloryfromca]

Blocking: serialize LiteLLM initialization before this can merge.

The current delta fixes the strict-idle instability: exact-head CI run 34983686574 passed all four strict unit shards and the coverage gate, and I closed that thread. No additional finding survived review.

The existing LiteLLM race remains the sole blocker. For the late-round bar: this change introduces the competing background import; gateway or serve startup followed by an ordinary first request can reach the overlap; and after it corrupts the process-global logger levels there is no reliable in-process recovery, while restarting reopens the same race. The background and on-demand paths still need one shared serialization boundary.

I covered the repository rules, full diff and current delta, callers and history, backward compatibility, test-strength changes, concurrency, filesystem side effects, performance intent, and required-CI behavior. Verification: exact-head CI passed in full; git diff --check github/main...HEAD passed.

Patch-risk assessment: revise; moderate impact, high regression likelihood, partial protection, easy recovery, high confidence. Immutable patch: a5f84b7164fe24acc17e8802c1113d2cb249a8e2...cf7199c5a74748fefff9b947cf9663c81689fe48, SHA-256 4c92c6c9742608b715f43c242f1b2e085723e27d089de6dbfb1365ecbb43d3ff.

[gloryfromca]

No blockers; this can merge as far as I am concerned.

The new commit fixes the remaining LiteLLM race by serializing the complete initialization sequence, and it also isolates strict-mode warm-up from the developer's real Raven home. The two-thread regression test and a clean-HOME strict reproduction both pass; all four exact-head unit shards and the coverage gate pass in CI run 35049479697. All threads I opened are resolved.

I covered the repository rules, the full diff and new-head delta, callers and history, backward compatibility, whether tests were weakened, concurrency and failure behavior, filesystem isolation, and the CI/architecture constraints. git range-diff shows the rebase preserved every previously reviewed commit and added only this fix; git diff --check github/main...HEAD passes. No additional finding survived refutation.

Local verification: uv run pytest tests/test_litellm_setup.py -q -> 15 passed; strict clean-HOME reproduction -> 1 passed with no .raven/oauth created.