Skip to content

test: fix dev test-suite drift + harness bugs (no product changes) - #1520

Merged
dolho merged 1 commit into
devfrom
fix/dev-test-drift-and-venv
Jul 8, 2026
Merged

dolho merged 1 commit into
devfrom
fix/dev-test-drift-and-venv

Conversation

@vybe

@vybe vybe commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

A full-suite run on dev (Docker, local) surfaced 38 failures + 2 errors. Every one was root-caused to a stale test or a test-harness issue — zero product regressions. This PR is test-files-only (0 lines of production code).

Verified after fixes: 74 unit + 162 integration = 236 passed across the touched files.

Fixes (what drifted, why)

Cluster Fix Root cause
test_platform_prompt_unit (5) stub lambdas accept **_kwargs production compose path now forwards runtime= (#1187)
test_cb_probe_execution_close (7) add get_setting_value to _CanaryTempDB canary temp-DB stub leaks onto module-level from database import db; under full-suite order it serviced get_platform_default_model and AttributeError'd
test_fork_to_own (2 errors) defensive list_all_agents_fast backfill leaked bare services.docker_service stub (same sys.modules cross-file leak class)
test_files_guardrail_bypass, test_mcp_validator_endpoint (2) assert "Disallowed credential file path" intentional message wording change (#1305)
test_event_subscriptions (1) expect 403 not 400 intentional #186 enumeration-safety collapse (commit 34ddf71a)
test_activities (1) derive valid triggers from canonical _TRIGGER_BUCKETS hard-coded allow-set drifted behind webhook / channel triggers; now tied to source of truth
test_whatsapp_integration (4) read pending-login Redis via backend's authed client bare redis-cli GET fails NOAUTH (Redis ACL mandatory since #589) but exits 0, defeating the returncode guard. Product code verified correct.

Clusters 1–3 pass in isolation and only failed under full-suite ordering (sys.modules leakage). A durable follow-up would harden the tests/unit/conftest.py sys.modules baseline restore — out of scope here.

⚠️ Not in this diff (environment, not code)

Running the full suite locally still shows a handful of failures/skips that are not fixed by this PR because they are environmental — a clean CI run is unaffected:

  • 9 unit failures (bcrypt 72-byte error ×3, ModuleNotFoundError: PIL ×6) — a drifted local tests/.venv (bcrypt 5.0.0, no Pillow). requirements-test.txt already pins bcrypt>=4.2.0,<5 and Pillow>=11.1.0. Resync your local venv: cd tests && .venv/bin/pip install -r requirements-test.txt.
  • ~60 infra-broken skips ("Agent server not ready", "testfix container not found", 429s) — the shared test agent was never ready (TEST_AGENT_NAME unset, no testfix fixture). Needs a persistent test agent.
  • 4 quota failures (test_dynamic_thinking_status async ×3, one 429) — the real Claude subscription hit its usage cap during the 50-min run.
  • 3 environmental (workspace_available/auto_switch defaults, agent_rename name collision) — pass in a clean CI env / fresh DB; code defaults are correct.

Test plan

  • pytest on all touched files (leak-order preserved for canary→cb_probe): 236 passed
  • Confirmed diff touches no src/backend/ production code

🤖 Generated with Claude Code

A full-suite run on `dev` surfaced 38 failures; all root-caused to stale
tests or test-harness issues — zero product regressions found.

- platform_prompt_unit: 5 stub lambdas now accept the `runtime=` kwarg the
  production compose path forwards (#1187 runtime-aware platform prompt).
- cb_probe_execution_close: add `get_setting_value` to `_CanaryTempDB`. The
  canary temp-DB stub leaks onto the module-level `from database import db`
  reference, so under full-suite ordering it services
  task_execution_service.get_platform_default_model and AttributeError'd.
- fork_to_own: defensive `list_all_agents_fast` backfill for a leaked bare
  `services.docker_service` stub (same sys.modules cross-file leak class;
  both pass in isolation, only error under full-suite order).
- files_guardrail_bypass / mcp_validator_endpoint: assert the current
  "Disallowed credential file path" wording (#1305 curated credential
  file-type injection).
- event_subscriptions: nonexistent source now expects 403, not 400 —
  intentional #186 enumeration-safety collapse (commit 34ddf71).
- activities: derive the valid `triggered_by` set from the canonical
  db/schedules._TRIGGER_BUCKETS map so it stops drifting each time a
  trigger/channel value is added (webhook, telegram/slack/whatsapp, ...).
- whatsapp_integration: read the pending-login Redis key via the backend's
  authenticated client instead of a bare `redis-cli GET`. Redis ACL auth is
  mandatory since #589; `redis-cli` exits 0 on NOAUTH, so the helper's
  returncode guard never tripped and the value never matched. Product code
  is correct (verified by writing/reading the key via the real adapter).

Not in this diff (local environment, not code):
- tests/.venv had drifted (bcrypt 5.0.0, no Pillow), causing 9 unit failures
  (bcrypt 72-byte error, ModuleNotFoundError: PIL). requirements-test.txt
  already pins `bcrypt>=4.2.0,<5` and `Pillow>=11.1.0`, so a clean CI venv is
  unaffected — resync the local venv with `pip install -r requirements-test.txt`.

Verified: 74 unit + 162 integration = 236 passed on the touched files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dolho

dolho commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Review: ✅ Approve — safe to merge

Test-files-only, zero production lines (verified: diff touches only tests/). Each of the 40 failures is credibly root-caused to test drift or a harness bug, not a regression:

CI green across all 3 seeds × base/head. One note (already called out in the PR): clusters 1–3 are band-aided per-file against sys.modules cross-file leakage; the durable fix is a conftest.py baseline restore. Fine to defer, but worth a follow-up issue so this doesn't recur file-by-file.

@dolho
dolho merged commit 4f6813f into dev Jul 8, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants