You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug(tests): the journey tier's model gate reads the pytest host's environment, so keyed journeys skip invisibly on subscription-authenticated stacks #2812
The two journeys that assert a real model answer — J03's first turn (tests/journeys/test_first_turn_journey.py) and J10's "I can read what they said" and "every fan-out subtask completes" (tests/journeys/test_j10_agent_calls_agent_journey.py) — gate on os.getenv("ANTHROPIC_API_KEY") in the pytest process. That is the harness host, not the instance under test. A stack whose agents authenticate by subscription (SUB-003: CLAUDE_CODE_OAUTH_TOKEN inside the container, rows in subscription_credentials) has no such variable on the host yet answers normally, so on that stack the keyed journeys skip permanently, and invisibly — the skip reason is already allowlisted in tests/harness/audit_skips.py, so the skip audit does not surface it.
Context
Raised in review of #2809 (finding I2) and deferred there because the right replacement signal is not obvious. A new agent receives ANTHROPIC_API_KEY from the backend's own resolver at creation (services/agent_service/crud.py, get_anthropic_api_key()), and a subscription replaces it only through the auto-switch service (services/subscription_auto_switch.py, ensure_serviceable_subscription). So gating on "GET /api/subscriptions is non-empty" could turn today's permanent skip into a false failure if a freshly created ephemeral agent never gets a subscription assigned. The gate must reflect what the callee will actually have, not what the instance has somewhere.
Acceptance Criteria
One shared helper in tests/journeys/conftest.py decides "a real model will answer on this stack", used by J03 and J10 alike (no per-file os.getenv gate)
The decision reflects the instance, not the harness host: either an instance-side "a model credential is configured for new agents" signal, or a one-time session probe of an ephemeral agent
On a stack where agents cannot answer, the keyed tests still skip with the allowlisted reason; on a subscription-only stack where they can, they run
docs/testing/STRATEGY.md's note on credential-bound journeys says how the gate decides
Technical Notes
Candidate signals, in order of honesty: a session-scoped probe that sends one short turn to the first ephemeral agent the tier creates and records the outcome; or an endpoint that reports whether the platform key or a subscription is configured, if one exists without disclosing values. Related: the debt-inbox item on a keyed nightly lane, which is what would make this gate matter in CI at all.
Activity
test(journeys): J10 review follow-ups — clear_edges proves its reset,…
feat(tests): J10 journey — agents call each other through the MCP ser…
fix(tests): the model gate needs the instance credential too, not jus…
fix(tests): the journey model gate asks the instance, not the pytest …