Skip to content

bug: inbound A2A message/send never returns a working task for a long agent turn, so the caller's 30 s timeout loses the answer #3749

Closed

Description

@vybe

Summary

An inbound A2A message/send whose agent turn runs longer than the caller's RPC budget is lost to the caller. Trinity's own outbound client (call_a2a_agent) gives one request 30 s (A2A_RPC_TIMEOUT) and reports the timeout as HTTP 504 with no task id. The server, meanwhile, runs the whole turn inside the request (_run_a2a_task → dispatch_and_await_terminal) and only answers once it is terminal — so for a long turn it never hands back a Task in submitted/working state that the caller could poll. The work completes on the callee, is audited and billed, and nobody can retrieve it from the calling side.

Seen live on 2026-10-15 between two Trinity instances over an AAuth-authenticated call: /design-agent … ran 115 s on the callee (execution completed, full answer in response), the caller got 504 after 30 s and no task_id, and the calling agent's skill fell back as if the advisor had refused. The spec-shaped escape hatch already exists on both sides — the outbound tasks/get poll route (#736) and the inbound tasks/get handler map a running execution to working — but nothing ever produces the working Task that would make them reachable.

Context

A2A is the rail for every cross-instance call (advisor agents, paid calls via x402, AAuth agent identity). Agent turns routinely take one to several minutes. Today only a sub-30-second answer survives the round trip; message/stream (SSE) emits working first, but the bundled client speaks message/send. The client-side docstring states the intent: "a remote that answers working hands back a task_id, and without a poll route that id would be a handle to nothing. The two are one decision." The server half of that decision is missing.

Acceptance Criteria

  • message/send on the inbound A2A server returns a Task in state working (or submitted when the row is still queued) carrying the execution id as id when the turn has not reached a terminal state within a server-side budget that is strictly below the bundled client's 30 s RPC timeout; a turn that finishes inside the budget still returns the terminal Task exactly as today (byte-identical for the fast path).
  • A subsequent tasks/get with that id from the same caller returns the terminal Task with the response text once the execution finishes; this holds for bearer callers, AAuth callers (the execution must be remembered as owned by the caller identity before the early return, or tasks/get answers "Task not found") and paid callers (payer binding).
  • The outbound client surfaces the working state and task_id to the calling agent unchanged (no new error), so an agent following the call_a2a_agent tool's instructions polls get_a2a_task and receives the answer.
  • The idempotency snapshot for the messageId is not frozen on the working Task: a replayed message/send after completion returns the terminal Task, not a stale working one; a replay while still running returns working again.
  • Audit: the a2a_task audit row is written once, on the terminal state, with the same fields as today (auth mode, caller identity, verification, allow-list, execution id); the early return must not produce a second row or a row with state: working posing as completion.
  • Unit tests cover: fast path unchanged; slow path returns working + id; tasks/get after completion for bearer, AAuth and paid callers; replay semantics for both windows; no double audit.
  • Docs: the A2A user docs and the call_a2a_agent tool description say that a long turn comes back as working + task_id to poll, and name the budget.

Technical Notes

  • Inbound: src/backend/routers/a2a.py — _dispatch_jsonrpc (message/send → _run_a2a_task), _task_object, the tasks/get branch (already maps running → working, queued → submitted), _stream_task for the SSE precedent; AAuth ownership via services/aauth/inbound.py remember_execution / execution_owned_by; the paid path has its own tasks/get bound to the payer.
  • Outbound: src/backend/services/a2a_client.py (A2A_RPC_TIMEOUT, _with_deadline), services/a2a_outbound_service.py (poll_task), the MCP tool text in src/mcp-server/src/tools/a2a.ts.
  • Pull-mode pilots: dispatch_and_await_terminal already waits out a queued row; the early return has to coexist with that (a queued row is submitted).
  • Not in scope: making message/send streaming, changing the client's 30 s budget, or push notifications.

Activity

  1. added
    complexity-mediumComplexity: medium (board points 5-8)
    status-readyGreenlit and ready for development (vetted; counterpart to status-incubating)
    on Oct 15, 2026
  2. naief9961-tech commented on Oct 15, 2026

    @naief9961-tech

    One implementation detail worth guarding against: adding asyncio.wait_for(_run_a2a_task(...), timeout=budget) alone would cancel the awaited coroutine when the budget expires. The response deadline needs to be separate from the execution lifetime.

    In src/backend/routers/a2a.py at commit 4357627, the message/send branch awaits _run_a2a_task before constructing the Task and completing the idempotency record.

    A possible implementation approach:

    • Separate dispatch from waiting for terminal state, obtaining the persisted execution ID before the response deadline.
    • Persist caller ownership and payer binding before returning that ID.
    • Wait within a budget below the client’s 30-second timeout. If still pending, return submitted or working using the persisted execution state.
    • Keep execution and terminal finalization independent of the request lifetime. Shielding a task can prevent timeout cancellation, but does not provide restart durability or supervision by itself.
    • Associate the idempotency key with the execution ID and resolve its current state on replay, rather than permanently caching the early working response.

    An additional regression test would disconnect the original caller immediately after receiving the early Task, then verify that execution completes, the same caller can retrieve the result, a different unauthorized caller cannot, and replaying the original message does not dispatch another execution. Also check the race where execution finishes exactly as the response budget expires, including exactly one terminal audit event.

    This is a source-based implementation suggestion; I have not run Trinity’s upstream tests or verified the AAuth/paid integration paths.

  3. vybe commented on Oct 15, 2026

    @vybe
    ContributorAuthor

    Draft PR: #3750 — branch feature/3749-a2a-long-turn-working-task (tip 9cc4b5c, 5 commits off dev 3e5622a). Review MERGEABLE, sweep green after two fixture fixes, mutations M1–M6 caught. Next: /validate-pr, then merge (operator-authorised).

  4. added a commit that references this issue on Oct 15, 2026
    f5fc04f
  5. removed
    status-readyGreenlit and ready for development (vetted; counterpart to status-incubating)
    on Oct 15, 2026
  6. vybe commented on Oct 15, 2026

    @vybe
    ContributorAuthor

    Merged to dev as f5fc04f (PR #3750, squash, 2026-10-15T07:13:57Z) after /validate-pr APPROVE — every guard mutation caught, fast path byte-identical, CI and dev green. Live acceptance after the next deploy: one cross-instance turn over 25 s comes back working + task_id and resolves through get_a2a_task. Follow-ups filed: #3751 (bearer tasks/get read scope, deferred from the plan) and #3752 (AAuth replay-before-ownership + untested paid null-id branch, from validation). Closes with the release PR.

  7. removed
    status-in-devMerged to dev, awaiting release cut to main
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions