Repository navigation
bug(pull): the system agent can be put in PULL_MODE_PILOT_AGENTS but its system-scoped key is refused by every pull seam #3814
Description
Summary
The system agent cannot be a pull agent, but nothing stops it from being configured as one. Putting trinity-system in PULL_MODE_PILOT_AGENTS bakes TRINITY_PULL_MODE=true at recreate and starts a 3-worker pool. Every poll is then refused, because the pull seams accept only the agent's own scope == "agent" key and the system agent's key is scope == "system". Under Phase 5 (default-ON) this becomes the default state for every install.
Observed on eu2
Build 984f8f4c8, 2026-10-19, while widening the #1766 soak to every agent:
[#1081] pull worker trinity-system#w1..w3 started
GET /api/internal/next-task?agent_name=trinity-system&worker_id=... 403 (46 in 3 min)
POST /api/agents/trinity-system/heartbeat 403 (23 in 3 min)
Canary B-08 then fired workers_silent for trinity-system, with container_pull_mode: true, idle_workers: 3, last_worker_poll_at: null. That is the only signal. The heartbeat loop arming is a second regression: #1816 deliberately keeps it off the system agent for exactly this 403 reason, but pull mode turns it back on.
Workaround on eu2: trinity-system was removed from the allowlist and recreated, so it is back on push.
Cause (static, dev)
routers/internal.py::_pull_authorizeddelegates toheartbeat_service.authorize_heartbeat, which requiresscope == "agent"and a matchingagent_name. This is correct and intended ("a compromised pilot can only ever claim for ITSELF").pull_pilot.is_pull_pilot_agentand the recreate path have no system-agent exemption, so the flag is accepted and the pool starts anyway.
Decision needed
Before Phase 5 decides this by accident, one of these has to be chosen:
- Exempt the system agent from pull: never bake
TRINITY_PULL_MODEforis_system_agent, and keep it on push permanently. This is incompatible with deleting the push path in Phase 5 unless the system agent gets its own route. - Give it a claim path: let the pull seams accept the system agent's own system-scoped key, bound to its own name only.
- Refuse at configuration: reject or warn when the allowlist names a system agent, so an operator cannot reach this state silently.
Option 3 is cheap and worth doing regardless. Options 1 and 2 are the actual design choice.
Related: #1081, #1766, #1816, #2840 (B-08).
Decision
Option 2 + 3. Phase 5 deletes the push path for every agent (#1989 ruling; #429 carries out the deletion), so option 1 would strand trinity-system. Option 1 is not shipped, even as a stopgap.
- Option 3: log a WARN at boot when
PULL_MODE_PILOT_AGENTSnames the system agent. Kept after option 2 lands. - Option 2: every pull seam (next-task, complete/fail, lease renew, heartbeat) additionally accepts a
scope == "system"key, only when the requestedagent_namepassesis_system_agent_name()and is the agent that key belongs to. A system key never claims for another agent. Every other path for system-scoped keys is unchanged.
Acceptance Criteria
- a. With the system agent in the allowlist and recreated, it claims and completes a task with 0 × 403 and no B-08
workers_silent. - b. A system-scoped key requesting next-task for any other agent gets 403 (test).
- c. A system-scoped key sending a heartbeat for any other agent gets 403 (test).
- d. A boot WARN is logged when the allowlist contains the system agent.
- e. Agent-scoped key behaviour is unchanged (existing tests pass).
- The PR lists every seam the system key now passes.
Activity
obasilakis commentedon Oct 20, 2026
/claim
github-actions commentedon Oct 20, 2026
Claimed by @obasilakis. Label
status-in-progressadded.Next steps:
feature/3814-<short-slug>Fixes #3814fix(pull): review follow-ups for the system agent as a pull pilot (#3814
fix(pull): the system agent claims its own queue with its system-scop…
vybe commentedon Oct 20, 2026
Status →
status-in-dev: fixed ondevby #3832 (01707a1); the label had not advanced. Ships in v1.0.0.