Skip to content

bug(pull): the system agent can be put in PULL_MODE_PILOT_AGENTS but its system-scoped key is refused by every pull seam #3814

Closed

Description

@obasilakis

Summary

The system agent cannot be a pull agent, but nothing stops it from being configured as one. Putting trinity-system in PULL_MODE_PILOT_AGENTS bakes TRINITY_PULL_MODE=true at recreate and starts a 3-worker pool. Every poll is then refused, because the pull seams accept only the agent's own scope == "agent" key and the system agent's key is scope == "system". Under Phase 5 (default-ON) this becomes the default state for every install.

Observed on eu2

Build 984f8f4c8, 2026-10-19, while widening the #1766 soak to every agent:

[#1081] pull worker trinity-system#w1..w3 started
GET  /api/internal/next-task?agent_name=trinity-system&worker_id=...   403   (46 in 3 min)
POST /api/agents/trinity-system/heartbeat                               403   (23 in 3 min)

Canary B-08 then fired workers_silent for trinity-system, with container_pull_mode: true, idle_workers: 3, last_worker_poll_at: null. That is the only signal. The heartbeat loop arming is a second regression: #1816 deliberately keeps it off the system agent for exactly this 403 reason, but pull mode turns it back on.

Workaround on eu2: trinity-system was removed from the allowlist and recreated, so it is back on push.

Cause (static, dev)

  • routers/internal.py::_pull_authorized delegates to heartbeat_service.authorize_heartbeat, which requires scope == "agent" and a matching agent_name. This is correct and intended ("a compromised pilot can only ever claim for ITSELF").
  • pull_pilot.is_pull_pilot_agent and the recreate path have no system-agent exemption, so the flag is accepted and the pool starts anyway.

Decision needed

Before Phase 5 decides this by accident, one of these has to be chosen:

  1. Exempt the system agent from pull: never bake TRINITY_PULL_MODE for is_system_agent, and keep it on push permanently. This is incompatible with deleting the push path in Phase 5 unless the system agent gets its own route.
  2. Give it a claim path: let the pull seams accept the system agent's own system-scoped key, bound to its own name only.
  3. Refuse at configuration: reject or warn when the allowlist names a system agent, so an operator cannot reach this state silently.

Option 3 is cheap and worth doing regardless. Options 1 and 2 are the actual design choice.

Related: #1081, #1766, #1816, #2840 (B-08).

Decision

Option 2 + 3. Phase 5 deletes the push path for every agent (#1989 ruling; #429 carries out the deletion), so option 1 would strand trinity-system. Option 1 is not shipped, even as a stopgap.

  • Option 3: log a WARN at boot when PULL_MODE_PILOT_AGENTS names the system agent. Kept after option 2 lands.
  • Option 2: every pull seam (next-task, complete/fail, lease renew, heartbeat) additionally accepts a scope == "system" key, only when the requested agent_name passes is_system_agent_name() and is the agent that key belongs to. A system key never claims for another agent. Every other path for system-scoped keys is unchanged.

Acceptance Criteria

  • a. With the system agent in the allowlist and recreated, it claims and completes a task with 0 × 403 and no B-08 workers_silent.
  • b. A system-scoped key requesting next-task for any other agent gets 403 (test).
  • c. A system-scoped key sending a heartbeat for any other agent gets 403 (test).
  • d. A boot WARN is logged when the allowlist contains the system agent.
  • e. Agent-scoped key behaviour is unchanged (existing tests pass).
  • The PR lists every seam the system key now passes.

Activity

  1. obasilakis commented on Oct 20, 2026

    @obasilakis
    ContributorAuthor

    /claim

  2. github-actions commented on Oct 20, 2026

    @github-actions

    Claimed by @obasilakis. Label status-in-progress added.

    Next steps:

    1. Create branch: feature/3814-<short-slug>
    2. Implement the feature
    3. Open a PR with Fixes #3814
  3. added
    status-in-devMerged to dev, awaiting release cut to main
    and removed on Oct 20, 2026
  4. vybe commented on Oct 20, 2026

    @vybe
    Contributor

    Status → status-in-dev: fixed on dev by #3832 (01707a1); the label had not advanced. Ships in v1.0.0.

  5. removed
    status-in-devMerged to dev, awaiting release cut to main
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions