Skip to content

bug(recovery): soft-delete recovery dead-ends — recover tells you to start, but start 404s when the container is gone #1559

Closed

Description

@webmixgamer

Summary

Admin soft-delete recovery (#834 Phase 1c) can restore the agent_ownership row but cannot bring the agent back. recover_agent explicitly defers container creation to the operator, and the endpoint it defers to refuses to run without an existing container — which never exists after a delete.

Evidence

src/backend/routers/admin_recovery.py::recover_agent:

"""Clear `deleted_at` on the agent_ownership row. Audit-logged.
...
The container is NOT started automatically — operator does that explicitly.
"""

src/backend/services/agent_service/lifecycle.py::start_agent_internal:

container = get_agent_container(agent_name)
if not container:
    raise HTTPException(status_code=404, detail="Agent not found")

DELETE /api/agents/{name} removes the container (the workspace volume survives). So after any soft delete, POST /api/agents/{name}/start always 404s, and there is no other endpoint that creates a container for an existing agent (recreate_container_with_updated_config requires an old_container to copy config from).

create_agent_internal cannot fill the gap either — it 409s on a reserved name, and is_agent_name_reserved deliberately matches soft-deleted rows (crud.py:184-189).

Net effect: a recovered agent is visible in the API and UI, holds all its child rows, and can never be started.

Reproduction

  1. DELETE /api/agents/<name> (soft delete; container removed, agent-<name>-workspace volume retained)
  2. POST /api/admin/soft-deleted/agents/<name>/recover → 200, deleted_at cleared
  3. POST /api/agents/<name>/start → 404 Agent not found

Manual workaround (what recovery should be doing)

Deleting the agent_ownership row and re-creating the agent with the same name and template restores it completely: crud.py:805 calls volume_get before volume_create, so the existing agent-<name>-workspace is reattached untouched, and startup.sh skips the clone because .git already exists on the volume ("Repository already exists on persistent volume - skipping clone"). Child rows key off the agent_name string, so Telegram bindings, chat sessions, and executions reattach on their own. This proves the data model supports a clean rebuild — only the API surface is missing.

Acceptance criteria

  • Recovery restores a startable agent: either recover recreates the container, or start recreates a missing container from persisted config, or a dedicated POST /api/agents/{name}/recreate-container exists
  • The existing workspace volume is reused, never recreated (no data loss on recovery)
  • The rebuilt container carries the full security posture (cap-drop, no-new-privileges, noexec tmpfs, derived TRINITY_AGENT_AUTH_TOKEN) — i.e. it goes through the supported creation path, not a hand-rolled docker run
  • Subscription binding, resource limits, access policy, and other agent_ownership settings survive
  • recover_agent's needs_container_recreate flag is either honored by a real code path or removed
  • Regression test: soft-delete → recover → start → agent reaches running and /api/agents/{name}/stats returns 200

Activity

  1. added
    complexity-mediumComplexity: medium (board points 5-8)
    status-readyGreenlit and ready for development (vetted; counterpart to status-incubating)
    on Jul 9, 2026
  2. added and removed
    status-readyGreenlit and ready for development (vetted; counterpart to status-incubating)
    on Jul 10, 2026
  3. self-assigned this
    on Jul 10, 2026
  4. dolho commented on Jul 10, 2026

    @dolho
    Contributor

    Fix in #1567 — start now rebuilds the missing container for a recovered (live) agent from persisted config + the surviving workspace volume's template.yaml, reusing the volume and going through the same secured provision/run path as a normal recreate (no hand-rolled docker run). A nonexistent agent still 404s. Unit-tested the wiring + reconstruction; the full running-container assertion needs a real container and will be verified against a live stack.

  5. added
    status-in-devMerged to dev, awaiting release cut to main
    and removed on Jul 14, 2026
  6. removed
    status-in-devMerged to dev, awaiting release cut to main
    on Aug 7, 2026
  7. added a commit that references this issue on Oct 13, 2026
  8. added a commit that references this issue on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions