Repository navigation
bug(recovery): soft-delete recovery dead-ends — recover tells you to start, but start 404s when the container is gone #1559
Description
Summary
Admin soft-delete recovery (#834 Phase 1c) can restore the agent_ownership row but cannot bring the agent back. recover_agent explicitly defers container creation to the operator, and the endpoint it defers to refuses to run without an existing container — which never exists after a delete.
Evidence
src/backend/routers/admin_recovery.py::recover_agent:
"""Clear `deleted_at` on the agent_ownership row. Audit-logged.
...
The container is NOT started automatically — operator does that explicitly.
"""src/backend/services/agent_service/lifecycle.py::start_agent_internal:
container = get_agent_container(agent_name)
if not container:
raise HTTPException(status_code=404, detail="Agent not found")DELETE /api/agents/{name} removes the container (the workspace volume survives). So after any soft delete, POST /api/agents/{name}/start always 404s, and there is no other endpoint that creates a container for an existing agent (recreate_container_with_updated_config requires an old_container to copy config from).
create_agent_internal cannot fill the gap either — it 409s on a reserved name, and is_agent_name_reserved deliberately matches soft-deleted rows (crud.py:184-189).
Net effect: a recovered agent is visible in the API and UI, holds all its child rows, and can never be started.
Reproduction
DELETE /api/agents/<name>(soft delete; container removed,agent-<name>-workspacevolume retained)POST /api/admin/soft-deleted/agents/<name>/recover→200,deleted_atclearedPOST /api/agents/<name>/start→ 404Agent not found
Manual workaround (what recovery should be doing)
Deleting the agent_ownership row and re-creating the agent with the same name and template restores it completely: crud.py:805 calls volume_get before volume_create, so the existing agent-<name>-workspace is reattached untouched, and startup.sh skips the clone because .git already exists on the volume ("Repository already exists on persistent volume - skipping clone"). Child rows key off the agent_name string, so Telegram bindings, chat sessions, and executions reattach on their own. This proves the data model supports a clean rebuild — only the API surface is missing.
Acceptance criteria
- Recovery restores a startable agent: either
recoverrecreates the container, orstartrecreates a missing container from persisted config, or a dedicatedPOST /api/agents/{name}/recreate-containerexists - The existing workspace volume is reused, never recreated (no data loss on recovery)
- The rebuilt container carries the full security posture (cap-drop,
no-new-privileges, noexec tmpfs, derivedTRINITY_AGENT_AUTH_TOKEN) — i.e. it goes through the supported creation path, not a hand-rolleddocker run - Subscription binding, resource limits, access policy, and other
agent_ownershipsettings survive -
recover_agent'sneeds_container_recreateflag is either honored by a real code path or removed - Regression test: soft-delete → recover → start → agent reaches
runningand/api/agents/{name}/statsreturns 200
Activity
dolho commentedon Jul 10, 2026
Fix in #1567 —
startnow rebuilds the missing container for a recovered (live) agent from persisted config + the surviving workspace volume'stemplate.yaml, reusing the volume and going through the same secured provision/run path as a normal recreate (no hand-rolleddocker run). A nonexistent agent still 404s. Unit-tested the wiring + reconstruction; the full running-container assertion needs a real container and will be verified against a live stack.test(readiness-probe): stub get_next_available_port after lifecycle i…
fix(recovery): rebuild missing container on start for recovered agents (
fix(volumes): rename-safe volume identity — sweep, purge, create and …
fix: never flow a raw exception message into fleet-restart results (C…
fix: fleet restart adopts rebuilt base images through the canonical l…
fix(workspace): roster shows container availability honestly — never …
fix(agents): make SSH-port allocation atomic so concurrent creates ca…
Release: v0.9.0 (#2268)
fix(agents): agent containers are born unless-stopped (#2541) (#2591)
Merge pull request #3632 from Abilityai/feature/3590-rebuild-pull-env
fix(pull): the Abilityai#1559 rebuild bakes the pull-worker env; one …