Skip to content

Sync health observability (S1) #389

Description

@vybe

Summary

Add sync health tracking, dashboard indicators, and auto-sync to catch silent desync before it accumulates.

Context

Problem P1: An agent can run for weeks with failing syncs while appearing "healthy" in the dashboard. No sync-health signal exposed at fleet level, and no autonomous sync cadence.

Problem P6: The ahead/behind count compares against `origin/main`, not the working branch. External writes to the working branch are invisible.

Acceptance Criteria

Database: New agent_sync_state table

  • `last_sync_at`, `last_sync_status` (success/failed)
  • `consecutive_failures` counter
  • `last_error_summary`
  • `last_remote_sha_main`, `last_remote_sha_working`

Dashboard indicators

  • Each agent row shows sync-health dot:
    • Green: <24h since sync, no failures
    • Yellow: 24h–7d since sync
    • Red: >7d or last attempt failed or `behind_working > 0`

Operator queue alerts

  • If `consecutive_failures >= 3`, emit `sync_failing` entry to operator queue
  • Surfaced in Operating Room

Fix P6: Dual ahead/behind

  • `/api/git/status` returns both tuples:
    • `ahead_main` / `behind_main` (template improvements)
    • `ahead_working` / `behind_working` (working branch divergence)
  • Health indicator keys off `behind_working`

Auto-sync (S1a)

  • Per-agent flag `git_auto_sync_enabled` (default true for GitHub-template agents)
  • Agent server runs lightweight sync heartbeat every 15 min
  • Guarded by `GIT_SYNC_ENABLED` env var

Config flag

  • `freeze_schedules_if_sync_failing` (default false, opt-in)
  • If enabled, pauses schedules when sync health is red

Technical Notes

Files to modify:

  • `src/backend/db/schema.py`, `db/migrations.py` — new table
  • `src/backend/services/monitoring_service.py` — git sync checks
  • `docker/base-image/agent_server/routers/git.py` — dual ahead/behind, auto-sync loop
  • `src/frontend/src/views/Dashboard.vue` — sync health dots

Part of Epic #381

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions