Skip to content

Restore stays "ongoing" forever, with no log, when it fails before the restore command (download, decryption, archive preparation) #135

Open

Description

@adecid33

Describe the bug
In execute_restore (src/services/restore/executor.rs), the download and the archive preparation use ?:

let downloaded = self.download_backup(...).await?;
let backup_file = self.prepare_archive(...).await?;

If one of them fails, the function returns before send_result is called. The caller (src/services/restore/dispatcher.rs) only logs the error in the agent container (error!("Restore failed: {}", e)); nothing is sent to the server. The same applies to TempDir::new()?.

As a result:

  • the restoration stays ongoing in the dashboard forever, with no log at all;
  • the server then refuses any new restore of this database: A restoration is already waiting or ongoing for this database (409), and there seems to be no way to unblock it from the dashboard.

run_restore itself is fine: errors from pg_restore / sqlpackage are turned into a failed result with their logs.

To Reproduce

  1. Start a restore from a backup whose file cannot be downloaded or decrypted (for example storage unreachable from the agent, or a wrong master key).
  2. The agent container logs Restore failed: ....
  3. The restoration stays ongoing in the dashboard, without logs, and a new restore of the same database returns 409.

Expected behavior
Any error during the restore job is reported to the server with status failed and the logs collected so far (Download failed: ... is already logged by download_backup), so the dashboard shows the cause and a new restore can be started.

Suggested fix
In execute_restore, catch the errors of the steps before run_restore and send a failed result instead of returning early, as execute_restic_restore already does with its outcome match.

Environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions