[philipmw]

This implements the idea in rustic-rs/rustic#1430 and the subsequent feedback in rustic-rs#438

To use this feature, I wrote a proof-of-concept warmup-s3-archives program: https://gitlab.com/philipmw/warmup-s3-archives

Changes:

• new terminology: "singular" vs "plural" IDs and packs
• add --warm-up-batch <N> parameter
• add variables to --warm-up-command parameter to support singular and plural IDs and paths
• add warmup_path() to the backend interface that provides the S3 key (or other key usable by the warmup command) instead of pack ID.

Tested:

• unit tests;
• invoking the warmup program with "%id", "%ids", "%pack", "%packs", and batch size of 2 for a total restore size of 3 packs, verifying that the warmup command is invoked either separately per ID/pack or with two IDs/packs for the first invocation and with one ID/pack for the second invocation.
• killing and restarting restore; this works as long as the warmup program is idempotent (which works for S3)
• having the warmup command exit with an error code; rustic aborts the restore and prints a correct error message
• running multiple restore operations for different packs in parallel. The warmup program ignores notifications for packs that it does not recognize, leaving them in the queue and letting another warmup instance process them.

Known limitations and my thoughts for improvement opportunities:

• setup is non-trivial, between the AWS infrastructure and the warmup program configuration. It requires some AWS experience and cold storage motivation. But IMO all the complexity is specific to the domain; none is incidental.
• rustic does not pass the backend credentials to the warmup program. The warmup program is responsible for finding credentials on its own. Probably the best solution, at least for AWS, is for both rustic and the warmup program to use a common AWS credential provider.
• rustic's progress bar does not reflect warmup progress within a batch; only progress of entire batches. There is no protocol for communicating progress from a single invocation of the warmup command.
• rustic's warmup parameters could use a refactor as we discover and clarify cold storage backup scenarios. The distinction between --warm-up-command and --warm-up-wait-command seems too subtle. --warm-up-wait is too inflexible (since cold storage backends' estimates are measured in hours) and can be avoided entirely.