Add FirecREST autoalloc backend (hq alloc add firecrest)

[rjanalik]

Motivation

The PBS and Slurm autoalloc backends run qsub/sbatch locally, so the HQ server has to run on the
cluster's login node. On clusters with wall-time limits on login nodes (e.g. CSCS Alps), a long-running
server is not possible there. FirecREST is a REST API in front
of Slurm, so submitting allocations through it lets the server run anywhere with HTTPS access to the API
(a VM, a K8s pod, ...). Workers still connect back to the server over TCP as usual.

What this adds

• New backend hq alloc add firecrest / hq alloc dry-run firecrest. It takes the usual shared queue
  options plus the FirecREST connection settings: --api-url, --system, --token-url and --client-id.
  • Client secret: --client-secret-env names an environment variable of the server that holds the
    secret (default HQ_FIRECREST_CLIENT_SECRET). The secret itself is never stored in the queue
    definition or the journal.
  • Cluster-side paths: --remote-hq-path, --remote-server-dir and --remote-workdir. They are used
    verbatim on the cluster, because the server makes no assumptions about the cluster's filesystem.
• Queue handler. It submits, checks status and cancels allocations through the FirecREST v2 API, using
  an OAuth2 client-credentials token.
  • The submit script comes from the existing Slurm script builder. FirecREST allocations are ordinary
    Slurm jobs, so workers start with --manager slurm.
  • The token is cached and shared between concurrent requests. If a request gets HTTP 401, the handler
    fetches a fresh token and retries once.
  • Status is polled with one job-list request per queue rather than one request per allocation, which
    keeps API rate-limit pressure low. Allocations missing from the list are queried individually.
• Journal restore. If one allocation queue can't be recreated on restore (e.g. the secret variable
  isn't set after a restart), restore no longer aborts. That queue is skipped with an error in the log,
  and the rest of the state is restored.
• New firecrest cargo feature, on by default. It pulls in reqwest with rustls. A server built
  without the feature reports a clear error for FirecREST queues.
• Docs in docs/deployment/allocation.md, including the network requirements (the server must be
  reachable from compute nodes), and a CHANGELOG entry.

Testing

• Rust unit tests for job state mapping and response parsing.
• Python integration tests (tests/autoalloc/test_firecrest.py) against an in-process mock of the token
  endpoint and the FirecREST v2 API (tests/autoalloc/mock/firecrest.py), with fault injection. They cover:
  • the submit script, and dry-run success, token errors and submit errors
  • the allocation lifecycle (queued, running, finished, failed) and cancellation on queue removal
  • batched status queries and the retry after a 401
  • journal restore with and without the secret variable, and that the secret never reaches the journal
• Manual end-to-end test against the real CSCS FirecREST API on Alps (daint): dry-run, then
  alloc add → a Slurm job submitted via FirecREST → a worker on a compute node connected and ran the
  task → alloc remove --force cancelled the allocation via the API.

[Kobzol]

I'm no longer very involved in the maintenance of HyperQueue, but I think that the right way forward is to define a general API for autoalloc backends, and let HQ use it with an externally provided backend. Most likely it would make sense to define a JSON REST API, as performance is not super critical here, and that is as general as it gets.

That would allow anyone to implement their own autoalloc backend and use it with HyperQueue, rather than it being maintained in-tree. You could still implement it e.g. in Rust, and in this case it would essentially just do a translation between HQ's autoalloc REST API and the FirecREST API. It would require running the wrapper binary together with HyperQueue on some TCP/IP port, but that shouldn't be a big issue, it can just run on a single node togethe with the HQ server.

[spirali]

Disclaimer: I had an offline conversation about this with @rjanalik few weeks ago.

I did not have time to dig in the code yet, but as far as I understand, firecrest is basically a kind of API over SLURM. We do not need to invent our own.

I am ok to maintaining it directly in HQ, I see it as an alternative to two other backends that we already have.

Of course, if there would be like 3-5 other competing APIs in the future, than it is probably time to propose a unifying one for HQ. But in the current situation, having Firecrest directly in HQ makes sense to me.

This brings an out-of-the-box solution for people who does not run server on login node and wants to use autoallocator.

[Kobzol]

Fair enough :)