state: make jobs modify_index index non-unique

[afreidah]

What

The jobs-table modify_index memdb index was declared Unique: true, but ModifyIndex is not unique across jobs. When a single Raft transaction writes several jobs at once (e.g. rescheduling allocations after a node goes down), those jobs share a ModifyIndex. A unique memdb index can only hold one object per key, so colliding jobs were silently dropped from any query that iterates this index — notably Job.Statuses, which backs the /v1/jobs/statuses endpoint and the UI jobs page — while remaining visible via the unique id index used by /v1/jobs and the CLI.

This change marks the index non-unique and adds a regression test.

Fixes #28132.

Reproduction (on a live cluster)

Jobs sharing a ModifyIndex (one Raft txn) vanish from statuses but not /v1/jobs, even on a single unpaginated page:

$ curl -s "$NOMAD_ADDR/v1/jobs?namespace=*" | jq -r '.[].ModifyIndex' | sort | uniq -d
2019281
2020598

$ curl -s "$NOMAD_ADDR/v1/jobs/statuses?namespace=*&per_page=200" | jq length
52
$ curl -s "$NOMAD_ADDR/v1/jobs?namespace=*&per_page=200" | jq length
60

The 8 missing jobs are exactly the ones sharing ModifyIndex 2020598 (7 jobs) and 2019281 (1 of the pair).

A self-contained Docker A/B harness that reproduces this and verifies the fix (1 server + 1 client, no Consul) is here: https://github.com/afreidah/nomad/tree/repro-jobs-statuses-28132/repro-28132

Root cause

nomad/state/schema.go, jobTableSchema() — the modify_index index used Unique: true on a non-unique field. Job.Statuses iterates it via JobsByModifyIndex (getSorted(txn, sort, "jobs", "modify_index")); /v1/jobs iterates the unique id index and is unaffected. The index was born Unique: true in #20130 (May 2024) and never changed, so this has been latent since the statuses endpoint shipped.

Testing

TestStateStore_JobsByModifyIndex_SharedModifyIndex asserts every job sharing a ModifyIndex is returned. It fails before the fix (only the last writer survives) and passes after:

go test ./nomad/state/ -run TestStateStore_JobsByModifyIndex_SharedModifyIndex -v

AI usage

I found this on my own cluster and personally dug through my logs/metrics/etc to try and figure it out and used an AI assistant to help dig through the state/paginator source to quickly find the relevant parts of the code I wanted to look at based on my analysis and theories until I was confident I had reached the root cause after chasing a few red herrings and had what I needed to setup the reproduction + test to prove it existed and what condition would cause it. I Manually verified the issue against my live v2.0.3 cluster with nomad cli and curl commands because the issue is actively persisting until I re-deploy those jobs. No AI assistance is relevant in the actual go source changes since the entire fix is flipping a bool on one line. I 100% stand behind everything in this PR as the result of personal analysis and debugging on my real homelab cluster.

[hashicorp-cla-app]

All committers have signed the CLA.

[afreidah]

glad I went back and read @gulducat 's original PR for this....made me realize that I missed something and while this is still an improvement, it exposes a previously hidden pagination issue on the jobs page.

ModifyIndexTokenizer builds its cursor from ModifyIndex alone, with no unique identifier involved (since previously the ModifyIndex WAS the unique identifier) — fine when the index was unique, but now jobs sharing a ModifyIndex can span a page boundary so jobs at the cursor boundary get returned on two pages, and if more jobs share a ModifyIndex than fit in one page, the cursor never advances past that group, so a paginating client can't reach anything older.

I updated my a/b reproduction harness to also show what pagination does with the change in this PR:
https://github.com/afreidah/nomad/tree/repro-jobs-statuses-28132/repro-28132

output (condensed since the gh issue has the full output there prior to this pagination update):

afreidah ~/tools/nomad on  repro-jobs-statuses-28132* ❯ ./repro-28132/ab.sh
...
==> 0. Binary under test
    source : /tmp/nomad-hashicorp-main
...

    /v1/jobs            = 20   (authoritative: CLI, topology, scheduler)
    /v1/jobs/statuses   = 5   (what the web UI jobs page renders)
...

==> 7. Full pagination walk (next_token cursor, like the web UI)
    paging /v1/jobs/statuses with per_page=7, following next_token to the end:

    page  1:  5 job(s)   next_token=<none>

    walk ended after 1 page(s): server returned no next_token.
    duplicated: none
    never returned (in /v1/jobs but no page showed them): j02 j03 j04 j05 j06 j07 j09 j10 j11 j13 j14 j15 j16 j19 j20
...

========================================================================
...
==> 0. Binary under test
source : /tmp/nomad-fork-main

==> 6. Result
...
    /v1/jobs            = 20   (authoritative: CLI, topology, scheduler)
    /v1/jobs/statuses   = 20   (what the web UI jobs page renders)

    FIXED: jobs share a ModifyIndex (75 76 77 78 ) yet ALL 20 appear in
    /v1/jobs/statuses — nothing dropped. The non-unique index retains them.

==> 7. Full pagination walk (next_token cursor, like the web UI)
    paging /v1/jobs/statuses with per_page=6, following next_token to the end:

    page  1:  6 job(s)   next_token=77
    page  2:  6 job(s)   next_token=76
    page  3:  6 job(s)   next_token=76

    walk could not finish: page 3 returned the same next_token (76) as the one requested — it repeats forever.
    duplicated (returned on more than one page): j03 j04 j05 j20
    never returned (in /v1/jobs but no page showed them): j09 j10 j11 j12 j13 j14
...

The fix for that is slightly more involved so I haven't pushed anything for that to keep this PR narrow and a follow-up gh issue could be filed. I'm starting to look into it because it is fun, but I'll leave just the narrowly scoped PR for now.

[gulducat]

Thank you @afreidah!

This is really great work: enthusiastic and shrewd troubleshooting with some history-spelunking and clear communication. I would be pleased as punch to receive more from you on the extended full-page-spanning singular-ModifyIndex issue, if you have time and interest to pursue it. Just file an issue if a PR will take a bit of time to put together, to be sure we don't lose track of it!

As is often the case, a lot of effort went into a simple one-line fix, nice and easy to evaluate. I only have the one nit-picky ask to move the test into the state_store_test hive, then we'll get this merged and backported.

[gulducat]

As much as I hate to say "go add more code to a 10,000 line file", state_store_test.go currently houses all the tests for methods like this one, so I think we should put this little fella with its family.

There's a chunk of them called TestStateStore_JobsBy*; maybe put this one after those, under TestStateStore_JobsByGC.

[afreidah]

awesome! I moved the test into that file and re-pushed.

I'll drop a gh issue for the extended full-page-spanning singular-ModifyIndex issue that I have started looking into as well.

Thanks!

[afreidah]

@gulducat not sure if those two tests are regularly flaky but I'm pretty sure it isn't anything related to this PR. Looks like they are failing in other CI runs too, but if there is anything I need to do on my end just let me know!

[gulducat]

Looks great, thanks!

I verified that the failing tests were unrelated; the fix has been merged, and I'll make sure this commit lands happy in main before backporting :)