fix(schedule): allow concurrent pinned startups without lock-contention skips

[vishr]

Summary

Pinned scheduled jobs unnecessarily take the application-wide schedule.lock exclusively during startup and use non-blocking acquisition. When systemd activates multiple pinned jobs together, one establishes its release lease and every other job immediately records a skipped firing.

This is observable production data loss/staleness for high-frequency ingestion schedules, not a stuck lock or workload release failure.

Production evidence

Observed on Monk with Onebox v2026.9.1:

• 13:46:09Z: sync-news-live succeeded; sync-news-backfill skipped.
• 13:47:02Z: sync-news-live succeeded; sync-nws-alerts skipped.
• 13:48:00Z: sync-sec-form4 succeeded; sync-news-live and sync-news-backfill skipped.
• 13:50:07Z: analytics-cleanup succeeded; CFTC, SEC submissions, live news, and backfill skipped.

The losing records consistently report an application operation is taking its lock. Successful runs occur between collisions, proving the lock is not permanently stuck.

At 2026-09-11T14:01:48Z, all workloads, services, and the proxy were individually non-divergent. Cipher was running on the recorded release. Only scheduled-job skip streaks made the aggregate status divergent. Examples included:

• SEC submissions: 7 consecutive skips
• Treasury auctions: 6
• NWS alerts: 5
• News backfill: 3

Repeated ob exec inspections can add durable application-lock windows, but they are not the root cause: the pattern existed for hours and follows coalesced timer activations.

Root cause

scheduleLockLines currently emits:

exec 8>/var/lib/ob/<app>/schedule.lock
/usr/bin/flock --exclusive --nonblock 8 || skip 'an application operation is taking its lock'

The same exclusive mode is used for both deployment policies:

• exclusive jobs correctly need an exclusive lock for their whole run.
• pinned jobs need only rendezvous with application writers while resolving current and establishing a shared release lease. Different pinned jobs are otherwise explicitly allowed to run concurrently and already have distinct per-job locks.

The rendezvous is therefore naturally a reader/writer lock:

• pinned startup: shared reader
• application operation: exclusive writer
• exclusive scheduled job: exclusive writer

There is a second amplifier: generated job timers omit AccuracySec=. Systemd defaults it to one minute and intentionally coalesces local timers. This defeats deliberate minute-level staggering; an every-minute job collides with every other schedule that fires in that minute.

A bounded wait alone is not a complete fix. It continues to serialize every pinned startup and can exhaust the wait budget when a batch arrives together or a durable startup performs additional preparation.

Proposed fix

1. Derive the application-wide rendezvous mode from deploy_lock:
   • pinned: flock --shared
   • exclusive: flock --exclusive
2. Keep the same-job lock exclusive and non-blocking. A later firing of the same running job should still record a skip.
3. Add a short bounded wait (about 10 seconds) to the application-wide rendezvous for both scheduler and application-operation acquisition. This handles brief reader/writer handoffs; it must not become an unbounded wait.
4. Continue to skip after observing an already-established, fresh durable application lock. An ob exec or deploy may last minutes and should retain its existing exclusion semantics.
5. Use --conflict-exit-code and classify only contention/timeout as a skip. Other flock failures must fail visibly instead of being recorded as benign timing.
6. Emit AccuracySec=1s in generated scheduled-job timers so five-field cron staggering is not coalesced across the default one-minute accuracy window.
7. Reconcile documentation describing the scheduling mutex and skip reasons.

Status wording

Keep repeated skip streaks visible as divergence: missed scheduled work is a real operational issue.

However, the aggregate divergence_detected safe message currently says the live release does not match the recorded release state. That is incorrect when only schedules are degraded. Change it to describe aggregate application-state divergence rather than release mismatch.

Acceptance criteria

• Two different pinned jobs can establish leases and proceed concurrently.
• A pinned startup cannot cross an application operation's exclusive rendezvous.
• An exclusive scheduled job still excludes pinned jobs, other exclusive jobs, and application operations for its required scope.
• A short-lived rendezvous holder is tolerated within the bounded wait.
• Wait expiry records a skip with an accurate reason.
• A non-contention flock error fails the unit and is not recorded as a skip.
• The same job cannot overlap itself and its later activation is still recorded as skipped.
• Generated job timers contain AccuracySec=1s.
• A snapshot with only schedule issues remains divergent, but its public error does not claim release mismatch.
• ob schedule apply installs the corrected runner and timer units without requiring an application release deployment.

Test gap

The current suite validates generated lock strings and a single pinned runner's lease lifecycle, but does not exercise two different pinned runners starting concurrently. Add a Linux concurrency test covering shared pinned readers and an exclusive writer, plus generated-unit assertions for the bounded wait and timer accuracy.

Baseline before the change: go test ./... passes 2,019 tests across 22 packages.