Skip to content

Add per-lane hang timeouts for stuck paper PDF translations - #22

Open
znsoftm wants to merge 3 commits into
cursor/papers-html-syntax-f61efrom
cursor/papers-translate-timeout-1f96
Open

znsoftm wants to merge 3 commits into
cursor/papers-html-syntax-f61efrom
cursor/papers-translate-timeout-1f96

Conversation

@znsoftm

@znsoftm znsoftm commented Sep 15, 2026

Copy link
Copy Markdown
Member

A stuck BabelDOC/Google job can occupy the only translate slot for hours (production: 2609.15818 sat ~6h on HTTP 429 with 「166 排队 / 1 翻译中」). This adds a queue watchdog so a hang cannot block the dual-lane pool forever.

Behavior

  • While a job is running, wall-clock is measured from persisted started_at.
  • Fast lane default 45 minutes; slow lane default 180 minutes (3 hours).
  • On timeout: kill the translate_worker / BabelDOC process tree (pidfile + cmdline + descendants), free the in-memory slot immediately, increment persistent timeout_count.
  • If timeout_count < max: requeue at the end of the same lane (so other work proceeds) with error timeout (attempt N/3); requeued.
  • If timeout_count >= max (default 3): status skipped, error timed out 3 times; permanently skipped. Auto-translate and translate-all will not pick it up again.
  • Manual POST /papers/translate for that id resets timeout_count so an admin can retry.
  • Applies to in-process runOne jobs and OS workers adopted by reconcileExternal.

Env knobs

Variable Default Format
PAPERS_TRANSLATE_FAST_TIMEOUT 45m Go duration (45m) or integer minutes
PAPERS_TRANSLATE_SLOW_TIMEOUT 3h same
PAPERS_TRANSLATE_MAX_TIMEOUTS 3 positive integer

Google QPS and default concurrency are unchanged.

Tests

go test ./internal/api/ covers duration/env parsing, requeue + timeout_count, permanent skip, and leaving in-window jobs alone.

Open in Web Open in Cursor 

cursoragent and others added 3 commits September 15, 2026 22:48
… jobs.

A watchdog on the translate loop kills the BabelDOC/worker tree when a
running job exceeds 45m (fast) or 3h (slow), then requeues it at the end
of the same lane or skips it after three strikes.

Co-authored-by: Daniel <znsoft@163.com>
…not rewrite status during TempDir cleanup.

Co-authored-by: Daniel <znsoft@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants