Faster papers first paint: paged catalog, HTML cache, cache-first HF Daily

[znsoftm]

Problem

After the catalog gzip/SWR pass, cold first paint on papers.maclaw.top still waited on a ~1.5MB lean snapshot (~495KB gzip on disk). HF Daily list GETs could block on Hugging Face fetch + catalog merge and 503 through the tunnel. /papers HTML was Cache-Control: no-store.

What changed

Stacked on cursor/papers-catalog-cache-6508.

1. Smaller first catalog payload

   • Card snapshot: abstracts truncated to 280 EN / 140 ZH (what cards show). Drops brief, query_hits, pdf_url, doi, filename, and unused metadata; omitempty on empty review/local flags; authors capped at 8.
   • GET /papers/api/catalog?offset=0&limit=25 returns the first page (partial, next_offset, count = full size). Client paints those cards, then merges the tail (loadCatalogTail). Search/filter work as soon as the tail lands. Full GET /papers/api/catalog remains for compatibility.
   • localStorage key bumped to rs_papers_catalog_v2.

2. HTML cache

   • GET /papers uses ETag + Cache-Control: public, max-age=0, stale-while-revalidate=300 + gzip. max-age=0 so revalidation still hits the server and visit bumps still run. Settings HTML stays no-store.

3. Cheaper progress

   • GET /papers/api/progress uses the in-memory snapshot + live job overlay. It no longer rebuilds/overlays the full catalog (no per-paper zh/dual stat) on every 4s/30s poll.

4. HF Daily (ported onto this stack + cache-first)

   • This tip did not have Daily; the public site does. Handlers + UI are included so Daily and review cross-links keep working.
   • GET /papers/daily/{date} is cache-first: a local day file is returned immediately; stale days refresh in the background. Missing cache may do one short fetch; failures return an empty 200, not tunnel 503.
   • Day/dates JSON: gzip + public SWR + ETag/304. can_manage stays on progress. Client SWR via rs_papers_daily_v1. Day GET no longer calls catalog() or mergeIntoCatalog on the request path.

Relay DisableCompression from the previous PR is unchanged.

Before / after (measured)

80-paper unique-abstract fixture (same generator as PR 26):

payload	bytes	notes
full abstracts (old snapshot shape)	232,251	100%
previous lean (400/220 + brief)	~124,756	PR 26
this lean (280/140, no brief/unused)	69,641	30% of full
previous lean gzip	~24,514	PR 26
this lean gzip	16,395	7% of full
first page gzip (limit=25)	5,798	35% of this lean gzip

Local curl (80-paper fixture, 127.0.0.1:18765):

request	bytes	headers
GET /papers	99,628	previously no-store
same + gzip	22,776	stale-while-revalidate, ETag
GET /papers/api/catalog gzip	1,740	public SWR
GET /papers/api/catalog?offset=0&limit=25 gzip	890	partial=true, next_offset=25, count=80
GET /papers/api/progress	227	still no-store
GET /papers/daily/dates gzip	146	public SWR

Scaled to the public ~1.5MB / ~495KB-gzip catalog: first paint is a ~6–20KB gzip first page instead of the whole snapshot. Repeat visits: HTML 304, catalog 304 or localStorage, Daily last-day localStorage.

HF Daily stale-cache test: GET returns the local day in <2s while upstream is blocked; from_cache + stale + SWR headers.

Public catalog stays far under the ~2MiB tunnel safety margin. Progress remains no-store and small.

Tests

• go test ./...
• First-page gzip smaller than full lean; page shape partial / next_offset / count
• HTML gzip + 304; no no-store on catalog/HTML GETs
• Daily cache-first (stale GET does not block); Daily gzip + 304
• Existing Daily ingest / translate / review / trend / UI string tests

Operator notes

No nginx change required. Relay must keep DisableCompression: true (already on the catalog-cache tip) so gzip Content-Encoding survives the tunnel.

Snapshot files are still auto-written (catalog-snapshot.json + .gz). Daily caches stay under $PAPERS_DIR/hf-daily/ (gitignored).

Do not merge onto main until the papers stack under cursor/papers-catalog-cache-6508 lands.