Repository navigation
#144 Phase 2: retention is a rule, not a hope - #149
Merged
Merged
Conversation
|
Warning Review limit reached
This review includes 13 billable files and costs up to $3.25.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 41 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Your 68 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (13)
Comment |
…nds`, a cleanup that exempted the current display root from the age cutoff so a displayed run lived until it was superseded, and a run deletion that cascaded to its results and left the provider's payload in `source_records` behind it. A source's terms — the Guardian's "not retained longer than 24 hours" (#142), Artsy's "removed on termination" — could be kept by a sentence in a document and by nothing in the code. Two clocks, not one `retention_seconds` and `failure_backoff_seconds` join `Policy.@keys`, so both are per source and both derive `<SLUG>_DISCOVERY_*` from the same registry the refresh keys do. `config/runtime.exs` now reads one `{key, suffix}` list three times — the allowlist, the global defaults, the per-source overrides — so a key added to `Policy` cannot end up settable in one place and ignored in another. Refresh is how long an answer may be reused; retention is how long it may be held, and they were the same number. `expires_at` is real `completed_at + retention`, written at publication and read by the sweep. It was written as `now` and read by nothing: measured on `devils_dictionary_v2`, **180 of 180** succeeded runs were "expired" at the moment they completed. It is written once and never updated because a completed run is immutable in the database, and rows written before this field meant anything fall back to `completed_at +` the source's current policy — otherwise the first tick after this ships would withdraw every shelf at once. `cleanup/0` is four things, in order Recovery; **retention**; the bounded sweep; the ledger. Retention withdraws a run past its deadline (`display_allowed = false`, the branch that existed and nothing could reach), deletes its results, and deletes the `source_records` those results owned — including when the run is the page's current display root. The run row stays: it is the ledger of what was spent. The bounded sweep now purges results and records before deleting runs, for the same reason. And `discovery_request_attempts` is pruned to `retained_attempts_per_position` per position, including for protected roots, which is the only path that ever reaches them — an attempt row went only when its run did, and a protected run is never deleted. What retention does not take: a `source_record` is deleted only when nothing else stands on it — no other result, and no `external_identifiers` row, `sense_revision` or other reference to it or its revisions. Deleting one would cascade its revisions and silently null the provenance of something durable. The consequence, stated in the README: a source whose results resolve to durable identities cannot promise a short hold, and the providers that *do* have a retention obligation are exactly the ones with no `identity_record/1`. The page asks again `fresh_run/3` filters `display_allowed`, so a withdrawn root inside its refresh window is no longer a cache hit — without this a page would show the honest empty and never ask again for up to thirty days. And a run withdrawn by retention reads as `:expired` rather than `:withdrawn`: its results are gone, so there is nothing withheld to describe. A withdrawn *result* is a different thing and still says `:withdrawn`. The reader is told how old the shelf is `Discovery.state/2` carries `fetched_at` and `refresh_after` beside the `refresh_due` that was computed for every shelf since #109 and read by one test. The shelf header says *Fetched 3 days ago · refreshes on your next visit after 21 Oct.*, or *refreshing on this visit* when the render drawing it is the one going again, and a corpus says *Held since 17 Sep 2026* from its manifest's own `generated_at`, read at compile time with `@external_resource`. One line for the shelf and not one per source, and in the right-hand column where the other sentences are rather than the 80 px column of names and counts — the `/ui` guidelines refuse `text-xs` for body text, and an 80 px column cannot hold a sentence at `text-base`. Same container and classes as the status line beside it, which is already proven at 375 px. Measured, read-only, against `devils_dictionary_v2` 413 attempt rows, of which **169** are beyond the 3-per-position bound and had no path to deletion at all. 29 CineGraph runs past the default seven days, holding 190 results, every one of them protected by the old sweep. Nothing was written: the destructive first tick is the owner's to run. 1,571 tests, 0 failures. `mix precommit` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
holden
force-pushed
the
claude/144-kit-phase2
branch
from
September 22, 2026 10:10
561ba87 to
499dd2e
Compare
Two branches built per-source retention against the same main: #145 shipped a narrow sweep for the Guardian's 24-hour clause, and this phase the general one — `retention_seconds` a policy key, `expires_at` real, withdraw-and-purge with the encyclopedia's references checked, the ledger pruned. Rebasing the phase over the merge kept the general mechanism and retired the narrow one, and this commit is what that cost around it. `Policy.@keys` named `:retention_seconds` twice, once from each branch, and `for!/1`'s missing-defaults check subtracts one occurrence — so every policy read raised "missing defaults" for a key that was there. The Guardian's four retention tests now assert Phase 2's shape: the run is withdrawn rather than deleted and its result and source record go; `expires_at` is written from the source's own window, as `publish` writes it, because that is what the sweep reads; and Bing's comparison run sits inside the suite's two-hour shared window, since each source is now judged against its own. `Policy.source_retentions/0` is gone with the sweep that read it; the tests read `retention_seconds/1`. mix precommit: 1,691 tests, 0 failures. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
holden
force-pushed
the
claude/144-kit-phase2
branch
from
September 22, 2026 10:10
499dd2e to
a8c23be
Compare
16 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of #144, stacked on #148 (Phase 1), which is stacked on #147 (Phase 0). Review in that order; this branch contains both.
The one-line verdict the issue opens with was right: refresh was per-provider and automatic; retention was neither.
The acceptance, measured
cleanup/0including its source recordretention_test.exs, at the suite's own scale (one hour against a half-hour policy, becauseconfig/test.exs's default is two hours). The run is the page's display root, which is the case the old sweep exemptedexternal_identifiersrow on the record's revision keeps it; the result still goes:expired, empty items, and a new run admitted — see the cache-hit bug belowPlus:
expires_atiscompleted_at + retentionand is what the sweep reads; attempts are pruned per position for protected roots;failure_backoff_secondsis per source.mix precommitclean: 1,571 tests, 0 failures.The ledger, before and after — measured read-only on
devils_dictionary_v2discovery_runsdiscovery_resultsdiscovery_request_attemptsexpires_at(<= completed_at)Nothing was written. The first real tick is destructive against your dev data — it withdraws shelves and deletes payloads — so I measured it with
Discovery.retention_due/1and a dry-run count and left it for you. Say the word and I'll run it, or the 15-minute cron will once this merges.Two bugs this found on the way
fresh_run/3did not filterdisplay_allowed, so a root withdrawn by retention would still answer{:cached, run}for the rest of its refresh window — up to thirty days of a page showing the honest empty and never asking again. Fixed, and the re-queue is asserted.:withdrawnwas the wrong word for it. That status is for a result withheld by policy, which has a reason to show. Retention deletes the results first, so there is nothing withheld; the run now reads:expired, which is what the reader already renders as will retry when available.The decision worth your attention
A source record the encyclopedia is standing on survives its result. A record is deleted only when nothing references it or any of its revisions — no
external_identifiersrow, nosense_revision, no corpus provenance. Deleting one cascades its revisions and silently nulls the provenance of something durable.The consequence, and it is a real constraint rather than an implementation detail: a source whose results resolve to durable identities cannot promise a short hold. CineGraph, the Met, Commons, Open Library and PoetryDB all have
identity_record/1. The providers that do carry a retention obligation — the news ones — are exactly the ones without it, so the rule lands where #142 needs it. This is stated in the README's new Freshness and retention section rather than left as a surprise for whoever writes the Guardian.The reader
Fetched 3 days ago · refreshes on your next visit after 21 Oct.— orrefreshing on this visitwhen the render drawing it is the one going again, sinceDiscovery.request/3is called by that same connected render and promising a date that has passed would be the page lying about its own machinery. A corpus saysHeld since 17 Sep 2026, from the committed manifest's owngenerated_at, read at compile time with@external_resource.I ran
/uibefore touching the markup, and it changed the placement: the guidelines refusetext-xsfor body text and requiretext-baseon mobile, which an 80 px column cannot hold. So the line went into the right-hand column with the other sentences, attext-base text-mist-500 sm:text-sm— the same container and classes as the status line already beside it.No screenshots this phase. The one dev server on 4007 is on
devils_dictionary_v2withmain's code, and a second Oban node on that database would have it executing my queued runs with its own code — the exact hazardadding-a-provider.md§6 describes. Phase 3 owes a browser proof at 1280 and 375 px and the freshness line will be in it, on the real page rather than a static mock. Until then the claim is what the component test asserts, and the no-horizontal-scroll claim rests on reusing an element already proven at 375 px rather than on a measurement I did not take.Shared files touched
lib/devils_dictionary/discovery.ex,discovery/policy.ex,artworks/corpus/manifest.ex,config/runtime.exs,lib/devils_dictionary_web/components/culture.ex,lib/devils_dictionary_web/live/word_live.ex,docs/discovery/README.md(a new Freshness and retention section, and promise 6 now carries retention).One test to watch, and it is not mine
test/devils_dictionary/lexicon/word_page_test.exs:324fails intermittently in a full run —quoll's cards come back["wordnet", "poetrydb"]— and passes in isolation every time, on this branch and onmainbefore any of #144. Its own comment says why: it makes an exact claim about which cards a page has, and an@tag :unboxedtest elsewhere commits rows that reach it. Order-dependent, pre-existing, and worth its own fix rather than a silent re-run.Next phase's assumptions
Phase 3 holds. The Met's
min_retry_interval_ms: 1_500is still the one hard-coded pacing key andinterval/3from Phase 1 is waiting for it; CineGraph and GIPHY still declare no pacing;WordLivestill namesGiphy.browser_config/1andCulturestill rendersGiphyShelfby name. Artsy is still registry-only and still passes conformance only because it declaresbackground: false.🤖 Generated with Claude Code