Skip to content

Harden the encyclopedia backbone: durable word identities, correct source relationships, safe refresh, and ConceptNet integration #72

Closed

Description

@holden

Purpose: establish the encyclopedia backbone before adding layers

The backbone is the words themselves, their meanings, and the relationships between them, enriched by properly attributed dictionary, encyclopedic, and commonsense sources. Examples, quotations, media, votes, and curator bots must attach to this durable foundation. They must not compensate for missing or unreliable identity and relationship semantics.

Does the current infrastructure do this? Partially. It has the right broad architecture and substantial working imports, but it is not yet robust enough to promise correct relationships across source updates or safe long-lived attachments. ConceptNet is not integrated in the current source registry.

Audit grade: B− overall; B+ for static MVP-0 engineering; C for readiness to become a living, participatory encyclopedia. Continue with Phoenix/Postgres and the existing context boundaries. Insert a backbone-correctness milestone before permanent community data. Allow justified schema migrations; preserving exactly thirteen tables is not an acceptance criterion.

Audited commit: a2ef867, verified against main on 2026-09-07. Related: #69, #70, #71, #64, #65, #67.

What is already demonstrated

  • mix precommit: 487 tests, zero failures.
  • Animals: 36/36 graded checks pass, including the existing global parity scan.
  • Emotions: 34/34 graded checks pass with parity explicitly skipped; global parity was already run across all six sources.
  • 1,541,669 lexical rows; 195,368 marked enriched. Approximately 87.3% have neither a sense nor a lexical entry.
  • Six implemented sources: WordNet, Wiktionary/Kaikki, Wikidata, Wikipedia, Bierce, Johnson.
  • WordNet: 120,564 synsets, over one million resolved lexical edges.
  • Bierce: 997 entries; Johnson: 42,726 entries.
  • Animals: 25,402 scope members; Emotions: 809.
  • Local search p95: 65–92 ms in the two scorecard runs.
  • Pure source materializers, batched transactions, keyset pagination, source-specific definitions, and working provenance drawers are useful foundations.
  • Johnson without a domain migration and Emotions from scope data are real extensibility proofs.

These figures prove substantial static assembly, not universal English coverage, semantic correctness, or recovery from source changes.

The product's culturally relevant examples are still sparse: nepotism has Johnson/Bierce/WordNet but no linked primary concept; performative and situationship have no definition cards. The sample evidence wall is not a shipped contribution system.

Required backbone contract

  1. Lossless identities: distinguish spelling, language, part of speech, source-specific meaning, and real-world referent. URL normalization must not assert equivalence.
  2. Typed relationships: preserve direction, endpoint identity, source relation type, scope of assertion, and relevant qualifiers. Similarity, synonymy, broader/narrower, part/whole, instance/class, and commonsense association are not interchangeable.
  3. Evidence for every assertion: retain source-native IDs, dataset/version, supporting record, relevant attribution and rights metadata, and derivation method.
  4. Safe change over time: reconcile additions, corrections, reordering, redirects, merges, and withdrawals. Existing examples and quotes must never silently move to a different meaning.
  5. Explicit uncertainty: distinguish polysemy, unresolved matching, conflicting claims, and human decisions. Heuristic weights are not calibrated probabilities.
  6. Deterministic recovery: document all required inputs and prove an offline reconstruction in a separate database.
  7. Stable attachment targets: examples may illustrate a word, a particular meaning, or a concept; the illustrating object is separate from the subject and from the submitted claim.

Words remain the reader-facing backbone. Concepts complement words where an encyclopedia needs to distinguish referents; they must not replace lexical identity or force every word through a Wikidata QID.

Audit findings

P1 — Source refresh leaves obsolete assertions active

The materializer upserts current outputs but does not reconcile outputs that disappeared. The linker likewise leaves unsupported old links. Shared lexeme fields use fill-empty/merge policies that are inadequate for corrections.

A rolled-back test using the real Wiktionary adapter replaced two meanings with one, replaced a form, and removed a hypernym and QID. Two senses remained, the old hypernym/link/form remained, and parity reported zero gaps.

P1 — Sense IDs can silently acquire a different meaning

Wiktionary keys senses by word/pos/etymology_number#position. Inserting, removing, or reordering source senses can overwrite an existing ID with another meaning. The test reproduced this. Future attachments can remain referentially valid while becoming semantically wrong.

Define stable assertion identity and version/reconciliation policy before attaching permanent user content.

P1 — Slugs conflate different words

Browser reproduction: search C++, click its result → /define/c, headed -c-, with unrelated meanings.

There are 28,306 slug groups with multiple distinct lowercased lemmas; not all are erroneous, but slug collisions must not determine equivalence. Examples include C++, C+, c and resume/résumé.

P1 — Wikipedia probe identity becomes duplicate article identity

The real cat page displays the same 1,840-character Wikipedia summary six times, from separate lookup/redirect probes. 3,852 concepts have multiple Wikipedia entries, totaling 6,173 entries beyond one per concept; the cat example independently verifies identical content.

Preserve probe provenance but normalize publication identity by source/canonical article, and display the article once.

P1 before public deployment — Operational routes are unguarded

/admin/imports and /health use the ordinary browser pipeline in all environments. Anonymous connected visitors can enqueue imports or trigger expensive diagnostics. Import jobs have no configured uniqueness.

Add authentication/authorization on server-side actions, input validation, duplicate-job coalescing, and operational resource controls.

P2 — Green scorecard does not establish semantic correctness

  • M1 checks expected key presence, not complete field equality or unexpected extra rows. It does not comprehensively compare lexeme fields or concept links.
  • Deliberately corrupting stored glosses in the sandbox still produced zero parity gaps.
  • M2 reads historical row counts rather than proving semantic equality or empty-database recovery.
  • X1 builds sampled page structures; it does not test that search navigation retains the intended word.
  • U3/U6 use six named words; link presence does not verify remote resolution.
  • U4/E3 check existence of dated evidence files.
  • A7 is global even when a scope is selected.
  • L1 divides linked rows by a separately measured reachable set without intersecting the numerator with that set.
  • E1 would reject a healthy future migration because it expects exactly two total migrations.

Separate current measurements, samples, historical attestations, and acceptance tests. Add bidirectional semantic parity and independently reviewed relationship precision.

P2 — Relationship and canonicalization semantics need repair

  • Five reciprocal canonical pairs already exist: paddymelon/pademelon, Jacobin/jacobin, flatty/flattie, marten/martin, gill/jill.
  • Wikidata readers ignore statement rank. Stored data contains 906 deprecated statements among retained properties; 131 concept edges match deprecated statements. This does not establish that every matching edge lacks other valid support, but deprecated evidence is not distinguished.
  • Wikidata trimming removes statement IDs, qualifiers, and references.
  • A rejected automatic QID link returns to auto when its linking rung reruns.
  • Ordinary polysemy is treated internally as conflict/disagreement.
  • Lexical relationship uniqueness omits target POS/group/subtype, potentially collapsing distinct assertions.
  • Confidence bounds and sense-to-lexeme consistency are not enforced by the relevant database constraints. The audit found zero existing concept-link sense ownership mismatches; this is a missing protection.

P2 — The full index cannot be restored from source_records alone

The Wiktionary index writes projected lexemes directly. Replaying stored raw records cannot recreate all 1.54 million lexical rows. Recovery requires archived dumps and explicit index/materialize/resolve/link/scope sequencing.

Shared forms/etymologies also require deterministic ownership and multi-etymology handling rather than whichever record wins a merge.

P2 — Community sketch cannot directly represent its promised concept-to-concept example

The documentation describes nepotism the concept → a person the concept. The sketch requires a lexeme and has only one concept field for the example object; it lacks a separate subject concept.

Additional defects: text-example uniqueness omits body; polymorphic votes lack target foreign keys; voter OR permits both a human and bot; required provenance and target-kind checks are missing.

Keep subject, illustrating object, and claim/attachment distinct. Applying a migration proves compatibility, not these semantics.

Source integration requirements, including ConceptNet

“Use the sources correctly” means a reviewed mapping contract per source, not importing everything into generic related-word edges.

Source Backbone role Required treatment
WordNet Source-specific senses/synsets and lexical/semantic relationships Preserve sense/group identity, relation direction, and distinctions between word-level and sense-level edges
Wiktionary/Kaikki Headwords, forms, pronunciations, etymologies, senses, lexical relations Preserve multiple etymologies and source identity; safely reconcile reordered/withdrawn content
Wikipedia Encyclopedic articles, redirects, disambiguation Separate probes from canonical publications; do not treat disambiguation as a definition or redirect as a new article
Wikidata Referents and typed encyclopedic assertions Honor rank policy; retain necessary statement provenance/qualifiers; handle redirects/merges and non-QID subjects explicitly
ConceptNet — not integrated Commonsense relationships involving words and phrases Preserve native node/edge URIs, relation, dataset, supporting sources, license, surface evidence and weight; explicitly map endpoints without assuming QID or exact dictionary-sense equivalence
Historical dictionaries Attributed historical definitions and prose Preserve authorship, edition, spelling, and temporal perspective; do not equate age/source tier with truth

ConceptNet exposes distinct relationship types such as IsA, PartOf, UsedFor, CapableOf, and RelatedTo. Do not collapse these into synonymy or one vague relation. Its edges can name underlying datasets, including WordNet: preserve lineage so reimported evidence is not counted as independent corroboration.

References: ConceptNet API, ConceptNet relations, Wikidata ranks.

A bounded ConceptNet pilot should prove the mapping and replay contract after the identity fixes. This issue does not assume that a full ConceptNet import, universal scope expansion, or a new database technology is necessary.

Work sequence and acceptance criteria

A. Stabilize identities and source assertions

  • Lossless word URLs: searching/clicking C++ reaches C++; explicit policy for casing, accents, punctuation, forms, and variants.
  • Canonical Wikipedia publication identity: cat shows one article, retaining all supporting probe citations.
  • Durable meaning identities: source insertion/reordering/deletion cannot silently retarget a quote/example.
  • Source-owned current/historical assertions: additions and withdrawals reconcile correctly without deleting another source's support.
  • Editorial rejection/confirmation survives automated reruns.
  • Deterministic shared-field and multi-etymology policy; cycle-safe canonicalization.

B. Make the graph trustworthy across providers

  • Document source-native → internal relation mappings, endpoint types, direction, identity, versioning, provenance and unsupported cases.
  • Honor Wikidata rank policy; retain necessary statement evidence and qualifiers.
  • Enforce relationship/target integrity and confidence bounds.
  • Distinguish polysemy from disagreement; expose alternative referents without pretending one primary concept covers every meaning.
  • Implement and evaluate a bounded ConceptNet pilot with native IDs, typed relationships, source lineage and offline replay.
  • Evaluate human-reviewed examples across animal, abstract, cultural, multiword and ambiguous terms; report coverage separately from precision.

C. Prove change, recovery, and safe operation

  • Semantic parity fails on corrupted fields, stale extra assertions, missing assertions and wrong endpoints.
  • Restore a separate empty database from explicitly archived inputs with deterministic semantic results.
  • Separate scorecard measurements from historical attestations; correct scoped populations.
  • Guard operational routes/events, coalesce duplicate work, and document refresh/expiry and rate-limiting policy.
  • Establish CI and a tested backup/restoration procedure before public deployment.

D. Prove the foundation supports a real layer

Completion standard

The milestone is complete when the encyclopedia can explain what relates to what, in which sense, according to which source, and what happens when that source changes, and the first real contribution layer survives those changes.

Keep the existing architecture. Prioritize the robustness of the word-and-relationship backbone over additional mockups, aggregate row counts, or preserving the original table count.

Activity

  1. holden commented on Sep 18, 2026

    @holden
    MemberAuthor

    Closing as superseded by the work it asked for: the identity contract (#73 → ADR 0001), the rebuild on the new model (#74, completion report docs/rebuild/completion-2026-09-09.md), the reader milestone (#82 / PR #83) and connection reliability (#84 / PR #87). One residual is unowned and is now parked in #109's checklist: ConceptNet is still not in the source registry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions