Repository navigation
feat(repo): projector + link registry for the PM stack - #859
Conversation
Build the glue layer between GitHub and the board/roadmap tools: a projector that turns GitHub App webhook events and reconciliation reads into a link registry and a joined read model in Turso (libSQL). Glue only - it never writes back to GitHub/Kaneo/Pad/Grist and never becomes a source of truth. Components (per the design record's two aggregator requirements): - Webhook intake: HMAC-SHA256-verified receiver for issues, issue_comment, pull_request, pull_request_review, pull_request_review_comment, push, with delivery-GUID idempotency (edge-gateway HMAC discipline). - Reconciliation reads: GitHub App JWT -> installation token, paged issues/PRs/comments/reviews with per-(repo, stream) cursors, idempotent monotonic upserts keyed by repo-qualified IDs. - Link registry: repo-qualified refs (owner/repo#N) attachable to kaneo:<project>/<taskId> and grist:<doc>/<table>/<rowId> subjects, across repos (requirement #1). - Joined read model + comment projection: pmp_v_links_items and pmp_v_subject_activity views plus an HTTP API, incl. PR-interior reviews and review/line comments Kaneo never syncs (requirement #2). Proof: bun test (14 tests) runs a realistic fixture webhook sequence through the real handler with real HMAC verification and asserts the resulting registry + read-model rows. Local HTTP smoke run verified. Reconciliation is implemented but not exercised against live GitHub (no App creds exported here; App installed on code only).
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 16 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f315ace9d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (stream === "issue_comments" || stream === "review_comments") { | ||
| const kind = stream === "issue_comments" ? "issue_comment" : "review_comment"; | ||
| // These endpoints do not support `since`; pull the full list and upsert. | ||
| const rows = await listResource<Record<string, any>>(auth, `repos/${repo}/${kind}s?per_page=100`, fetchFn); |
There was a problem hiding this comment.
Both comment reconciliation streams construct nonexistent endpoints: /repos/{repo}/issue_comments and /repos/{repo}/review_comments. GitHub documents the routes as /issues/comments and /pulls/comments. Consequently, both streams fail instead of rebuilding historical comments or recovering missed webhook deliveries.
Useful? React with 👍 / 👎.
| const out: T[] = []; | ||
| let path: string | null = initialPath; | ||
| while (path) { | ||
| const res = await fetchFn(`https://api.github.com/${path}`, { |
There was a problem hiding this comment.
When a resource exceeds 100 entries, nextPageFromLink returns GitHub's absolute next-page URL, but this request prepends the API origin again. A mocked two-page response reproduces a request to https://api.github.com/https://api.github.com/repos/o/r/issues?page=2. Since listResource throws before returning any rows, affected reconciliation streams import nothing, including the first page. Resolve the initial relative path and subsequent absolute links appropriately, as shown in GitHub's pagination documentation.
Useful? React with 👍 / 👎.
| } catch (e) { | ||
| await store.recordDelivery(db, { deliveryId, event, outcome: "error", detail: String((e as Error).message) }); | ||
| return jsonResponse(500, { error: "apply_failed" }); |
There was a problem hiding this comment.
If projection fails transiently but recording the error succeeds, this catch persists the delivery ID even though application did not complete. deliveryApplied checks only whether that ID exists, so redelivering the same webhook returns 200 duplicate without retrying the failed writes. A mocked transient store failure reproduces 500 apply_failed followed by 200 duplicate. Distinguish successful deliveries from failed attempts and allow an error record to transition to success.
Useful? React with 👍 / 👎.
| const row = fromIssueOrPr(repo, obj, isPr ? "pr" : "issue"); | ||
| await store.upsertItem(db, row); |
There was a problem hiding this comment.
The PR stream reads /issues, whose response represents PRs as issue objects, rather than full pull-request objects with top-level merge metadata. Passing these directly to fromIssueOrPr defaults merged to 0 and merged_at to null. Because upsertItem accepts equal source timestamps, a reconciliation after a merge webhook can erase the correct merge status; a fresh rebuild also marks merged PRs as unmerged. Fetch or derive the PR-specific metadata before replacing these fields.
Useful? React with 👍 / 👎.
| submitted_at: null, | ||
| commit_id: null, | ||
| deleted: 0, | ||
| source_updated_at: createdAt, |
There was a problem hiding this comment.
For issue comments and review comments, every edit retains the same created_at, so assigning it to source_updated_at defeats the store's stale-write protection. If an older edit webhook arrives after a newer one, both guard values are equal and upsertComment overwrites the current body with stale content. Prefer updated_at with a creation-time fallback, and apply the same rule to comment rows constructed by reconciliation.
Useful? React with 👍 / 👎.
| const base = commentCommon(review, repo, ref, "review"); | ||
| base.review_state = iso(review.state); | ||
| base.submitted_at = iso(review.submitted_at) ?? base.created_at; |
There was a problem hiding this comment.
Reviews carrying submitted_at but no created_at or updated_at fall through commentCommon to the current processing time. Setting only base.submitted_at afterward leaves created_at and source_updated_at incorrect. Delayed or replayed reviews therefore appear as new activity, and reconciliation uses the original submission timestamp, which the monotonic guard rejects as older—even when reconciliation contains a corrected body or dismissed state. Initialize the review's creation and guard timestamps from its submission time.
Useful? React with 👍 / 👎.
| const b64 = decodePem(rawKey); | ||
| const der = Uint8Array.from(Buffer.from(b64, "base64")); | ||
| const key = await crypto.subtle.importKey( | ||
| "pkcs8", | ||
| der as unknown as BufferSource, |
There was a problem hiding this comment.
When GITHUB_APP_PRIVATE_KEY contains base64 of a PEM file, as explicitly supported by the README and environment example, decodePem leaves the outer encoding intact. This single base64 decode then produces PEM text rather than DER, so importKey('pkcs8', ...) throws DataError and App-authenticated reconciliation cannot start. Decode the outer wrapper first, then strip the decoded PEM armor and decode its DER contents.
Useful? React with 👍 / 👎.
| sql: `SELECT c.* FROM pmp_comments c | ||
| JOIN pmp_links l ON l.ref = c.ref | ||
| WHERE l.subject = ? AND c.deleted = 0 |
There was a problem hiding this comment.
When a subject links to the same item under two relations, such as tracks and implements, both links are valid under the registry's (subject, ref, relation) primary key. This join returns each comment once per relation, duplicating activity in both /comments?subject=... and /subjects/.... The duplicates also consume the 500-row limit and can exclude otherwise eligible comments. Join against distinct linked refs or use an existence predicate so each comment appears only once per subject.
Useful? React with 👍 / 👎.
| const installId = | ||
| installationIds?.[repo] ?? (env.GITHUB_APP_INSTALL_ID ? Number(env.GITHUB_APP_INSTALL_ID) : undefined); | ||
| if (env.GITHUB_APP_ID && env.GITHUB_APP_PRIVATE_KEY && installId) { |
There was a problem hiding this comment.
When App credentials, a global installation ID, and GITHUB_TOKEN are configured together, this branch returns the same installation token for every repo without checking repo. In the documented deployment where the App covers developing-today/code but dezren39/nix requires the token fallback, reconciliation therefore uses the wrong credentials for nix and never reaches the fallback. GitHub's installation authentication documentation limits installation tokens to repositories granted to that installation. Resolve installation access per repository and use the configured fallback when the App cannot access it.
Useful? React with 👍 / 👎.
| if (url.startsWith("file:")) { | ||
| return createClient({ url }); |
There was a problem hiding this comment.
On a fresh checkout with DATABASE_URL unset, the documented local startup uses file:data/projector.db, but data/ does not exist and this path opens the database without creating its parent directory. Searching the tracked tree and projector setup scripts found no directory creation for it; SQLite opening the same path reproduces unable to open database file. Both the server and reconciliation CLI therefore fail before migration under the default configuration. Create the local database's parent directory before calling createClient.
Useful? React with 👍 / 👎.
PR #859's wire schema differs from the sketch the aggregator was built against: snake_case fields, state open|closed with merged as a separate 0/1 flag, labels/assignees as JSON array strings, comment rows with created_at/review_state, and 0/1 deleted tombstones. Map rows to the read model in live.ts (mapProjectorItem/mapProjectorComment/parseProjectorLabels), percent-encode refs in query strings (# would otherwise start a fragment), enrich board cards and roadmap rows with GET /links?subject= registry links, use kaneo:<slug>/<taskId> subjects with the slug->repo map from kaneo-projects.json, and default PM_PROJECTOR_URL to 127.0.0.1:7801. Verified end-to-end against Session J's real projector service (temp DB in /tmp, seeded item/comment/link): merged state, seeded title, parsed labels, and the seeded review all rendered; footer reported items/comments live (projector). Tests: 28 pass, tsc clean.
…-align feat(web): align aggregator projector reader with landed schema (#859)
Session J (PM-stack wave) — projector + link registry. Stacked on
pm/connect-repos(Session H); this PR targetspm/connect-repos. Do not merge — Muse reviews/merges. Branch was merged forward to H's current tip (37265e02) with a merge commit, no conflicts.What this is
The glue layer from the design record (
doc/2026-10-24T11-05-38Z_design_project-management-stack/): a projector that turns GitHub App webhook events + reconciliation reads into a link registry and a joined read model in Turso (libSQL). It links and projects; it never writes back to GitHub/Kaneo/Pad/Grist and never becomes a fifth source of truth. Code:pm-stack/projector/with its own README (runbook, env/secret NAMES, read schema for Session K).issues,issue_comment,pull_request,pull_request_review,pull_request_review_comment,push; delivery-GUID idempotency.owner/repo#N) attachable tokaneo:<projectSlug>/<taskId>andgrist:<docId>/<tableId>/<rowId>subjects, across repos. H's project slugs (code,opencode,openchamber,nixinpm-stack/kaneo-projects.json) are thekaneo:slug namespace.pmp_v_links_items/pmp_v_subject_activity+ HTTP API (/items,/comments,/links,/subjects/<subject>), including PR-interior reviews and review/line comments that Kaneo never syncs, as one unified stream per board item / roadmap row.Proven
bun test: 14 tests, 0 fail — store upserts + monotonic guard, projector event mapping (incl. PR-interior fields, delete → tombstone, allow-list), and an end-to-end test that feeds a realistic fixture webhook sequence (push → issue → issue comment → PR → PR review → PR review comment → cross-repo issue) through the real handler with real HMAC signatures and asserts the resulting registry + read-model rows, plus duplicate-delivery idempotency.tsc --noEmitclean.codeissue + anopenchamberissue) and the PR-interior comment stream for the PR.pm-stack/projector/{fixtures,test}.Stubbed / not proven
developing-today/codeonly (dezren39/nixwould need its own installation or theGITHUB_TOKENfallback). Turso CLI is present but not logged in; the projector runs againstfile:libSQL locally and is env-switched to a remote Turso URL/token — no dedicated database was created (fields arepmp_-prefixed and can share the fleet's general DB; rationale in the README).bun installcan't reach the npm registry from this laptop (hangs);scripts/setup-deps.shprovisions@libsql/client@0.18.0from the existing local opencode-mem cache.node_modules/is gitignored.For Session K
Base URL
PM_PROJECTOR_URL(defaulthttp://127.0.0.1:7801):GET /items?refs=…,GET /links?subject=…,GET /comments?refs=…|?subject=…,GET /subjects/<urlencoded subject>(joined links + live item state + unified activity), or the same via the SQL views above. Subjects:kaneo:<projectSlug>/<taskId>,grist:<docId>/<tableId>/<rowId>(full path — Grist row ids are unique only per table),gh:<ref>.Need help on this PR? Tag
@codesmith-botwith what you need.