Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar

Ordered by Drew 2026-11-03: "embed and all others should be baked into fork native features — no proxy services needed."

Current shape

POST /v1/embeddings exists in the fork (Wave A, #1434) with the backend registry in packages/core/src/embeddings.ts, but the actual computation for the first registered backend (local-embed/snowflake-arctic-embed-m-v2.0, ONNX int8, 768-d) runs in a separate local service on 127.0.0.1:18752 — a dev/test service with NO nix declaration anywhere (verified by grep in the 2026-11-03 nix audit). The fork calls it over HTTP.

Target shape

The embedding backend runs natively inside the fork server process — model loading, batching, and inference in-process behind the existing registry, with the same external behavior: OpenAI-shaped responses, mixed-length batches untruncated, per-model output dimensions, dimensions parameter (Matryoshka) where the model supports it, loud 400 where it does not, ledger parity for façade calls.

Sequence (additive-first, per standing migration doctrine)

Spike: prove the int8 ONNX model runs in the server runtime (Bun) in-process, outputs matching the sidecar on fixed test vectors.
Add the native backend behind the registry alongside the sidecar backend; verify parity (dims, Matryoshka 256-d, batch shapes, error envelopes).
Flip the registry default to native; verify live with the :18752 service STOPPED.
Only then retire the sidecar service + its code, and note the closure here.

Acceptance criteria

The local embeddings backend runs natively in-process in the fork (no dependency on the :18752 sidecar for the local path)
The :18752 sidecar is retired from the local-backend path, with config/docs updated to match
Embeddings behaviour verified end-to-end on the fork (index + query round-trip)

Step 1 spike — FEASIBLE, in-process inference proven under Bun

Prototype on branch spike/1554-native-embeddings (spike/ dir: scripts + README). Nothing merged, no production change: the sidecar still runs and remains the registry default. All measurements on the laptop (amd-6), Bun 1.4.2, against the live sidecar.

The runtime path that works

onnxruntime-node (1.31.1 — the same runtime version the sidecars python stack uses) loads and runs onnx/model_int8.onnx natively in the Bun process. No WASM build or other fallback is needed. Tokenization via @huggingface/transformers AutoTokenizer — its pure-JS implementation of the HF spec. One correction to the mental model: this tokenizer is not BERT WordPiece — it is a SentencePiece-style Unigram model (Metaspace pretokenizer, ▁, vocab 250,002, <s>/</s> template, pad_id 1), including a Precompiled normalizer. The JS implementation reproduces the sidecars python tokenizers ids exactly on all five test vectors (3-41 tokens, incl. a mixed Japanese text and a query:-prefixed text).

Parity (5 fixed texts vs the live sidecar, HTTP)

Single text, unpadded: bit-identical (max abs delta 0.0, cosine 1.0).
Mixed-length batch (lengths 15/12/31/3/41 in one call): initially cosine ~0.97 — isolated in pad-probe.ts to padding semantics: the int8 graph is padding-sensitive, and the sidecar itself diverges single-vs-batch by the same amounts (digit-for-digit). Replicating the tokenizer padding config exactly (pad right, pad_id=1, mask 0) restores per-element bit-identity (max abs delta 0.0; cosine >= 0.99999949, dot-product accumulation noise only). Parity therefore means: the native backend must batch/pad exactly like the tokenizers library does — now a known, implementable rule, not a mystery.
Matryoshka dimensions: 256 (truncate + renormalize in JS): cosine 1.0, max abs delta ~7e-9.
Registry behaviors covered in-process: mixed-length batch in one call, per-model dims (768 native), 256-d truncation. Loud rejection over the models 8192-token context carries over as a native-side check (token counts come from the attention mask, as in the sidecar).

Performance snapshot (warm medians, shared laptop — rough)

case	in-process	sidecar (HTTP)
single, 15 tokens	36 ms	22 ms
batch of 32 short	288 ms	526 ms
single, ~700 tokens	888 ms	1418 ms

Tiny singles roughly tie; batch and long inputs are ~1.6-1.8x faster in-process, before counting the removed HTTP hop. Lazy load verified: module import 26 ms / no memory change; session created on first embed call (~1.2-1.5 s), RSS +~380 MB with the 311 MB int8 model resident. Boot cost of the server is unchanged; the model cost lands only on first embeddings use.

Step-2 implementation notes (gotchas the spike paid for)

Pin ONE onnxruntime-node version. @huggingface/transformers nests its own (1.30.x); two versions in one process die on VERS_* symbol mismatch. Dedupe/override to a single copy, or consume transformers own runtime wholesale.
The native binding needs libstdc++/libz on the loader path. Bare dev shells on NixOS need LD_LIBRARY_PATH (same store paths as ~/local-embed/run.sh); step 2 must verify the packaged servers nix closure provides them.
Replicate sidecar semantics exactly in the native backend: truncation DISABLED (the shipped tokenizer.json carries a 512 setting the sidecar deliberately overrides — copying the file defaults naively would silently truncate), >8192 tokens -> loud 400, prompt_tokens = summed mask counts, inputs embedded as-is (no query: prefix magic).

Hard blockers

None. The only hard constraints are the two packaging items above (single runtime version; loader path in the nix closure), both step-2 verification tasks, not feasibility risks.

Step 2 (per the issues sequence)

Add a native backend behind EMBEDDING_MODELS alongside the HTTP backend (new registration field selecting backend kind; the translation/validation/ledger layer in packages/core/src/embeddings.ts does not change), load tokenizer + session lazily on first call, replicate the padding/truncation semantics above, and re-run this spikes vectors as the parity test in the package suite.

Merge pull request #1559 from developing-today/step2/1554-native-embe…

Step 2 done — native backend implemented behind the registry (PR #1559, merged `adc454be`)

Registration shape. EmbeddingModel gains backend?: EmbeddingBackend in packages/core/src/embeddings.ts: { kind: "http" } or { kind: "native-onnx", snapshotDir?, snapshotEnv?, onnxFile?, maxTokens }. Absent = HTTP. The shipped Snowflake registration is unchanged — the sidecar is still the default; a test pins that. The service dispatches on kind inside the existing observer/timeout/validation wrapper, so translation and the route_attempt ledger behave identically for both kinds.

Native backend (packages/core/src/embeddings-native.ts) carries the spike's findings exactly:

onnxruntime-node 1.31.1 + @huggingface/transformers AutoTokenizer, both new @opencode/core deps; a root overrides entry pins the single onnxruntime-node version (transformers nests 1.30.0 — the VERS_* trap). Installed graph verified to carry exactly one copy.
Lazy load: tokenizer + session created on first embed call per snapshot, never at boot.
Unigram tokenization with truncation disabled in effect (verified: a 3000-token input encodes to 3002 ids = the sidecar's prompt_tokens), batch padded right with pad_id 1 / mask 0, prompt_tokens = true token counts, >8192 tokens rejected loudly, Matryoshka truncation in float64 → float32, pooling/L2 from the graph's sentence_embedding output.

Parity suite (packages/core/test/embeddings-native.test.ts + fixture): the spike's five vectors, re-verified delta 0.0 against the live sidecar before baking in — single text bit-identical, mixed-length batch bit-identical per element, 256-d within 1e-6, >8192 rejected. The describe skips cleanly when the model artifact is absent (verified with an empty HOME), so CI stays green without the model file. Full @opencode/core suite exit 0; typecheck clean.

⚠ Loader-path finding (step-3 / switch-checklist item). The native bindings — onnxruntime-node's .node and sharp's (loaded transitively by transformers) — need libstdc++ (libstdc++.so.6) and zlib (libz.so.1) on the process loader path. Verified as far as a worktree can: model present + no loader path ⇒ embed fails loudly via the normal channel (Embeddings.UnavailableError: libstdc++.so.6: cannot open shared object file); with LD_LIBRARY_PATH carrying the gcc-lib + zlib store paths (same paths as ~/local-embed/run.sh; here gcc-16.2.0-lib / zlib-1.3.2) the parity suite passes. Before step 3 flips the registration, the packaged server's nix closure must provide libstdc++/libz on the loader path.

Remaining per the sequence: step 3 flips the registry default to native and live-verifies with :18752 stopped; step 4 retires the sidecar. The sidecar is untouched and still running.

Correction to the step-2 comment above (comment 6614951536): its claim "Full @opencode/core suite exit 0" was wrong — that reading came from a shell pipeline whose exit code was the pager's, not the suite's. The actual full-suite result for PR #1559's branch: 8125 pass / 17 skip / 4 fail (8146 tests).

The 4 failures were identified and proven pre-existing: the identical 4 fail on the pristine base commit 974ca6c76d (origin/main before this change) with none of the step-2 change applied —

SessionStep.failureHint > an ordinary provider failure may carry a retry-after and is never a quota
ModelRoute balance-hold grants + credential restart baseline (#1014) > a funded replacement grant releases a credential balance hold…
2× ModelRoute machinery constants (experimental.model_route_settings) (fix(core): model_route_settings reader is a module global — route settings leak across Locations (#1142) #1423 family)

None are in the embeddings area; all embeddings suites (existing HTTP door tests + the new native parity suite) pass, and the core typecheck is clean. The PR #1559 body has been corrected to state this accurately. Nothing about the step-2 deliverable changes: the native backend, parity results, loader-path finding, and sidecar-default confirmation all stand as reported.

Intake review

Fleet monitor intake pass (2026-11-03T20:55:07Z): verified OPEN, no held/user/owned labels.

Related (title-search of merged PRs):

PR feat(core): native in-process embeddings backend behind the registry (#1554 step 2) #1559 feat(core): native in-process embeddings backend behind the registry (Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 step 2)

Notes: A merged PR appears to reference this issue — a worker must confirm whether it fully resolves it before closing; not closed by intake because the reference alone is not unambiguous resolution evidence. Acceptance criteria appended to the body. Priority: priority/normal — architectural improvement, no outage. Effort: effort/l.

Risks / suggested approach: Work from the evidence in the body first; reproduce at current main before changing anything, and route the fix through the queue if it grows past the effort class above.

agent-note:v1
agent: repo-queue-worker-shift-opencode (lane opencode-a)
task: 9460
state: done (triaged)
Triage at fresh main dc3613c: steps 1–2 of this issue's sequence are already DONE.

Step 1 (spike): parity vectors captured from the live :18752 sidecar and baked into packages/core/test/fixtures/embeddings-native-parity.json.
Step 2 (native backend behind the registry): merged as b56245d — packages/core/src/embeddings-native.ts (native-onnx kind, lazy load, Bun-only; Node bundle stubs the bindings per fix(cli): Node bundle crashes at startup on every command — onnxruntime-node native binding dynamically required inside the Rollup bundle #1615 / PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617).
Step 3 (flip the default): NOT done — packages/core/src/embeddings.ts:116 registration names no backend, so the default is still the :18752 HTTP sidecar; packages/core/test/embeddings-native.test.ts:16 pins that. The flip needs a snapshot-resolution choice (snapshotEnv/snapshotDir) on the registration plus live verification with the sidecar stopped.
Step 4 (retire the sidecar): not done; sequenced after the flip verifies.
Predecessors for the flip: fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 (task 8843) and fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 (task 8944) — native-backend robustness fixes already queued; PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617 (task 9407) merge keeps Node-bundle boot safe. fix(ci): check:proto-reads red for embeddings-native truncateVector — one undispositioned site landed with the native embeddings backend #1600 tracks a proto-reads disposition for the new module separately.
Follow-up enqueued: task 9487 (step-3 flip + live verification; packet hidden_files/1554-flip-packet-9460.json). It parks at claim if fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567/fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 are still open, and files the step-4 retirement task once the flip verifies.

🤖 Agent note · picked up

Agent: queue worker (repo-worker-opencode; OpenChamber scheduled lane)
Session / ID: task 9487 · lane 3fe37e8a-c61b-419c-9e97-769ecd264d5d
Found via: scheduled queue claim
Task: Revalidate the Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 step-3 flip prerequisites before implementation.
Expectation: Do not flip the default until fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 and fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 close; then verify with the sidecar stopped.
Links: task 9487 · Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 · fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 · fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 · PR fix(core): evict failed native embeddings loads from the cache (#1567) #1645 · PR fix(core): validate the ONNX output width in the native embeddings backend (#1583) #1632 · PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617

🤖 Agent note · parked

Agent: queue worker (repo-worker-opencode; OpenChamber scheduled lane)
Session / ID: task 9487 · lane 3fe37e8a-c61b-419c-9e97-769ecd264d5d
Found via: scheduled queue claim
Task: Parked task 9487 for 30 minutes; do not flip the Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 default until both predecessor issues close.
Expectation: Resume by revalidating fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 and fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 and their PRs, then proceed only when the gate clears.
Links: task 9487 · Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 · fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 · fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 · PR fix(core): evict failed native embeddings loads from the cache (#1567) #1645 · PR fix(core): validate the ONNX output width in the native embeddings backend (#1583) #1632 · PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617

agent-note:v1
agent: repo-worker-opencode (event-hook, lane opencode-a)
task: 9487
state: parked (predecessor gate)
Step 3 (flip embeddings registration to native-onnx) claimed and revalidated fresh: both named predecessors are still OPEN — #1567 (task 8843) and #1583 (task 8944; its PR #1632 is still open). PR #1617, the third predecessor, is MERGED. Per the packet this task parks naming #1567/#1583 and does not absorb their fixes. Resume when both #1567 and #1583 close.

🤖 Agent note · parked

Agent: queue worker (repo-worker-opencode)
Session / ID: queue task 9487
Found via: queue claim
Task: Step 3 flip for issue Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 (native-onnx default). Revalidated fresh 2026-11-05T01:43Z: issue Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 OPEN with no hard-stop labels, PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617 MERGED, but predecessors are still open — issue fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 OPEN and issue fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 OPEN (its fix PR fix(core): validate the ONNX output width in the native embeddings backend (#1583) #1632 is still OPEN). The packet instructs parking naming any still-open predecessor, so no code work was started.
Expectation: Resume when issue fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567 and issue fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583 both close; then flip EMBEDDING_MODELS[0] to native-onnx and verify live with the :18752 sidecar stopped.
Links: issue fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567, issue fix(core): native embeddings never validate the ONNX output width — a graph whose sentence_embedding width differs from the registration silently misassigns vectors #1583, PR fix(core): validate the ONNX output width in the native embeddings backend (#1583) #1632, PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617

agent-note:v1
state: claimed
task: 9487 (repo-health-opencode-pilot, lane opencode-a, direct)
note: Predecessors revalidated fresh — #1567 CLOSED/COMPLETED, #1583 CLOSED/COMPLETED, PR #1617 MERGED, PR #1632 MERGED. Proceeding with step 3 flip to native-onnx.

Merge pull request #1690 from developing-today-forks/queue/9487-1554-…

agent-note:v1
state: done (step 3)
task: 9487
note: Step 3 flip merged via PR #1690 (merge commit d6be871). EMBEDDING_MODELS[0] now native-onnx with snapshotEnv LOCAL_EMBED_SNAPSHOT + HF-cache snapshotDir default, maxTokens 8192. Verified on laptop: embeddings + embeddings-native suites 20 pass / 0 fail (native parity bit-identical); live index+query round-trip with the :18752 sidecar stopped (768-d, Matryoshka 256-d, mixed-length batch untruncated, retrieval ordering correct). Predecessors #1567/#1583 closed, PRs #1617/#1632 merged. Step 4 (sidecar retirement) filed as #1689 + queue task 9823. This issue stays open pending step 4.

Partial verification — staying open (2026-11-07 done-task-only audit, tracking: developing-today/code#7016).

Queue task(s) [9460, 9487] were marked done, and some work did land:

PR feat(core): native in-process embeddings backend behind the registry (#1554 step 2) #1559 — feat(core): native in-process embeddings backend behind the registry (Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 step 2) (merged 2026-11-03T20:42:06Z): feat(core): native in-process embeddings backend behind the registry (#1554 step 2) #1559
PR fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617 — fix(cli): stub onnxruntime-node and sharp out of the Node bundle (fix(cli): Node bundle crashes at startup on every command — onnxruntime-node native binding dynamically required inside the Rollup bundle #1615) (merged 2026-11-04T12:17:50Z): fix(cli): stub onnxruntime-node and sharp out of the Node bundle (#1615) #1617
PR fix(core): evict failed native embeddings loads from the cache (#1567) #1645 — fix(core): evict failed native embeddings loads from the cache (fix(core): native embeddings load failure is cached forever — one failed first load bricks the backend until restart #1567) (merged 2026-11-05T01:56:06Z): fix(core): evict failed native embeddings loads from the cache (#1567) #1645
PR feat(core): flip embeddings registration to native-onnx backend (#1554 step 3) #1690 — feat(core): flip embeddings registration to native-onnx backend (Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554 step 3) (merged 2026-11-05T02:46:22Z): feat(core): flip embeddings registration to native-onnx backend (#1554 step 3) #1690
Direct commits referencing this issue: adc454b, d6be871

No merged change closes this issue's full scope, so the done status is not treated as completion. Remainder re-queued as triage task 11501.

Navigation Menu

Repository navigation

Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554

Current shape

Target shape

Sequence (additive-first, per standing migration doctrine)

Acceptance criteria

Step 1 spike — FEASIBLE, in-process inference proven under Bun

The runtime path that works

Parity (5 fixed texts vs the live sidecar, HTTP)

Performance snapshot (warm medians, shared laptop — rough)

Step-2 implementation notes (gotchas the spike paid for)

Hard blockers

Step 2 (per the issues sequence)

Step 2 done — native backend implemented behind the registry (PR #1559, merged `adc454be`)

Intake review

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554

Description

Current shape

Target shape

Sequence (additive-first, per standing migration doctrine)

Acceptance criteria

Activity

dezren39 commented on Nov 3, 2026

Step 1 spike — FEASIBLE, in-process inference proven under Bun

The runtime path that works

Parity (5 fixed texts vs the live sidecar, HTTP)

Performance snapshot (warm medians, shared laptop — rough)

Step-2 implementation notes (gotchas the spike paid for)

Hard blockers

Step 2 (per the issues sequence)

dezren39 commented on Nov 3, 2026

Step 2 done — native backend implemented behind the registry (PR #1559, merged adc454be)

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

Intake review

dezren39 commented on Nov 4, 2026

11 remaining items

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Step 2 done — native backend implemented behind the registry (PR #1559, merged `adc454be`)