feat(stack): EQL v3 JSON support — types.Json + containment

[coderdan]

EQL v3 JSON support (part 1)

Adds encrypted JSON columns to the EQL v3 typed schema: types.Json('col'), a
lossless round-trip, and Drizzle containment querying. This is the last PR in the
integration-tests → type-robustness → live-suite-reorg → JSON sequence.

Stacked on #620. Base is feat/eql-v3-identity-clerk-m2m; the diff is only

Scope: containment only

This PR implements JSON containment (ops.contains(col, subObject) → @>). Two JSON concerns are deliberately out of scope, tracked as follow-ups:

• JSONPath selector-with-constraint (metadata->'plan' = $1) — the eql_v3_json domain supports ->/->> at the SQL level, but the Drizzle operator and the selector-string needle typing are not wired up. → EQL v3 JSON: implement JSONPath selector-with-constraint querying (->, ->>) #623
• Migrating all v3 Drizzle operands to encryptQuery (today only JSON containment uses it; scalar ops still use client.encrypt). → EQL v3 Drizzle: encrypt all query operands with encryptQuery (not client.encrypt) #622

Also noted for the EQL team: v3 exposes only one JSON domain (eql_v3_json), which requires the ste_vec document structure — there is no storage-only / no-query JSON domain equivalent to v2's dataType('json') without .searchableJson().

the three feat/eql-v3-json commits. Merge after #616 → #618 → #619 → #620.

What's new

types.Json('metadata') declares a public.eql_v3_json column. It encrypts a
whole JSON document to an ste_vec SteVecDocument and round-trips losslessly
through encrypt/decrypt and the model path. The new searchableJson query
capability emits the ste_vec index.

Drizzle containment — ops.contains(col, subObject) now answers
encrypted-JSONB containment on a Json column, returning exactly the rows whose
document contains the sub-object (jsonb @> semantics).

Two findings worth calling out

• json has no eql_v3.contains overload. Containment is the @> operator —
  public.@>(eql_v3_json, eql_v3.query_jsonb). The dialect emits
  col OPERATOR(public.@>) needle::eql_v3.query_jsonb, where the needle is a
  narrowed query_jsonb term from encryptQuery (no ciphertext), not the full
  storage envelope. This is what resolves the "function is not unique" (42725)
  ambiguity.
• Array containment required dropping positional indexing. v2's
  searchableJson() uses array_index_mode: 'all', which emits a positional term
  per element — so {roles:['eng']} matched a doc with eng at index 0 but not
  one with eng at index 1. The Json column keeps item + wildcard and drops
  position, making containment a true subset test regardless of element order.
  That is the correct default for a containment column.

mode: 'compat' is required

eql_v3_json orders ste_vec entries by the CLLW-OPE op term, so the index must
emit compat terms. v2's 'standard' (CLLW-ORE oc) is rejected at encrypt time.

Naming note (ste_vec vs eql_v3_json)

EQL 3.0.0 renamed the column/domain types (public.eql_v3_json,
eql_v3_jsonb_entry — there is no ste_vec type). But ste_vec remains the name
of the index kind in the protect-ffi config, the payload shape
(SteVecDocument, k:"sv"), and internal SQL functions. All ste_vec references
in the code sit in exactly those layers; nothing user-facing uses it.

Tests & verification (live pg)

• integration/shared/json-crypto — round-trip, scalar + model paths (2/2)
• integration/drizzle-v3/json-contains — object / scalar / nested-boolean /
  array-subset / no-match containment (4/4)
• operators.test — json @> SQL emission, encryptQuery-failure surfacing,
  null-operand rejection (3 new); full stack unit suite 1650 passing
• source typechecks clean, biome clean

Skills

Per the same-PR rule: stash-encryption gains types.Json in its typed-schema
catalog and an encrypted-JSONB query section; stash-drizzle documents json
containment on contains. Carries a stash patch changeset (skills ship in the
tarball) alongside the @cipherstash/stack minor.

https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w

[changeset-bot]

🦋 Changeset detected

Latest commit: f4d45dd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
stash	Patch
@cipherstash/stack	Minor
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/bench	Patch
@cipherstash/prisma-next	Patch
@cipherstash/test-kit	Patch
@cipherstash/prisma-next-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4f1b51e1-6a26-4d6a-bc3c-3a76a89794a6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/eql-v3-json

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderdan]

Test-coverage review — EQL v3 JSON (ste_vec) containment

Strong coverage overall: the built column shape is pinned in v3-matrix/matrix.test.ts (build() toStrictEqual over the catalog), the registry key in eql-v3-domain-registry.test.ts, the family partition/deferral in test-kit-families.test.ts, plus live round-trip (json-crypto) and live containment incl. position-independent array matching (json-contains, ada matches with eng at index 1). The unit suite covers the happy @> path, encryptQuery failure, and null-operand rejection.

Two gaps stand out, both mirroring anti-patterns already handled elsewhere in this file:

1. The new encryptJsonContainmentTerm guard for a client without encryptQuery has no test — the setup() double always supplies it, so the branch the code comments call out as load-bearing is never exercised. (inline)
2. Empty-object containment (contains(col, {})) is unpinned. The bloom path deliberately keeps requireAnswerableNeedle because an empty needle makes @> '{}' match every row; the ste_vec path added here has no equivalent guard and no test documenting the behaviour. This is a lopsided-negative-case gap with a whole-table-leak flavour — flagging it as a coverage gap, not asserting a bug. (inline)

Additional coverage gaps not posted inline

• Top-level non-object JSON round-trip. JsonValue (and types.Json) admit a top-level array, scalar, or null, but json-crypto.integration.test.ts only round-trips objects. A regression in ste_vec encoding of non-object roots would go unnoticed. Add a case round-tripping e.g. [1, 2, 3] and null (or, if only object roots are supported, assert the rejection so the boundary is explicit).

[Copilot]

Pull request overview

Adds first-class EQL v3 encrypted JSON support to @cipherstash/stack via a new typed-schema column factory (types.Json) backed by an ste_vec index, plus Drizzle-side JSON containment querying that emits @> with a narrowed query_jsonb needle from encryptQuery. Also updates the published skills docs and expands the shared integration test-kit catalog to include (deferred) JSON domains while covering behavior with dedicated live suites.

Changes:

• Add types.Json('col') / EncryptedJsonColumn and JsonValue typing, including ste_vec index emission (mode: 'compat', positionless array containment).
• Implement Drizzle v3 JSON containment dispatch in contains() to emit OPERATOR(public.@>) …::eql_v3.query_jsonb using encryptQuery.
• Add/adjust dedicated integration + unit tests, plus update shipped skills/*/SKILL.md and include changesets for stack + stash (skills).

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
skills/stash-encryption/SKILL.md	Documents types.Json in the v3 typed schema catalog and adds encrypted-JSONB query guidance.
skills/stash-drizzle/SKILL.md	Documents that contains supports JSON containment for types.Json columns.
packages/test-kit/src/run-family-suite.ts	Handles fully-deferred families (like json) without silently skipping.
packages/test-kit/src/oracle.ts	Narrows widened samples back to scalar Plain for the scalar oracle.
packages/test-kit/src/ops.ts	Adds searchableJson capability key mapping (for type completeness).
packages/test-kit/src/families.ts	Adds json as a family and marks it deferred for coverage accounting.
packages/test-kit/src/catalog.ts	Adds the public.eql_v3_json domain spec (deferred) with ste_vec index config and samples.
packages/stack/src/eql/v3/types.ts	Adds types.Json factory to construct EncryptedJsonColumn.
packages/stack/src/eql/v3/index.ts	Re-exports EncryptedJsonColumn and JsonValue.
packages/stack/src/eql/v3/drizzle/sql-dialect.ts	Adds containsJson dialect helper emitting @> with query_jsonb cast.
packages/stack/src/eql/v3/drizzle/operators.ts	Routes contains() to JSON containment when ste_vec index is present; uses encryptQuery for the needle.
packages/stack/src/eql/v3/columns.ts	Adds searchableJson capability plumbing, ste_vec index defaults, EncryptedJsonColumn, and JsonValue plaintext mapping.
packages/stack/integration/supabase/introspect.integration.test.ts	Updates introspection expectations now that eql_v3_json is modelled by types.Json.
packages/stack/integration/shared/matrix-crypto.integration.test.ts	Adjusts ciphertext-shape guard to handle ste_vec (k:'sv') payloads.
packages/stack/integration/shared/json-crypto.integration.test.ts	New live round-trip suite proving JSON encrypt/decrypt and model-path round-trip.
packages/stack/integration/drizzle-v3/json-contains.integration.test.ts	New live Drizzle containment suite for encrypted JSONB (ops.contains on types.Json).
packages/stack/tests/test-kit-families.test.ts	Updates deferred-family expectations to include json.
packages/stack/tests/supabase-v3-matrix.test.ts	Updates domain tier counts to include eql_v3_json.
packages/stack/tests/eql-v3-domain-registry.test.ts	Pins the new eql_v3_json registry key.
packages/stack/tests/drizzle-v3/operators.test.ts	Adds unit assertions for JSON containment SQL emission and error surfacing.
.changeset/eql-v3-json.md	Changeset for @cipherstash/stack minor release describing v3 JSON support.
.changeset/eql-v3-json-skills.md	Changeset for stash patch release due to shipped skill updates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderdan]

Code review — EQL v3 JSON support (xhigh effort)

Reviewed the diff against feat/eql-v3-identity-clerk-m2m, reading the changed source plus enclosing functions and verifying each top finding against the real code (QueryTypesForColumn, inferQueryOpFromPlaintext, resolveIndexType, v2's prefix-rewrite path, the typed encryptQuery signature). 10 findings, correctness first.

The changesets and skill updates are present and otherwise accurate — the one exception is the encryptQuery example, which finding 1 makes wrong.

Correctness

1. Typed encryptQuery can't be called on a types.Json column — the shipped skill example won't compile. CONFIRMED
packages/stack/src/eql/v3/columns.ts:750
QueryCapabilities gained searchableJson, but QueryTypesForColumn (lines 750–760) still maps only equality/orderAndRange/freeTextSearch. A json column has all three false, so QueryTypesForColumn<Col> = never; QueryableColumnsOf<Table> (table.ts:199) drops it; and the typed encryptQuery signature Col extends QueryableColumnsOf<Table> (v3.ts:61) rejects it. The example added to skills/stash-encryption/SKILL.md —

await client.encryptQuery({ roles: ['admin'] }, { column: events.metadata, table: events })

— fails to type-check for any customer following the (customer-shipped) skill, and queryType: 'searchableJson' is never an accepted queryType on the typed client. Add a searchableJson extends true ? 'searchableJson' : never arm.

2. A string needle is mis-routed to a JSONPath selector under the @> operator. CONFIRMED
packages/stack/src/eql/v3/drizzle/operators.ts:473
contains() hard-codes queryType: 'searchableJson', whose queryOp is inferred from the plaintext. inferQueryOpFromPlaintext (infer-index-type.ts:42-43) classifies every string as a JSONPath selector (ste_vec_selector), but containsJson always emits the @> containment operator. So ops.contains(jsonCol, 'admin') builds a selector term — not an { sv: [...] } containment needle — and emits col OPERATOR(public.@>) '<selector>'::eql_v3.query_jsonb. The eql_v3.query_jsonb domain CHECK requires an object with an sv array, so Postgres raises a domain-constraint violation (23514); for a legit '$.path' selector string it silently mis-queries under @>. String/scalar needles are documented (stash-encryption skill: "pass an object/array/scalar") and entirely untested — the integration tests use only object needles.

3. Empty-object needle {} matches every row. PLAUSIBLE
packages/stack/src/eql/v3/drizzle/operators.ts:437
The JSON branch has no unanswerable-needle guard (the ste_vec analogue of requireAnswerableNeedle). ops.contains(jsonCol, {}) passes requireNonNullOperand and emits col OPERATOR(public.@>) '{}'::eql_v3.query_jsonb; under jsonb @> semantics doc @> '{}' holds for every object document, so the predicate silently returns the whole table — the exact whole-table failure the free-text path was hardened against. Reject a trivially-matching needle before emitting SQL.

4. lockContext / audit are silently dropped on the JSON branch. CONFIRMED
packages/stack/src/eql/v3/drizzle/operators.ts:438
The free-text branch calls encryptOperand(ctx, right, operator, opts) → applyOperationOptions → withLockContext()/audit(). The json branch calls encryptJsonContainmentTerm(ctx, right, operator) with no opts, and that function calls encryptQuery directly with no applyOperationOptions. EncryptQueryOperation supports withLockContext (encrypt-query.ts:34), so a json column encrypted under a lock context builds a query term without that context and matches nothing, and ops.contains(col, needle, { audit }) drops the audit config entirely. Thread opts through and wrap the encryptQuery call.

5. ste_vec prefix: 'enabled' is never rewritten per-column in the v3 path. CONFIRMED
packages/stack/src/eql/v3/columns.ts:388
The comment (lines 372–375) claims "the table's build() rewrites [it]" to ${tableName}/${columnName}, but v3 EncryptedTable.build() (table.ts:52) copies builder.build() unchanged. The rewrite that does this in v2 (schema/index.ts:498-511, covered by schema-builders.test.ts:220) has no v3 counterpart, and steVecIndexOptsSchema accepts any string — so prefix: 'enabled' reaches protect-ffi verbatim. Consequence: unlike v2, all json columns across all tables compute selector/term hashes under the same constant prefix, so identical JSON structure/values yield identical ste_vec hashes in different columns — the cross-column correlation the per-column prefix exists to prevent. Single-column round-trip/containment still works (encrypt and query share the config), which is why the integration tests pass and the regression is silent.

6. JSON.stringify(result.data) is unguarded when data is null/undefined. PLAUSIBLE (latent)
packages/stack/src/eql/v3/drizzle/operators.ts:479
encryptJsonContainmentTerm checks only result.failure. If encryptQuery ever resolves { data: undefined } with no failure, sql${JSON.stringify(undefined)}`` binds an undefined param (driver-dependent throw/NULL) or 'null'::eql_v3.query_jsonb fails the domain CHECK. `requireNonNullOperand` currently makes this unreachable, but the v2 json operator guards `if (encrypted === undefined) throw` and the v3 path dropped it.

Cleanup / altitude

7. searchableJson is an optional 4th capability instead of a required boolean. columns.ts:21
Being optional forces (capabilities.searchableJson ?? false) (columns.ts:413), a bespoke searchableJson?: false | undefined arm in QueryableFlag (columns.ts:70), and — as finding 1 shows — makes it easy to forget a consumer entirely. A required searchableJson: boolean (adding searchableJson: false to the ~13 scalar domain constants once) removes the asymmetry and the scattered ?? false.

8. The prefix: 'enabled' sentinel + rewrite convention is duplicated, never extracted. columns.ts:389
v2 owns the rewrite inline in build(); v3 re-emits the identical sentinel (and again in test-kit/catalog.ts) but never rewrites it. A shared rewriteSteVecPrefix(built, tableName, colName) called by both build paths collapses the duplication and makes the v3 omission (finding 5) a compile-time obligation rather than a silent divergence.

9. contains() funnels two unrelated SQL paths through one function. operators.ts:425
requireIndex(ctx, ['match','ste_vec'], …, 'free-text search or JSON containment') then branches on ctx.indexes.ste_vec to emit either eql_v3.contains(col, operand) or col @> needle — unrelated overloads with different needle builders and validation (requireAnswerableNeedle applies only to match). Calling contains on a plain storage column yields the conflated error message. Dispatching by ctx.builder domain (json vs text) at the top keeps the two paths from sharing a surface they don't actually share.

10. searchableJson: ['contains'] in ops.ts is an inert entry and a latent trap. packages/test-kit/src/ops.ts:64
It exists only to satisfy Record<keyof QueryCapabilities, …> and never gates a run (json is deferred). But if a json domain were ever un-deferred, enabledBy('contains', jsonCaps) would route it into the scalar oracle's text-needle contains and mis-test the ste_vec column. Mapping it to [] keeps the type satisfied while making "this driver runs no op for json" explicit and safe.

---

Findings 1, 2, 4, 5 are CONFIRMED against source; 3 and 6 are PLAUSIBLE. Generated by an automated xhigh-effort review pass.

[coderdan]

Review findings addressed — 58b9ef8

Thanks both. All four addressed (none refuted — each was a real gap or a real defect):

1. encryptQuery-absent guard untested (coderdan) — added the suggested unit test: a { encrypt }-only client hitting JSON containment now asserts the typed EncryptionOperatorError. The load-bearing guard is exercised.

2. Empty-object containment (coderdan) — probed live: contains(col, {}) did return all rows (doc @> '{}' holds for every document). That's the same whole-table footgun the bloom path guards, so I added a guard rather than only pinning it: the ste_vec path now rejects an empty-object needle with a typed error (matches every row) before it ever queries. Pinned in the unit suite and the live suite.

3. Non-object JSON roots (coderdan) — the finding surfaced a genuine type/runtime mismatch. Live behaviour: object, array, and null roots round-trip; a top-level scalar is rejected by protect-ffi (Cannot convert … to Json). But JsonValue (the column plaintext) admitted scalars, so encrypt('foo', { column: jsonCol }) compiled and failed only at runtime. Fixed at the type level: the column plaintext is now JsonDocument (object | array | null), with JsonValue retained for nested values. Added live round-trips for array and null roots, and an assertion that a top-level scalar is rejected — so the boundary is both compile-enforced and pinned.

4. Misleading header comment (Copilot) — corrected: the json path emits @> with a query_jsonb needle from encryptQuery, not eql_v3.contains.

New/changed public surface: JsonDocument type exported from @cipherstash/stack/eql/v3; types.Json plaintext narrowed to it. Skill updated to match. Verified: type tests 93/93, operators unit 195/195, json-crypto 7/7, json-contains 5/5, matrix-crypto 206/206, full stack unit 1680 passing.

[coderdan]

Review triage — responses in-thread

Replied to each thread above. Summary of dispositions:

Follow-up issues (out of scope for this PR, by design — keeping it to column + round-trip + containment):

• EQL v3 Drizzle: encrypt all query operands with encryptQuery (not client.encrypt) #622 — migrate all v3 Drizzle operators from client.encrypt to client.encryptQuery (query terms, no ciphertext) and make encryptQuery non-optional. This is what the operators.ts:62 guard comment is a symptom of; it predates this PR.
• EQL v3 JSON: implement JSONPath selector-with-constraint querying (->, ->>) #623 — JSONPath selector-with-constraint querying (foo->'bar' = $1) via ->/->>. This PR is containment (@>) only; I'll add an explicit "containment only" note to the PR description + skill so the omission is stated.

Wording / hygiene to fix in this PR (once you give the go-ahead):

• Sweep ste_vec out of user-facing comments/test names/skills → "encrypted JSONB document" / eql_v3_json, keeping it only for the protect-ffi config key/payload.
• Reclassify eql_v3_json in the operators test as a queryable (containment) domain, not storage-only.
• Rework the "deferred" framing for json in the family model (it's covered-by-dedicated-suites, not deferred like the ORE domains).
• Rewrite the opaque introspect comment; drop makeEqlV3Column/as never/the drizzleEq/drizzleAsc aliases in the tests.

Parked: whether EQL v3 should have a storage-only (no-query) JSON domain at all. Today there's exactly one JSON domain, eql_v3_json, whose CHECK requires a valid ste_vec document — so there's no encrypt-only JSON equivalent to v2's dataType('json') without .searchableJson(). @coderdan is confirming intent with the EQL team before we decide whether types.Json needs a storage-only sibling.

[coderdan]

Wording / hygiene batch landed — e27ce9d

The non-design review points are addressed (design ones split to #622/#623):

• ste_vec language swept out of user-facing comments, test names, and the skill → "encrypted JSONB document" / eql_v3_json. Kept only where it's literally the protect-ffi config key (indexes.ste_vec) or the SteVecDocument payload type.
• json-contains test: dropped makeEqlV3Column (use the Drizzle types.Json), which let both as never casts go (the pgTable and the bulkEncryptModels call — the table is now properly typed); un-aliased the Drizzle eq/asc imports.
• operators unit: storageDomains → nonScalarQueryDomains; json is a queryable (containment) domain that answers no scalar op, not a storage-only one.
• deferred framing: reworded across catalog/families/run-family-suite/test — for json it means "not run by the scalar op-matrix, covered by dedicated suites", explicitly distinguished from the ORE domains' superuser-only deferral.
• introspect comment rewritten as the explicit three-way case.
• skill: JSON documented as containment-only today; JSONPath selector marked not-yet-implemented (EQL v3 JSON: implement JSONPath selector-with-constraint querying (->, ->>) #623); JsonDocument (no top-level scalar) noted.

Scope note added to the PR description. All green locally: unit 1680, type tests 93, json-contains 5, json-crypto 7, introspect 6.