Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/backfill-missing-column-message.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'stash': patch
---

`stash encrypt backfill` now distinguishes a missing encrypted column from a
legacy EQL v2 one. The domain probe returns the same "not v3" answer for both,
so a user who had simply not added the `<col>_encrypted` column yet was told
they were on a legacy EQL v2 column and advised to migrate a domain that did not
exist. The command now reports that the column is absent, points at adding an
`eql_v3_*`-domain column and applying the migration, and mentions
`--encrypted-column` for non-standard names. The EQL v2 message is unchanged for
columns that really are present.
2 changes: 2 additions & 0 deletions .changeset/cli-v2-cutover-prompt-correction.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ that cannot describe the column it just cut over to. The prompt now says
explicitly not to use `types.*` for a v2 column, and points at the deprecated
`@cipherstash/stack/schema` builders with decryption through
`@cipherstash/stack`, which is the actual read path for legacy v2 rows.

Superseded later in this release: `stash encrypt cutover` and all v2 mutation guidance are removed; legacy v2 remains read-only.
2 changes: 2 additions & 0 deletions .changeset/encrypt-lifecycle-mixed-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,5 @@ Two cases DO newly exit 1, both deliberately:
unconventionally, where `cutover` previously exited 0 with "not applicable".
Re-run `stash encrypt backfill --table T --column C --encrypted-column <name>`
to record the pairing.

Superseded later in this release: no v2 lifecycle can be driven by `stash encrypt`; mixed and pure-v2 state now fail with migration/recovery guidance.
2 changes: 2 additions & 0 deletions .changeset/eql-v3-cli-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ stripped). v3 currently supports the direct install path only —
`--drizzle`/`--migration`/`--migrations-dir`/`--latest` are rejected — and the
installer keys `isInstalled`/version checks and Supabase grants to the `eql_v3`
schema.

Superseded later in this release: `--eql-version` and the v2 installer are removed; installs and upgrades are v3-only.
2 changes: 2 additions & 0 deletions .changeset/eql-v3-sole-docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,5 @@ than a Legacy section, because EQL v2 is still reachable there:
Also corrects the legacy `@cipherstash/drizzle` README's pointer to the removed
`@cipherstash/stack/drizzle` subpath (now the separate `@cipherstash/stack-drizzle`
package).

Superseded later in this release: CLI/migrate v2 mutation guidance is removed; only legacy ciphertext reads and status diagnostics remain.
7 changes: 4 additions & 3 deletions .changeset/init-drizzle-eql-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The generated migration also carries the `cs_migrations` tracking schema, so one
isn't installed or configured, init now reports EQL as not installed and points
at `stash eql migration --drizzle` rather than aborting the run.

The v2 Drizzle path remains available for existing deployments via an explicit
`stash eql install --drizzle --eql-version 2`; that command's error message now
points at the v3 alternative instead of only suggesting `--eql-version 2`.
The final CLI installation and mutation surface is v3-only: the explicit v2
Drizzle install path is removed. Legacy v2 remains readable and visible in
diagnostics. Generate a checked-in install migration with `stash eql migration
--drizzle`.
2 changes: 2 additions & 0 deletions .changeset/init-scaffold-compiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,5 @@ typecheck step, the codegen tests only string-match fragments of the template,
and the step test stubs the generator out entirely. Both templates are now
committed as fixtures that CI typechecks, pinned byte-for-byte to the generator
so they cannot drift.

Superseded later in this release: the generated guidance no longer references removed `db push` or `encrypt cutover` commands.
2 changes: 2 additions & 0 deletions .changeset/migrate-eql-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,5 @@ right lifecycle, no new flags:
The `stash-cli` and `stash-encryption` skills and the `@cipherstash/migrate`
README document the two lifecycles (v2: backfill → cutover → drop;
v3: backfill → switch-by-name → drop).

Superseded later in this release: `@cipherstash/migrate` and the CLI now author and mutate v3 only; legacy v2 manifest fields remain readable.
8 changes: 8 additions & 0 deletions .changeset/remove-cli-migrate-v2-leaf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'stash': major
'@cipherstash/migrate': major
---

Remove the remaining EQL v2 installation and rollout surface. CLI installs,
upgrades, backfills, and drops now mutate EQL v3 state only, while legacy v2
status diagnostics and migration-manifest compatibility remain read-only.
2 changes: 2 additions & 0 deletions .changeset/skills-v3-lifecycle-honesty.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,5 @@ case, expect the wrong column to be dropped.

Skills ship inside the `stash` tarball and are copied into user projects at
`stash init`, so this guidance was being installed into customer repos.

Superseded later in this release: the bundled skills no longer document v2/Proxy mutation commands because those CLI paths are removed.
4 changes: 4 additions & 0 deletions .changeset/stash-cli-eql-v3-default.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,7 @@ Default EQL to v3 and stop the CLI recommending `stash db push` (#585).
- **`stash db push` is no longer recommended in CLI output.** `db push` writes the `public.eql_v2_configuration` table, which is a v2 + CipherStash Proxy artifact — EQL v3 has no configuration table (config lives in each column's `eql_v3.*` type) and nothing in the v3 stack reads it. The push recommendations are removed from `eql status`, the help banner, and the init/plan/cutover guidance. `db push` (and `db activate`) remain available for EQL v2 + Proxy users; they're now labelled as such.
- **`eql status` is v3-aware.** On a v3-only database it reports that encrypt config lives in the column types instead of hitting a "table not found" dead-end that told users to run `db push` (which neither creates that table nor applies to v3).
- **`stash db push` guards a v3-only database** with a clear "not needed under EQL v3" message instead of a raw `relation "public.eql_v2_configuration" does not exist` error.

**Superseded later in this release:** the CLI no longer installs or mutates EQL
v2 state; `db push`, `db activate`, the Proxy choice, and the v2 install flags
described above are removed. Legacy state remains visible through status only.
2 changes: 2 additions & 0 deletions .changeset/stash-cli-skill-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,5 @@ handoff time, so a stale skill becomes stale guidance in the user's project.
and `--region` flags; corrects six programmatic API signatures; fixes the README's claim
that `stash init` ends in an agent-handoff menu (that belongs to `stash plan` / `stash impl`);
and marks `stash env` as the non-functional stub it currently is.

Superseded later in this release: the v2/Proxy commands and flags listed above are removed from both the CLI and bundled skill.
11 changes: 9 additions & 2 deletions docs/plans/cli-help-and-manifest.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# Proposal: a command-descriptor registry for `stash` help + a `manifest --json`

**Status:** proposal / for discussion
**Status:** implemented — the registry and `stash manifest --json` shipped.
Kept as the design record. **Note (2026-07-29):** the command names used as
examples below are a snapshot of the surface at proposal time. `db push`, `db
install`, `db upgrade`, and `db status` no longer exist (the installer is `stash
eql install`), and `stash encrypt cutover` was removed with EQL v2. Run `stash
manifest --json` for the current surface — that is the point of the registry.
**Area:** `packages/cli`
**Motivated by:** the docs V2 CLI reference, which is generated from the CLI
(cipherstash/docs#45). Today it parses `stash --help`; this proposal gives it —
Expand All @@ -20,7 +25,9 @@ and agents — a real structured source, and makes per-command help consistent.
2. **Per-command help is inconsistent.** `auth` implements its own `--help`
([`commands/auth/index.ts`](../../packages/cli/src/commands/auth/index.ts)),
but `stash eql install --help`, `stash db push --help`, etc. fall through to
the top-level help. There is no per-command help for most commands.
the top-level help. There is no per-command help for most commands. (`db
push` has since been removed — the drift it illustrates was real, and is
what the registry now prevents.)

3. **No machine-readable output.** Docs have to scrape `--help`, and agents
running `npx stash` have no authoritative, versioned command surface to read.
Expand Down
65 changes: 51 additions & 14 deletions docs/plans/encryption-migrations.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
# Encryption Migrations — Implementation Plan

> **Status (2026-07-29): historical. Parts of the surface below no longer exist.**
>
> This document is kept as the design record for `cs_migrations`, the manifest,
> and the backfill engine — all of which shipped and are current. What was
> retired with the EQL v2 removal:
>
> - **`stash encrypt cutover` and the whole `cut-over` phase.** EQL v3 has no
> rename swap. The lifecycle is `schema-added → dual-writing → backfilling →
> backfilled → dropped`: backfill, switch the application to the encrypted
> column *by name*, then drop the plaintext column (which is the original
> `<col>`, not a `<col>_plaintext` left behind by a rename).
> - **`stash db push`, `db install`, `db upgrade`, `db status`.** `db push` and
> its `eql_v2_configuration` DAO are gone entirely; the installer is now
> `stash eql install`.
> - **Every `eql_v2.*` function call.** `stash` no longer installs or drives EQL
> v2. Existing v2 ciphertext stays readable; it is not an authoring or rollout
> target.
>
> Read the sections below as "what we planned in Phase 1", not as a description
> of the current CLI. `skills/stash-cli/SKILL.md` and
> `skills/stash-encryption/SKILL.md` are the current lifecycle reference.

## Context

CipherStash today can encrypt a column at rest via EQL + either Stack/Protect.js (client-side) or the CipherStash Proxy (transparent). What it *doesn't* have is a first-class way to migrate an **existing plaintext column** into an encrypted one safely in production. EQL ships the schema/config primitives (`add_column`, `migrate_config`, `rename_encrypted_columns`) but no backfill orchestrator, no per-column phase tracking, and no resumable data mover. Today users have to wire this up themselves, which is both the biggest onboarding friction and the biggest correctness risk (partial backfills, reads on the wrong column, silent plaintext leaks).
Expand All @@ -10,6 +32,10 @@ This plan adds a shared migration substrate — CLI + library — that walks eac
schema-added → dual-writing → backfilling → backfilled → cut-over → dropped
```

*The shipped lifecycle drops `cut-over`: `schema-added → dual-writing →
backfilling → backfilled → dropped`. Status readers still display legacy
`cut_over` rows so old history remains printable.*

The same mechanism serves Stack and Proxy users. Phase 1 ships the status inspector and the backfill engine (the two pieces with no good existing workaround). The other phases get lightweight commands that mostly orchestrate existing EQL functions and delegate the code changes (e.g. wiring dual-writes into the persistence layer) to the agent handoff that `stash init` set up — Claude / Codex / AGENTS.md, with the relevant skills already installed.

## Scope (Phase 1)
Expand All @@ -19,8 +45,8 @@ The same mechanism serves Stack and Proxy users. Phase 1 ships the status inspec
3. A new `cs_migrations` table + small library (`@cipherstash/migrate` or co-located in `@cipherstash/stack`) that the CLI commands drive. Library is exported so users can embed backfill in their own workers/cron later without new infra.
4. `.cipherstash/migrations.json` repo manifest = intent (desired columns + index set + target phase). `stash encrypt plan` diffs intent vs. observed state.
5. Thin wrappers for the post-backfill phases so users can drive end-to-end from the CLI today, even if those phases are mostly pass-throughs:
- `stash encrypt cutover` — wraps `eql_v2.rename_encrypted_columns()` + `eql_v2.reload_config()` (via Proxy if present).
- `stash encrypt drop` — emits a migration file that drops `<col>_plaintext`.
- `stash encrypt cutover` — wraps `eql_v2.rename_encrypted_columns()` + `eql_v2.reload_config()` (via Proxy if present). *(Removed — see Status. There is no cut-over rename.)*
- `stash encrypt drop` — emits a migration file that drops `<col>_plaintext`. *(Shipped, but it drops the original `<col>`: with no rename, there is no `<col>_plaintext`.)*

**Out of Phase 1:** Proxy-mode backfill (Phase 2), CS-hosted backfill runner (Phase 3), upstreaming `cs_migrations` into EQL as `eql_v2_migrations` (Phase 3), `stash encrypt update` for re-encrypting an already-cut-over column with new EQL config (next change).

Expand Down Expand Up @@ -131,7 +157,12 @@ Backfill now owns the bookmark. The first time backfill runs against a column, i

If the user lies (says yes but dual-writes aren't actually live), rows inserted during the backfill land in plaintext only and the recovery is `stash encrypt backfill --force`, which drops the `<col>_encrypted IS NULL` guard and re-encrypts every row regardless of current state. Audit trail: the `force` run is recorded with `details.force = true` in `cs_migrations` so it shows up in `encrypt status` history as a distinct event.

### 5. `stash encrypt cutover`
### 5. `stash encrypt cutover` *(never shipped in this form; removed)*

> Nothing in this section exists. The command, the `cut-over` phase transition,
> and every `eql_v2.*` call below were removed with EQL v2. In EQL v3 the
> application moves to the encrypted column by name after backfill — no rename,
> no config promotion, no Proxy reload.

For each column in `backfilled` phase, in a single transaction:

Expand All @@ -155,6 +186,10 @@ Record `cut_over` event. App's existing `SELECT email FROM users` returns the en

### 6. `stash encrypt drop`

> Shipped, but gated on `backfilled` (not `cut_over`, which no longer exists)
> and it drops the original `<col>`. It also re-verifies coverage under an
> `ACCESS EXCLUSIVE` lock inside the generated migration.

For columns in `cut_over` phase:

1. Read Drizzle / Prisma / other migration tooling from repo (we already detect this in init).
Expand All @@ -168,17 +203,17 @@ For columns in `cut_over` phase:
- `status.ts` — new
- `plan.ts` — new (diffs intent vs. observed)
- `backfill.ts` — new (also handles the dual-write confirmation + `--force` recovery path)
- `cutover.ts` — new
- `cutover.ts` — new *(never shipped / removed)*
- `drop.ts` — new
- `stack/packages/cli/src/bin/stash.ts` — register `encrypt` subcommand (analogous to existing `db` registration at ~line 237)
- `stack/packages/migrate/` — **new package** (library the CLI drives)
- `src/state.ts` — `cs_migrations` DAO (append event, get latest, get progress)
- `src/backfill.ts` — the chunked loop, exported as `runBackfill({ table, column, client, db, chunkSize, signal })`
- `src/cursor.ts` — keyset pagination primitive
- `src/eql.ts` — thin wrappers over `eql_v2.*` functions (rename, reload, config read)
- `src/eql.ts` — thin wrappers over `eql_v2.*` functions (rename, reload, config read) *(never shipped; `@cipherstash/migrate` has no `eql_v2` wrappers — see `src/version.ts` for the v3 domain classifier that replaced this idea)*
- `src/manifest.ts` — read/write `.cipherstash/migrations.json`
- `src/schema.sql` — `cs_migrations` DDL, installed by `db install` or an explicit `encrypt install` step
- `stack/packages/cli/src/commands/db/install.ts` — extend to install `cs_migrations` schema alongside EQL
- `src/schema.sql` — `cs_migrations` DDL, installed by `db install` or an explicit `encrypt install` step *(`db install` is now `stash eql install`)*
- `stack/packages/cli/src/commands/db/install.ts` — extend to install `cs_migrations` schema alongside EQL *(moved: the installer is `commands/eql/install.ts`)*
- `stack/packages/cli/src/commands/init/lib/introspect.ts` — `introspectDatabase` lives here post-#395; `status.ts` reuses it via direct import
- `stack/packages/cli/src/config/` — extend `stash.config.ts` loader so backfill subprocess can dynamically import user's encryption client
- `stack/packages/cli/package.json` — add `@cipherstash/migrate` dep
Expand All @@ -190,8 +225,8 @@ For columns in `cut_over` phase:
- `introspectDatabase` in `packages/cli/src/commands/init/lib/introspect.ts` (moved here from the old wizard package as part of the #395 init handoff work).
- `loadStashConfig` + dynamic encryption-client import lives at `packages/cli/src/config/`. Re-export from `@cipherstash/migrate` so library consumers don't need a hidden cross-package import.
- `rewriteEncryptedAlterColumns` in `packages/cli/src/commands/db/rewrite-migrations.ts` — the phase-1 schema-add is already solved by drizzle-kit + this rewriter. The new commands **will not** re-solve it.
- EQL functions (Postgres): `eql_v2.add_column`, `eql_v2.add_search_config`, `eql_v2.migrate_config`, `eql_v2.activate_config`, `eql_v2.rename_encrypted_columns`, `eql_v2.reload_config`, `eql_v2.count_encrypted_with_active_config`, `eql_v2.select_pending_columns`, `eql_v2.ready_for_encryption`.
- `db push` in `packages/cli/src/commands/db/push.ts` — already handles writing to `eql_v2_configuration`; reuse the DAO.
- EQL functions (Postgres): `eql_v2.add_column`, `eql_v2.add_search_config`, `eql_v2.migrate_config`, `eql_v2.activate_config`, `eql_v2.rename_encrypted_columns`, `eql_v2.reload_config`, `eql_v2.count_encrypted_with_active_config`, `eql_v2.select_pending_columns`, `eql_v2.ready_for_encryption`. *(None of these are called any more — `stash` installs and drives EQL v3 only. EQL v3 needs no configuration table: the domain types carry the config.)*
- `db push` in `packages/cli/src/commands/db/push.ts` — already handles writing to `eql_v2_configuration`; reuse the DAO. *(Both the command and that file were deleted; there is no `packages/cli/src/commands/db/push.ts` to read.)*

## Verification

Expand All @@ -201,15 +236,15 @@ For columns in `cut_over` phase:
- Manifest reader: schema validation, drift detection.
2. **Integration (Drizzle, local Postgres)**
- Seed 100k-row `users` table with plaintext `email`.
- `stash db install` → EQL + `cs_migrations` installed.
- `stash db install` → EQL + `cs_migrations` installed. *(now `stash eql install`)*
- Manually wire dual-write in the test app's insert code (simulates user + agent handoff).
- `stash encrypt backfill --table users --column email` → interactive prompt confirms dual-writes are deployed, appends `dual_writing` event, runs to completion; progress output sane; `COUNT(*) WHERE email_encrypted IS NULL` = 0.
- Same flow non-interactively: `--confirm-dual-writes-deployed` accepted, loud warning printed.
- Kill mid-backfill (SIGINT) → re-run with `--resume` → completes without duplicate encryption; `cs_migrations` shows continuous cursor progression; no second `dual_writing` event.
- `stash encrypt backfill --force` after manually corrupting an encrypted row → re-encrypts every row; `details.force = true` recorded in `cs_migrations`.
- `stash encrypt status` → shows `backfilled`.
- `stash encrypt cutover` → rename executes; app (still running, reads `email`) now gets decrypted ciphertext transparently.
- `stash encrypt drop` → migration file emitted; apply; `email_plaintext` gone.
- `stash encrypt cutover` → rename executes; app (still running, reads `email`) now gets decrypted ciphertext transparently. *(Removed. The v3 equivalent is a deploy: point the app at `email_encrypted` and decrypt through the encryption client.)*
- `stash encrypt drop` → migration file emitted; apply; `email_plaintext` gone. *(Drops `email`, the original plaintext column.)*
3. **Idempotency**
- Run `backfill` twice with no kill — second run does 0 writes.
- Concurrent runners on two shells — both converge, no duplicate writes, no missed rows.
Expand Down Expand Up @@ -237,8 +272,10 @@ For columns in `cut_over` phase:

## Open items flagged (decisions already made)

> Two of these were later reversed by the move to EQL v3 — marked inline.

- Phase 1 runtime mode = Protect/Stack client-side only.
- Phase 4 default cutover mechanism = `eql_v2.rename_encrypted_columns()` (transparent to app code).
- State store = repo manifest + `eql_v2_configuration` (EQL intent) + new `cs_migrations` table (runtime state).
- Phase 4 default cutover mechanism = `eql_v2.rename_encrypted_columns()` (transparent to app code). *(Reversed: no cut-over mechanism at all.)*
- State store = repo manifest + `eql_v2_configuration` (EQL intent) + new `cs_migrations` table (runtime state). *(Reversed: EQL v3 carries intent in the column's domain type, so there is no configuration table.)*
- Phase 1 shipping scope = status + backfill first-class; other phases as thin wrappers.
- `cs_migrations` is CLI-owned for now, explicitly designed to be upstreamed into EQL as `eql_v2_migrations` in a later release so both Stack and Proxy own it jointly.
Loading