Skip to content

perf(desktop): index deleted observations - #954

Merged
r0ssing merged 1 commit into
OpenDataEnsemble:devfrom
HelloSapiens:perf/953-desktop-deleted-column
Oct 1, 2026
Merged

r0ssing merged 1 commit into
OpenDataEnsemble:devfrom
HelloSapiens:perf/953-desktop-deleted-column

Conversation

@r0ssing

@r0ssing r0ssing commented Oct 1, 2026

Copy link
Copy Markdown
Member

What & Why

Make the Desktop tombstone flag a first-class, indexed SQLite column instead of requiring JSON extraction from observation_extras.

This lets structured queries and exports exclude deleted observations in SQLite before payloads are read or deserialized. It also fixes post-pagination deletion filtering in form preview and makes payload-index and overview behavior consistent for tombstones.

Changes

  • add and backfill observations.deleted for existing profile databases
  • add a composite (deleted, form_type) index
  • keep observation_extras.deleted as a compatibility mirror while making the column authoritative
  • update sync, import, save, push, and row-mapping paths
  • filter deleted observations and requested forms in export SQL
  • use the first-class column in Desktop/shared query compilers
  • pass includeDeleted into native form-preview pagination
  • exclude tombstones from overview statistics and payload indexes
  • migrate profile databases when local CLI/MCP export runs before Desktop has opened them
  • fall back to payload JSON filtering for include-deleted queries because tombstones intentionally have no payload-index rows

Testing

  • cd desktop/src-tauri && cargo fmt --check && cargo test
  • cd desktop && pnpm run lint && pnpm run format:check && pnpm test -- --run && pnpm typecheck
  • cd packages/observation-query && CI=true pnpm test && CI=true pnpm exec tsc --noEmit
  • MCP baseline export against ODE AI test: one household_visit row
  • source-built CLI baseline export: one household_visit row
  • after manually marking the observation deleted in the branch build:
    • MCP export: zero rows, household_visit in formsWithoutRows
    • source-built CLI export: zero rows, household_visit in formsWithoutRows

Desktop lint reports two pre-existing react-hooks/exhaustive-deps warnings and no errors.

Closes #953

@r0ssing
r0ssing requested a review from najuna-brian October 1, 2026 16:07
@r0ssing
r0ssing merged commit 3807a61 into OpenDataEnsemble:dev Oct 1, 2026
19 checks passed
@r0ssing
r0ssing deleted the perf/953-desktop-deleted-column branch October 1, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(desktop): make deleted observations first-class

1 participant