Skip to content

Data lake: named queries over synthetic datasets, refused before they run - #19

Merged
gadzbi123 merged 1 commit into
mainfrom
datalake-named-queries
Oct 3, 2026
Merged

gadzbi123 merged 1 commit into
mainfrom
datalake-named-queries

Conversation

@gadzbi123

Copy link
Copy Markdown
Contributor

Replaces #18 (closed when its base branch was deleted after #16 merged). The first demo from the plan: named queries, unauthorised datasets rejected before execution, labels verified before release, unknown classes blocked, neutral denials, lower-class views only through an explicit transformation.

What

  • query_datalake takes only a query name from datalake.queries in policy/policy.yaml. Synthetic datasets live in demo/datalake.py: fx_reference (public), client_positions (P2), deal_pipeline (DP30).
  • The new access.datalake control refuses a call before it runs if the query is unknown, the role may not run it, or its result could not be sent on (above the model's limit, or above Jev's while the AI check is on). Every refusal has the same reason, so it reveals nothing about which datasets exist, their class or who may see them.
  • A result's class comes from the query the gateway let through, not from the arguments the agent echoes back. A missing label, or one that differs from the catalog, leaves the result unclassified, so it is withheld.
  • Lower-class views exist only as declared transformations. pipeline_by_sector gives the deal team counts per sector (groups under 3 dropped) as internal, never the deals. The public side gets no pipeline view at all, because even a count of pending deals is MNPI to them.
  • Policy validation: unknown datasets, transformations or classes in the catalog keep the last good policy.
  • query_datalake is added to the onboarding demo tools and to the onboarding_analyst and mna_banker roles.

Tests

tests/test_datalake.py (14 tests): refusals before execution, identical neutral reasons, the derived view leaks no deal and no small group, label mismatch / missing label / swapped query in history are withheld, live grant, bad catalog keeps the last good policy. Without the engine change 10 of 14 fail (the other 4 test the policy file and the demo lake). All committed tests pass.

🤖 Generated with Claude Code

… run

query_datalake takes only a query name from the `datalake` catalog in policy.yaml
(demo/datalake.py holds the synthetic datasets). Unknown queries, queries the role may
not run, and queries whose result could not be sent on (model limit, or Jev's while the
AI check is on) are refused before they run, with one neutral reason. A result's class
comes from the query the gateway let through; a missing or different label withholds it.
Lower-class views exist only as declared transformations (sector counts for the deal
team, small groups dropped). tests/test_datalake.py; decisions.md D7.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gadzbi123
gadzbi123 merged commit b4169a1 into main Oct 3, 2026
1 check passed
@gadzbi123
gadzbi123 deleted the datalake-named-queries branch October 3, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant