Skip to content

test(BA-7805): add scenario tests for the prometheus query preset adapters - #14508

Closed
jopemachine wants to merge 6 commits into
mainfrom
docs/BA-7805-metric-scenarios
Closed

test(BA-7805): add scenario tests for the prometheus query preset adapters#14508
jopemachine wants to merge 6 commits into
mainfrom
docs/BA-7805-metric-scenarios

Conversation

@jopemachine

@jopemachine jopemachine commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Resolves BA-7805

Summary

  • Replace the TODO stubs in api/adapters/prometheus_query_preset/KNOWLEDGE.md and api/adapters/prometheus_query_preset_category/KNOWLEDGE.md with the scenarios each adapter guarantees — 8 calls / 55 rows for the preset, 5 calls / 19 rows for the category — in the shape api/adapters/domain/KNOWLEDGE.md and api/adapters/deployment/KNOWLEDGE.md (test(BA-7806): add scenario tests for the deployment adapter's own calls #14501) set.
  • Transcribe them into tests/scenario/bai_scenario/manager/prometheus_query_preset/ (45 rows over 8 modules) and .../prometheus_query_preset_category/ (18 rows over 5 modules). Every public call of both adapters is called by at least one row.
  • Add SeedPreset / SeedPresetIn / SeedCategory (rows laid through the creators alone, so a template the renderer would refuse can still be stored) and FakePrometheusClient (answers one sample for every query and keeps what it was asked).

What the rows pin, worth a reviewer's eye:

  • Both entities are global, so no role ever reaches a single preset or category: the entity-gated calls pass only for a superadmin, or for anyone once enforcement is switched off.
  • Reading a preset is open to every authenticated caller while executing it is entity-gated — a user who can read a definition cannot run it.
  • Preview is the only role-gated read here, so the monitor role passes it and is refused on create.
  • Template validation runs only when the request carries a template: a stored template the renderer refuses does not block renaming.
  • Changing only the filter labels keeps the group labels, though both share one column.
  • A preset name is not unique while a category name is; neither the unique nor the foreign-key violation is mapped to a domain error, so the rows expect the storage-level errors.
  • The three container-metric wirings of the metric concern (global_search_container_metrics, public_search_container_metric_metadata, search_user_container_metrics) have no adapter and are consumed by legacy GraphQL only, so they are named in the preset document and kept out of the tables.

Not covered here:

Rows Why
execute: instant vs range query, the window fallback (request → preset → server default), the allowed labels reaching the query, unrestricted labels; preview: the server default window read what the stand-in received — run_scenario does not hand fakes to then (BA-7830)
execute / preview: Prometheus refusing the query need the stand-in scripted per row, which the runner has no channel for yet
category: purging a category empties the category of the presets pointing at it then cannot read the presets back after the call

Test plan

  • python3 scripts/knowledge/check.py passes
  • pants lint / pants check on tests/scenario::
  • The scenario job of CI runs both components green (45 + 18 rows)
  • Generate the two report.md with scripts/scenario-report.py --split from that run and commit them

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5

jopemachine and others added 2 commits September 11, 2026 11:15
@jopemachine @claude
doc(BA-7805): fill the prometheus query preset adapter scenarios
86e5412 
Replace the TODO stubs in the prometheus_query_preset and
prometheus_query_preset_category adapter KNOWLEDGE.md files with the
scenarios each adapter guarantees, in the shape the domain and deployment
documents set. Every public call carries at least one row: 8 calls / 56 rows
for the preset, 5 calls / 19 rows for the category.

The rows pin what the code does today: the catalog reads open to every
authenticated caller, the superadmin role on create and preview (with the
monitor role passing the read), the entity gate on execute, update and purge
that no role can reach because both entities are global, the window fallback
and the label checks on execute, and the purge of a category emptying the
category of the presets pointing at it.

The three container-metric wirings of the metric concern have no adapter and
are consumed by legacy GraphQL only, so they are named in the preset document
and left out of the tables.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@jopemachine @claude
changelog: add news fragment for PR #14508
15eda4c 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@github-actions github-actions Bot added the size:L 100~500 LoC label Sep 11, 2026
@jopemachine @claude
test(BA-7805): add scenario tests for the prometheus query preset ada…
e0afd04 
…pters

Transcribe the rows of the two KNOWLEDGE.md documents into
tests/scenario/bai_scenario/manager/prometheus_query_preset/ (45 rows over 8
modules) and .../prometheus_query_preset_category/ (18 rows over 5 modules).

Add SeedPreset, SeedPresetIn and SeedCategory, which lay rows through the
creators alone so a template the renderer would refuse can still be stored,
and FakePrometheusClient, which answers one sample for every query and keeps
what it was asked. The rows that read what the stand-in received, or need it
to answer an error, wait for the runner to hand fakes to `then` (BA-7830).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@github-actions github-actions Bot added size:XL 500~ LoC and removed size:L 100~500 LoC labels Sep 11, 2026
@jopemachine @claude
changelog: retype the news fragment of PR #14508 as a test change
3da25e6 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@jopemachine jopemachine changed the title doc(BA-7805): fill the prometheus query preset adapter scenarios test(BA-7805): add scenario tests for the prometheus query preset adapters Sep 11, 2026
jopemachine and others added 2 commits September 11, 2026 11:49
@jopemachine @claude
test(BA-7805): add the prometheus query preset scenario reports
b99a607 
Split from the CI run of #14508, where all 45 preset rows and 18 category
rows passed. The one call the run leaves alone, batch_load_fields, is the
base adapter's field-row read; both documents now say so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@jopemachine @claude
test(BA-7805): sort the scenario imports the way the changed-files li…
752774f 
…nt does

Ruff places `bai_scenario` with the third-party block when it lints the
changed files alone, which is how CI runs it; a whole-tree run sorts it as
first-party instead. The four new modules follow the order CI checks, as the
deployment scenario modules do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
This was referenced Sep 11, 2026
Draft
Draft
@jopemachine

jopemachine commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Split by entity into #14514 (prometheus_query_preset_category, BA-7835) and #14518 (prometheus_query_preset, BA-7836), stacked in that order.

@jopemachine
jopemachine deleted the docs/BA-7805-metric-scenarios branch September 11, 2026 03:34
jopemachine added a commit that referenced this pull request Sep 11, 2026
@jopemachine @claude
test(BA-7836): add scenario tests for the prometheus query preset ada…
23570c3 
…pter

Replace the TODO stub in the preset adapter's KNOWLEDGE.md with the 55 rows
the adapter guarantees (8 calls), transcribe 45 of them into
tests/scenario/bai_scenario/manager/prometheus_query_preset/, and add the
report.md split from the CI run of #14508, where every row passed.

Add SeedPreset and SeedPresetIn, which lay rows through the creator alone so
a template the renderer would refuse can still be stored, and
FakePrometheusClient, which answers one sample for every query and keeps what
it was asked. The rows that read what the stand-in received, or need it to
answer an error, wait for the runner to hand fakes to `then` (BA-7830).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
jopemachine added a commit that referenced this pull request Sep 11, 2026
@jopemachine @claude
test(BA-7836): add scenario tests for the prometheus query preset ada…
e130b84 
…pter

Replace the TODO stub in the preset adapter's KNOWLEDGE.md with the 55 rows
the adapter guarantees (8 calls), transcribe 45 of them into
tests/scenario/bai_scenario/manager/prometheus_query_preset/, and add the
report.md split from the CI run of #14508, where every row passed.

Add SeedPreset and SeedPresetIn, which lay rows through the creator alone so
a template the renderer would refuse can still be stored, and
FakePrometheusClient, which answers one sample for every query and keeps what
it was asked. The rows that read what the stand-in received, or need it to
answer an error, wait for the runner to hand fakes to `then` (BA-7830).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

@jopemachine jopemachine

Labels

size:XL 500~ LoC

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jopemachine