Skip to content

test(BA-7836): add scenario tests for the prometheus query preset adapter - #14518

Draft
jopemachine wants to merge 11 commits into
mainfrom
test/BA-7836-preset-scenarios
Draft

test(BA-7836): add scenario tests for the prometheus query preset adapter#14518
jopemachine wants to merge 11 commits into
mainfrom
test/BA-7836-preset-scenarios

Conversation

@jopemachine

@jopemachine jopemachine commented Sep 11, 2026

Copy link
Copy Markdown
Member

Resolves BA-7836

Summary

  • Replace the TODO stub in api/adapters/prometheus_query_preset/KNOWLEDGE.md with the scenarios the adapter guarantees — 8 calls / 54 rows — in the shape api/adapters/domain/KNOWLEDGE.md and api/adapters/deployment/KNOWLEDGE.md (test(BA-7806): add scenario tests for the deployment adapter's own calls #14501) set.
  • Transcribe every row into tests/scenario/bai_scenario/manager/prometheus_query_preset/ (8 modules) and add the report.md split from the CI run. Every public call of the adapter is called by at least one row.
  • Add SeedPreset / SeedPresetIn (rows laid through the creator alone, so a template the renderer would refuse can still be stored), FakePrometheusClient, and a SeedCategory of its own for the rows that file a preset under a category. The category seed and the caller helper (lay_someone) are duplicated in test(BA-7835): add scenario tests for the prometheus query preset category adapter #14514 so the two PRs land independently; whichever merges second can drop its copy.

How the Prometheus rows are verified without a runner change: the stand-in answers every query with one sample whose value is the PromQL it was asked, a matrix for a range query and a vector for an instant one, and refuses an empty query the way Prometheus does (parse error: no expression found). A row therefore states the exact query that reached Prometheus — avg by () (rate(container_cpu_seconds_total{}[1h])) — and the "Prometheus refuses" rows run a template that renders to an empty query when no label is given.

What the rows pin, worth a reviewer's eye:

  • The preset is global, so no role ever reaches one: execute / update / purge pass only for a superadmin, or for anyone once enforcement is switched off.
  • Reading a preset is open to every authenticated caller while executing it is entity-gated — a user who can read a definition cannot run it.
  • Preview is the only role-gated read here, so the monitor role passes it and is refused on create.
  • The window falls back request → preset → server default, and the window and the time range are orthogonal: the window lands in the query, the range only picks a range query over an instant one.
  • Template validation runs only when the request carries a template: a stored template the renderer refuses does not block renaming.
  • Changing only the filter labels keeps the group labels, though both share one column.
  • A preset name is not unique, and an unknown category id is refused by the storage-level foreign-key violation, not a domain error.
  • The three container-metric wirings of the metric concern (global_search_container_metrics, public_search_container_metric_metadata, search_user_container_metrics) have no adapter and are consumed by legacy GraphQL only, so they are named in the document and kept out of the tables.

Judgment calls

Decisions a reviewer may want to overturn; none of them is settled by a rule I could find.

Where What I decided The alternative
fakes/prometheus.py The stand-in echoes the rendered PromQL as the sample value, so then can read what reached Prometheus off the answer. It also runs the real renderer, so a template the client could not render is refused in the fake as in the client. Record only and answer a fixed sample, leaving the window / label / range rows to BA-7830 (the runner handing fakes to then).
fakes/prometheus.py An empty rendered query is refused with FailedToGetMetric, modelled on Prometheus's own 400 for an empty query; the "Prometheus refuses" rows lean on it. Wait for a per-row way to script the stand-in.
KNOWLEDGE.md, opening "No role reaches a preset" is stated for roles only; lending a preset into a scope through an entity share is scoped out to the share adapter's document. I did not verify whether a shared preset would pass the entity gate. Verify the share path and either write the granted-user row or say it cannot exist.
test_creating.py The duplicate-name row pins that preset names are not unique (there is only a non-unique index). If that is an omission rather than a design, this documents a bug as a guarantee. Add the constraint and flip the row.
test_creating.py, test_editing.py An unknown category id expects the storage-level ForeignKeyViolationError, because nothing maps it to a domain error. Map it first and expect that.
test_executing.py The not-found row for execute expects PrometheusQueryPresetNotFound (the service reads the preset itself) while update / purge expect EntityNotFoundError (the ops path); the asymmetry is pinned as is. Unify the two first.
test_creating.py, test_previewing.py Two rows for the monitor role (refused on create, passes on preview), since the superadmin gate lets that role through reads. No sibling table has monitor rows. Drop them as outside the four actors the investigation format names.
test_reading.py, test_searching.py Two rows call the adapter with no user context and expect the authentication refusal (UserNotFound). Drop them as a validator concern.
test_editing.py, test_executing.py, test_retiring.py Enforcement-off rows state that any authenticated user may edit, run and purge a global preset once enforcement is off. Same shape as the domain and deployment tables. Leave the switch's effect undocumented.
components/prometheus_query_preset.py The batch read compares each answered node whole against a node built from the seeded row (node_of), which mirrors the adapter's _data_to_dto field list in test code. Compare ids only, at the cost of the "whole answer" rule.
KNOWLEDGE.md The entity is called "정의" in the Korean text (the DTO says QueryDefinition, the code says preset). "프리셋".

Split out of #14508. Independent of #14514 (prometheus_query_preset_category), which covers the sibling entity.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5

@github-actions github-actions Bot added the size:XL 500~ LoC label Sep 11, 2026
jopemachine added a commit that referenced this pull request Sep 11, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
jopemachine added a commit that referenced this pull request Sep 11, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@jopemachine
jopemachine force-pushed the test/BA-7836-preset-scenarios branch from a27e021 to 6e6c77b Compare September 11, 2026 03:59
jopemachine added a commit that referenced this pull request Sep 11, 2026
Split from the CI run of #14518 after the stand-in began echoing the query,
where all 54 preset rows passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
jopemachine and others added 5 commits September 11, 2026 13:13
…pter

Replace the TODO stub in the preset adapter's KNOWLEDGE.md with the 55 rows
the adapter guarantees (8 calls), transcribe 45 of them into
tests/scenario/bai_scenario/manager/prometheus_query_preset/, and add the
report.md split from the CI run of #14508, where every row passed.

Add SeedPreset and SeedPresetIn, which lay rows through the creator alone so
a template the renderer would refuse can still be stored, and
FakePrometheusClient, which answers one sample for every query and keeps what
it was asked. The rows that read what the stand-in received, or need it to
answer an error, wait for the runner to hand fakes to `then` (BA-7830).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
The stand-in now answers every query with one sample whose value is the PromQL
it was asked, a matrix for a range query and a vector for an instant one, and
refuses an empty query as Prometheus does. That lets the rows on the window
fallback, the range, the labels reaching the query and Prometheus refusing a
query run against the answer, so the preset table covers every row of its
document (54 over 8 modules).

Also list the scenarios at module level as the other tables do, and say in the
document that lending a preset into a scope by a share is not this table's
concern.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
Split from the CI run of #14518 after the stand-in began echoing the query,
where all 54 preset rows passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
The preset table laid its categories and its caller through the category
table's seed and helper, which stacked this branch on #14514. It now carries a
category seed and the caller helper of its own, so the two land independently;
whichever merges second can drop its copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
@jopemachine
jopemachine force-pushed the test/BA-7836-preset-scenarios branch from 2dc9d33 to d3ff86e Compare September 11, 2026 04:15
@jopemachine
jopemachine changed the base branch from test/BA-7835-preset-category-scenarios to main September 11, 2026 04:15
jopemachine and others added 6 commits September 11, 2026 13:20
The preset has no soft delete, only a purge, so the module the domain and
deployment tables call retiring is purging here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
… user

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sm7zNnhABQq1BSebraixh5
…set scenarios for readability

The knowledge document, the component docstrings and every scenario sentence
used a terse, metaphorical style (문, 줄, 훑기, 심다, 실리다, 대역, 창, 묶음)
that did not read at a glance. Rewrite them in plain technical Korean:
프리셋/카테고리 for the entities, 모의 서버 for the fake Prometheus, 시간 창 and
그룹 라벨 for the window and group labels, 생성/조회/검색/수정/삭제 for the
operations, 권한 검사 for enforcement, 미리 만들어 둔 for seeded data, and
역할 부족으로 거부 for the role refusals. report.md follows from the CI run in
the next commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQiEEB2mfJtFWnMnHLDFnT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500~ LoC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant