Skip to content

Share the metrics response shape between REST API and MCP server - #316

Merged
mschwoer merged 117 commits into
refactor_db_XVIIIfrom
refactor_db_XIX
Nov 19, 2026
Merged

mschwoer merged 117 commits into
refactor_db_XVIIIfrom
refactor_db_XIX

Conversation

@mschwoer

Copy link
Copy Markdown
Contributor

Stacked on #315.

  • One metrics response shape for the REST API and the MCP server
  • Migration of legacy metric keys
  • Unit tests for the MCP server

🤖 Generated with Claude Code

mschwoer and others added 2 commits November 11, 2026 18:09
…ric key migration

---
# Conversation that produced these changes
---

## User prompt

"now help me resolve these code duplications

  - 4.3. mcp_server/server.py:203: Metrics flattening is duplicated in webapp/service/data_handling.py::_merge_metrics_by_type. GRADIENT_LENGTH_KEY is defined twice (mcp_server/server.py:68, monitoring/alerts/pump_pressure_alert.py:30).

  - 1.1 Presenting a raw file with its metrics, three shapes. The REST API returns a list of dicts, one per metrics type. The MCP server flattens all types into one dict and prefixes a key with the type only on a collision. The webapp prefixes every key with the type. Each client has its own
    loop over the same mapping to do this.

  - 1.2 The metrics metadata fields are listed twice. The MCP server and the webapp each spell out settings_name, settings_version, relative_output_path and output_display_base_path to put them next to the values.

  - 1.3 The display output path of a metrics entry is computed twice. The REST API and the webapp's output folder list both do "none if there is no relative path, else get_display_output_path(...)".

  - 1.4 "Drop unset fields" is written four times, in the REST API (twice) and the MCP server (twice), as the same dict comprehension over vars().

  - 2.1 Legacy metric names are harmonized in two places with different rules. The backend renames two legacy keys for everyone. The webapp additionally strips every msqc_ prefix and renames quanting_time_elapsed. So for old files the webapp shows evosep_... while the REST API and the MCP
    server return msqc_evosep_... for every key except the one the backend knows. I left this in slice 6b on purpose, because moving it into the backend changes REST and MCP keys for old files. It is a decision, not a cleanup."

## Clarifying round 1

Q: 2.1 Legacy metric names: old MSQC metrics are stored with an 'msqc_' prefix (current calculators no longer write it), and 'quanting_time_elapsed' is the old name of 'time_elapsed'. Only the webapp strips/renames these today. Where should this harmonization live?
  - Backend, for all (Recommended)
  - Leave as is
  - Migrate the data  <-- chosen

Q: 1.1 Three output shapes of 'raw file with metrics' (REST: list of dicts per type; MCP: one flat dict, type prefix only on key collision; webapp: every column prefixed '<type>__'). The shared part (metadata + values per type) gets one helper either way. Should the shapes themselves change?
  - Keep the three shapes (Recommended)
  - MCP always prefixes
  - MCP like REST  <-- chosen

Q: 1.4 'Drop unset fields' exists four times, but the four differ (plain; with excluded keys; with a whitelist; with value formatting). One shared helper 'fields without None' could be the base of all four, saving about one line each. Worth it?
  - Skip it (Recommended)  <-- chosen
  - Add the helper

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@claude

claude Bot commented Nov 12, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

mschwoer and others added 26 commits November 12, 2026 21:55
…onstraint

CI installs all requirements into one environment. fastapi==0.115.0 downgraded
starlette below 0.39, the MCP requirements then upgraded it again, and
fastapi 0.115.0 no longer imported: collecting rest_api/tests/test_routes.py
failed and no test ran.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
---
# Conversation that produced these changes
---

## User prompt

"Now create an implementation plan. Refactorings upfront, then several human-reviewable slices."

## Clarifying round 1

Q: Decision 4.4 of VALIDATION_DESIGN.md: where do the validation rules for writes live? This determines the shape of every slice.
  - shared.db API modules (Recommended)  <-- chosen
  - New shared/services

Q: Decision 4.1: how do clients see validation errors from the pydantic input types?
  - One domain error (Recommended)  <-- chosen
  - Two error types

Q: Decision 4.3: does the plan include write tools for the MCP server? Today the webapp is the only writing client.
  - No, leave out (Recommended)
  - Yes, final slice  <-- chosen

Q: Decision 4.5: forms and Mongo models have different max lengths (settings file names and software 64 vs 128, project description 256 vs 512, messenger id 64 vs 512). Which value becomes the single declared limit?
  - The form's (Recommended)  <-- chosen
  - The model's

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- add the input type helpers (shared/domain/inputs.py): `InputModel`, `parse`,
  `required`, `RequiredStr`, `OptionalStr`, `max_length`
- `NewSuppression` replaces `validate_suppression` of the admin page
- `toast_result` lists several validation errors

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `NewUser` and `UserUpdate` replace the cleaning on the admin page
- emails are stored and looked up stripped and lower-cased
- `set_status` rejects the current and unknown statuses
- the initials and email uniqueness checks query the backend
- the form limits are read from the input type

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `NewProject` replaces the project id check of the projects page,
  the id rule is one regex
- a project needs an active owner, if one is given
- only active projects can be archived or deleted
- the form limits are read from the input type

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…k in shared

- `check_config_params` is the one check of unresolved `config_params`,
  used by the settings page and at job time; its helpers are private
- move the runner check to shared/runner_checks.py, it also rejects an unknown runner
- remove the unused `allow_absolute_paths` of `check_for_malicious_content`

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `NewSettings` replaces the validation on submit of the settings page
- `settings.create` checks the runner and the owner, reports all rule errors
  together and archives the previous version itself (`archive_previous`)
- only active settings can be archived or deleted
- the form limits are read from the input type
- remove `empty_to_none` of the webapp, the input types do it

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- `NewAssignment` validates scopes and file name filters
- `assign`, `unassign` and `update_to_latest` take `allow_fallback`: the caller
  states whether it may change the assignments of the fallback project
- `assign` reports all rule errors together
- `update_to_latest` points the assignment to the latest version in one write
- `known_scopes` is the one list of scopes, also for the form

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- unknown scopes: neither the default scope, a vendor nor an instrument of the yaml
- scopes whose instruments are all excluded
- the default scope excluded, a scope both included and excluded
- file name filters that always match an exclude filter

`update_to_latest` does not re-check the stored scopes and filters.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A scope or file name filter that would never apply is rejected even if
other entries of the assignment still apply. Messages and docstrings said
"they would never apply", which read as the whole assignment.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`fully_excluded_scopes` already rejects an excluded default scope and a scope
that is both included and excluded, for every scope with an instrument in the
yaml. Remove the two rules from `NewAssignment`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A scope in both `scopes` and `excluded_scopes` was listed twice.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`get_instrument_types` in shared/yamlsettings.py replaces the separate reads
in the scope resolver and the projects, overview and admin pages.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`KNOWN_VENDOR_NAMES` becomes `KNOWN_INSTRUMENT_TYPES`; comments, docstrings,
test names, an error message and the texts of the projects page follow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The page only knows the active versions, so "current + 1" was wrong whenever
a newer version was archived or deleted. Add a test that the archive checkbox
names the version `settings.create` archives, the latest active one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…arams

The gap is in that check, which every client and the processor run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The field declares the lengths, the regex only the characters. A too short
id now gets the length message, like a too long one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Project.id and User.initials mixed a constant with a literal.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Only the projects page left it out; other clients could archive or delete it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
mschwoer and others added 28 commits November 19, 2026 13:10
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The record, with its possibly large file info, was loaded only to log its old status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The migration left such keys alone, and since the webapp no longer merges the
duplicate columns, those documents kept an extra column. Keys with another
value are still left alone and reported. Add the first tests of the migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_to_raw_file_dict` and `_to_dict` differed only in the excluded keys; the
metrics and the backup path are added in one pass over the raw files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The error list is the same, as the project errors are collected before; the
else branch and the type ignore on the version go away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard against an empty id list was redundant, as `id__in=[]` matches
nothing, and the backend sort was repeated by the facade, whose project query
returns the newest first anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Status records truncated too long details in the facade function, raw files in
their input type. `KrakenStatusUpdate` truncates them the same way and checks
the other field limits; the limit tests compare it with the model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NaN is not equal to itself, so a duplicate key with a NaN value was kept and
reported as a conflict. Also find the key holding the current name in one
expression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both were 1024 and fed the same truncation; nothing meant them to diverge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reject settings assignments that can never apply
validate settings assignments with an input type in shared.db
validate new settings with an input type in shared.db
prepare the settings validation: one config_params check, runner chec…
validate new projects with an input type in shared.db
validate users with input types in shared.db
Write validation: toast_result in the webapp, drop the webapp read filter check
Write validation: regex constants to shared/domain, one function per composite write
Write validation: design and implementation plan
validate alert suppressions with an input type in shared.db
@mschwoer
mschwoer merged commit 8984f4e into refactor_db_XVIII Nov 19, 2026
4 checks passed
@mschwoer
mschwoer deleted the refactor_db_XIX branch November 19, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant