Repository navigation
bug(validation): the validator is handed an empty run response and its "nothing to inspect" is recorded as failed_validation — a wrong high-priority alert per successful run #2959
Description
Summary
Schedule validation (validation_enabled + validation_prompt, VALIDATE-001) starts a second execution to judge the run's output PASS/FAIL. In the field the validator receives nothing to inspect, correctly says so, and the platform records that report as business_status: "failed_validation" on a parent that completed normally, then files a high-priority "Validation Failed" operator alert. The signal is inverted: it alerts on runs that succeeded, cannot isolate a run that genuinely produced nothing, and passed the one cycle that carried a real defect.
Journey Impact: extends: J05 ("I can schedule work, walk away, and find out what happened") — the "find out what happened" half returns the wrong answer.
Context
Observed in production on a self-hosted 0.9.0 instance across three weekly cycles (2026-08-31, 09-07, 09-14): four weekly content schedules on one agent, validation configured on all four. 1 pass in 12.
| Cycle | Validations | Passed | Alerts filed |
|---|---|---|---|
| 2026-08-31 | 4 | 1 | 3 |
| 2026-09-07 | 4 | 0 | 4 |
| 2026-09-14 | 4 | 0 | 4 |
Verbatim, from a validation execution's own response:
FAIL - no run output or transcript was provided to inspect, so there is no "RUN-STATUS:" line (or any completion summary/post list) anywhere to evaluate.
RUN-STATUS: FAILED
The parent it judged (1,109 s, success) ends with exactly the line the validator reports as absent: RUN-STATUS: COMPLETE | … | posts=2 | ….
Duration and cost separate the two populations: the one PASS took 46 s / $0.30 (a validator that had something to read); the 11 FAILs took 13-31 s / ~$0.17 each (an empty envelope). The schedules were last modified 2026-08-10 and not touched since, yet on the 08-17 and 08-24 cycles no validation execution ran at all (business_status: null, no triggered_by: "validation" rows), and from 08-31 validation fired on every run. The instance moved to 0.9.0 on 08-25 — stated as correlation, not proof.
The feature was disabled on these schedules on 09-15 and replaced with an out-of-band check against the artifact store. An alert that is wrong four times a week trains everyone to stop reading the queue, including when it is right.
Data path (verified on dev @ 7d851cb85)
- The scheduler polls its own
schedule_executionsrow until the status is terminal (src/scheduler/service.py_poll_execution_completion,SELECT *), then_maybe_trigger_validationpostsexecution_response=execution.response or ""toPOST /api/internal/validate-execution(~L2154-2161). services/validation_service.py(~L342-349) interpolatesexecution_response or "(no response)"into the validator prompt, truncated to the first 10,000 characters. A run's status line is the last line of its response, so a long response loses exactly the line the validator is asked to find even when the response is present.- The success writer (
services/task_execution_service.py~L2770) persistsresponse=sanitized_respin the same UPDATE asstatus, so asuccessrow read after the status flip should carry its response. The validator's verbatim wording says it did not on 0.9.x. Open trace: which terminal path on that release leavesresponseempty for a completed scheduled run (an emptysanitized_resp, a salvage/envelope path, or a second UPDATE after the status flip). - "The validator received no input" and "validation failed" are the same outcome today: same
failed_validation, same alert. The validator is reporting its own blindness and the platform reports it as a verdict about the job.
Acceptance Criteria
- The parent's response reaches the validator intact enough to judge: the prompt carries the tail as well as the head (or the whole response up to a stated cap), and a test with a >10,000-character response whose status line is last passes
- Empty or missing input is a distinct outcome — recorded as an
unavailable-classbusiness_status(name to choose), neverfailed_validation, and it does not file the "Validation Failed" alert; a validator that could read nothing cannot PASS either - The empty-
responsepath is traced and fixed or named: a scheduled run whose row issuccesshas itsresponsepresent when validation is triggered, on both the sync and the async/callback dispatch paths - The dormant-then-active timeline is explained: what starts validation on schedules configured before an upgrade, and why two cycles ran none
- Regression tests: (a) a
successparent with a long response validates against its last line; (b) an empty response yields the distinct outcome and no high-priority alert
Technical Notes
- Found by the same production audit as bug(operator-queue): a pending approval can sit unseen for days — the human's card goes stale while the container↔platform sync stays silent #2915 (defect 5 of 5). Adjacent, not the same: Post-execution validation hook: run a validation playbook in clean context and record business-success status #1573 (validation playbook in clean context,
business_statusrecording) — the "reads only the run's self-reported status line, never an artifact" limitation that let the one real defect pass belongs there, not here. - Labelled
priority-p2at filing; the field severity was "high" (inverted signal, four wrong alerts a week, feature disabled by the operator), so a bump to p1 is a one-label call.
Activity
vybe commentedon Oct 1, 2026
Product decision (maintainer, at claim time): the distinct outcome for a validator that received nothing to inspect is recorded as
business_status: "validation_unavailable". It never files the "Validation Failed" alert and can never count as a PASS.fix(validation): hand the validator the run it judges; empty input is…