Skip to content

bug(validation): the validator is handed an empty run response and its "nothing to inspect" is recorded as failed_validation — a wrong high-priority alert per successful run #2959

Closed

Description

@webmixgamer

Summary

Schedule validation (validation_enabled + validation_prompt, VALIDATE-001) starts a second execution to judge the run's output PASS/FAIL. In the field the validator receives nothing to inspect, correctly says so, and the platform records that report as business_status: "failed_validation" on a parent that completed normally, then files a high-priority "Validation Failed" operator alert. The signal is inverted: it alerts on runs that succeeded, cannot isolate a run that genuinely produced nothing, and passed the one cycle that carried a real defect.

Journey Impact: extends: J05 ("I can schedule work, walk away, and find out what happened") — the "find out what happened" half returns the wrong answer.

Context

Observed in production on a self-hosted 0.9.0 instance across three weekly cycles (2026-08-31, 09-07, 09-14): four weekly content schedules on one agent, validation configured on all four. 1 pass in 12.

Cycle Validations Passed Alerts filed
2026-08-31 4 1 3
2026-09-07 4 0 4
2026-09-14 4 0 4

Verbatim, from a validation execution's own response:

FAIL - no run output or transcript was provided to inspect, so there is no "RUN-STATUS:" line (or any completion summary/post list) anywhere to evaluate.
RUN-STATUS: FAILED

The parent it judged (1,109 s, success) ends with exactly the line the validator reports as absent: RUN-STATUS: COMPLETE | … | posts=2 | ….

Duration and cost separate the two populations: the one PASS took 46 s / $0.30 (a validator that had something to read); the 11 FAILs took 13-31 s / ~$0.17 each (an empty envelope). The schedules were last modified 2026-08-10 and not touched since, yet on the 08-17 and 08-24 cycles no validation execution ran at all (business_status: null, no triggered_by: "validation" rows), and from 08-31 validation fired on every run. The instance moved to 0.9.0 on 08-25 — stated as correlation, not proof.

The feature was disabled on these schedules on 09-15 and replaced with an out-of-band check against the artifact store. An alert that is wrong four times a week trains everyone to stop reading the queue, including when it is right.

Data path (verified on dev @ 7d851cb85)

  1. The scheduler polls its own schedule_executions row until the status is terminal (src/scheduler/service.py _poll_execution_completion, SELECT *), then _maybe_trigger_validation posts execution_response=execution.response or "" to POST /api/internal/validate-execution (~L2154-2161).
  2. services/validation_service.py (~L342-349) interpolates execution_response or "(no response)" into the validator prompt, truncated to the first 10,000 characters. A run's status line is the last line of its response, so a long response loses exactly the line the validator is asked to find even when the response is present.
  3. The success writer (services/task_execution_service.py ~L2770) persists response=sanitized_resp in the same UPDATE as status, so a success row read after the status flip should carry its response. The validator's verbatim wording says it did not on 0.9.x. Open trace: which terminal path on that release leaves response empty for a completed scheduled run (an empty sanitized_resp, a salvage/envelope path, or a second UPDATE after the status flip).
  4. "The validator received no input" and "validation failed" are the same outcome today: same failed_validation, same alert. The validator is reporting its own blindness and the platform reports it as a verdict about the job.

Acceptance Criteria

  • The parent's response reaches the validator intact enough to judge: the prompt carries the tail as well as the head (or the whole response up to a stated cap), and a test with a >10,000-character response whose status line is last passes
  • Empty or missing input is a distinct outcome — recorded as an unavailable-class business_status (name to choose), never failed_validation, and it does not file the "Validation Failed" alert; a validator that could read nothing cannot PASS either
  • The empty-response path is traced and fixed or named: a scheduled run whose row is success has its response present when validation is triggered, on both the sync and the async/callback dispatch paths
  • The dormant-then-active timeline is explained: what starts validation on schedules configured before an upgrade, and why two cycles ran none
  • Regression tests: (a) a success parent with a long response validates against its last line; (b) an empty response yields the distinct outcome and no high-priority alert

Technical Notes

Activity

  1. added and removed
    status-readyGreenlit and ready for development (vetted; counterpart to status-incubating)
    on Oct 1, 2026
  2. self-assigned this
    on Oct 1, 2026
  3. vybe commented on Oct 1, 2026

    @vybe
    Contributor

    Product decision (maintainer, at claim time): the distinct outcome for a validator that received nothing to inspect is recorded as business_status: "validation_unavailable". It never files the "Validation Failed" alert and can never count as a PASS.

  4. added
    status-in-devMerged to dev, awaiting release cut to main
    and removed on Oct 1, 2026
  5. removed
    status-in-devMerged to dev, awaiting release cut to main
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions