Skip to content

A gate set holds a floor per criterion, read from each case's expected criterion #692

Closed

Description

@don-petry

Why

A gate set's bars are one per kind, met and finding (#613). A reviewer blind to one criterion can still clear both, so long as that criterion's cases are few among the findings. The proposal on #562 asked for a floor per graded criterion as well.

The change

  • A gate set's manifest may give criterion_bars, {criterion: [n, d]}: of the case-runs of the cases whose expected.criterion is that criterion, at least n/d pass. A criterion the manifest does not name has no floor. Only a gate set may give it, as with bars and kind_least.
  • The job reads each case's criterion from its fetched case, as kinds_of reads its kind, so the manifest carries no criterion per case.
  • eval-aggregate/1 gains criteria: {criterion: {cases, passed, interval}} for each criterion of the set, counts alone. aggregate_verdict holds them to the set's counts, and pooled_verdict sums them over the pool and holds each named criterion to its floor (criterion_failed).

This publishes how many cases of the set name each criterion, in every gate report. The Maintainer accepted that.

Done when

  • Schema, job and verdict changes, with tests: a pool under one criterion's floor fails with the others clear; a criterion with no floor never fails one; counts that do not add up are count_differs; a development set's manifest may not give criterion_bars.
  • A mutant that skips the per-criterion check in pooled_verdict is killed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions