fix: the failure study was certifying its own answers, and one was wrong

[tschm]

Closes #36. Corrects claims already merged in #55, #57 and #58, and hands a finding to #22.

#36 was left open on purpose. Its code merged with #55, but the study there reports badly scaled as solved — which is the claim this PR shows to be false. Closing the issue on a wrong finding would have been the wrong record, so it closes here instead.

What was wrong

The failure-mode study trusted §6's residuals and never asked a reference solver. So solved meant "COSA is satisfied with itself". On twelve of thirteen families that is the same thing. On the thirteenth it is not.

badly scaled terminates reporting optimal, with all five conic KKT residuals between 1e-11 and 1e-15:

	objective	feasibility of that point
COSA	−0.00524	worst Az−b = −2.5e-2, |Ez−d| = 1.6e-15, cone slack 0
CLARABEL	−0.03956	worst Az−b = −5.0e-11, |Ez−d| = 2.2e-16, cone slack 1.5e-12

The reference's point is feasible for COSA's own feasibility check to 1e-11, and its objective is 7.5× better. The direction between the two points is a feasible descent direction with derivative −0.0343. No tolerance argument survives that: a better feasible point demonstrably exists.

Why the residual does not notice

The residual is not lying. Stationarity there is 1.93e-05 absolute. §14.2 reports it relative to the objective's scale, dividing by max(1, |c|_inf) — and this instance's |c|_inf is 2e6. So it prints as 9.7e-12.

A convex problem cannot have an exactly satisfied KKT system at a suboptimal point, and this one does not: the residual is real. It is simply small enough, relative to data of size 1e6, that a relative certificate cannot distinguish it from zero — while the conditioning amplifies it into a 3.4% objective error.

The certificate is relatively satisfied and the answer is wrong, and neither of those is a mistake in the other.

That is the finding, and it is about the stopping criterion, not the arithmetic. §6's residuals are "mathematically meaningful" as Success Criterion 2 asks, and they are not sufficient: on an instance whose data spans fourteen orders of magnitude a relative KKT residual admits a percent-wrong answer. An absolute floor, or a normalization that does not divide by |c|_inf, is what is missing. That belongs to #22, and it is stated in the paper's assumptions section as a required assumption the plan did not anticipate needing — not patched quietly.

What changed

• failures.study cross-checks every answer against a reference solver, which Success Criterion 5 asked for all along and this study was not doing.
• Five verdicts instead of four. The new one is wrong: optimal, certified, and disagreeing. Precedence matters and is asserted — disagreement outranks a clean certificate, because a certificate that certifies the wrong answer is worse than no certificate.
• Skipping the check gives unchecked, not solved. "Nothing disagreed with me" and "a reference agreed with me" are different claims, and conflating them is precisely what caused this.
• Verdicts: 36 solved, 3 wrong. Twelve families agree to 1e-9 or better.
• The paper's results section gains §15.6, its assumptions section gains the missing assumption, and Risk 3 is re-answered: the residue is the criterion, not the linear algebra. failure-modes.md and the README are corrected too.

Three conclusions, two of them wrong

This family has now had three diagnoses and this is the first that was not self-certified.

1. A conditioning failure that scaling fixes. It stalled at 5e-2; equilibration produced an apparent optimum at 9.7e-7. Clean before-and-after.
2. An initialization bug, and it solves unaided. raise_free_heads required a head row coefficient of exactly one, which no rescaled instance satisfies — so the retraction was silently unavailable and a boundary iterate could not move. Equilibration had been "fixing" it by perturbing that coefficient. With the restriction removed the family ran to completion with residuals under 1e-11, and was reported solved.
3. It runs to completion and gets the answer wrong, which only a reference check can see.

Each step had evidence. Two were wrong. A study that certifies itself will confirm whichever hypothesis it started with — and equilibration still does not help: equilibrated, the same instance lands at −0.00540 against the same reference.

All rhiza gates green, 100% coverage, 1187 tests.

[coderabbitai]

Warning

Review limit reached

Next included review available in 12 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: f4a0e76e-8b06-4540-8a12-4aa123240606

📥 Commits

Reviewing files that changed from the base of the PR and between 010cdf8 and aa65e98.

📒 Files selected for processing (7)

• README.md
• docs/development/failure-modes.md
• docs/experiments/benchmarks.txt
• docs/experiments/failure-modes.txt
• docs/paper/paper.tex
• src/cosa/experiments/failures.py
• tests/test_failures.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}