Skip to content

test: add parser complexity (scaling) regression tests - #611

Draft
enomado wants to merge 1 commit into
ron-rs:masterfrom
enomado:test/parser-complexity-scaling
Draft

enomado wants to merge 1 commit into
ron-rs:masterfrom
enomado:test/parser-complexity-scaling

Conversation

@enomado

@enomado enomado commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

ron has ~450 correctness tests and, until now, zero tests that guard the
asymptotic cost of parsing. That blind spot is exactly how two independent O(n²)
regressions shipped unnoticed:

Both fixes are now on master. This PR adds the missing layer: a test that would have
caught either regression the day it landed — and will catch the next one. Both bugs are
the same shape (an unbounded find(..) scanning to end-of-input, called once per
element in a loop over that same input), and no correctness test can see it: they all
check what is parsed, never how the cost grows.

What

tests/complexity_scaling.rs adds doubling-ratio tests that assert parse time grows
at most linearly
in input size:

  • Double the input, measure the time. Linear ⇒ ratio ≈ 2; quadratic ⇒ ratio ≈ 4. The
    assertion is on the ratio, so it does not depend on the speed of the machine
    running it.
  • Robust to timing noise: each point is the minimum of N runs (noise can only add
    time) and the verdict is the median of the doubling ratios (one slow sample
    cannot flip it). The threshold 3.0 sits in the wide gap between linear (~2) and
    quadratic (~4).
  • The tests are #[ignore]d, so they never slow down or flake the normal cargo test
    job. A dedicated Scaling CI job runs them in --release, single-threaded.

It's green — and it demonstrably bites

Now that #608 and #610 are merged, the Scaling job is green on this PR: it
confirms the parser is linear on the current code. Ratios from this PR's own CI run:

form guards ratio (fixed master) ratio (pre-fix d0e99bc0)
escaped_string #610 / #534 2.01 ✓ 4.02 ✗
floats_via_value #608 2.03 ✓ 3.85 ✗
int_list (control) — 2.03 ✓ 2.01 ✓
plain_strings (control) — 1.98 ✓ —

The right-hand column (verified locally against d0e99bc0, the commit right before the
two fixes) is the important one: run against the buggy tree, the two bug-catching forms
report 4.02 and 3.85 — well past the 3.0 threshold. The test fails exactly when
the regression is present and passes once it's fixed. The two linear controls stay ~2 on
both trees, so a failure is specific, not just "parsing got slower".

Note floats_via_value is not only about floats: on the self-describing Value path
next_bytes_is_float runs for every number to disambiguate int from float, so the
same blow-up hit integer sequences and map keys/values too (a 200k-entry integer map
took 103 s on the pre-fix tree).

Notes for maintainers

  • The Scaling job is isolated from the main test matrix, so a timing hiccup can never
    block a normal PR. If you'd prefer it non-blocking anyway, continue-on-error: true
    or moving it under the schedule: trigger both work — happy to adjust.
  • Input sizes keep every linear run in the low-millisecond range (the whole suite is
    ~1 s on CI); a form only gets slow when it is actually quadratic — which is the case
    you want to be loud.
  • Adding a form is ~15 lines: a builder fn(usize) -> String and a parse closure.

🤖 Generated with Claude Code

ron has ~450 correctness tests but none that guard the asymptotic cost of parsing. That gap let ron-rs#534 regress escaped_byte_buf from O(n) to O(n^2) across v0.9.0..=v0.12.2 — four releases — with every test green, and a second O(n^2) entered next_bytes_is_float in v0.12.2 (ron-rs#602).

tests/complexity_scaling.rs asserts that parse time grows at most linearly: it doubles the input and checks the time ratio (~2 linear vs ~4 quadratic; threshold 3.0), using min-of-N + median to stay robust to timing noise. The tests are #[ignore]d and run in a dedicated release CI job, so they never slow down or flake the normal test job.

Two forms guard the regressions fixed in ron-rs#608 (next_bytes_is_float) and ron-rs#610 (escaped_byte_buf); two linear controls keep the harness honest. On the pre-fix tree (d0e99bc) those two forms report ratio ~4.0 and fail the 3.0 threshold; on current master they are ~2.0 and pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant