Skip to content

[Relax][ONNX] Use ONNX Backend Tests to improve frontend coverage #19505

Open
Open
[Relax][ONNX] Use ONNX Backend Tests to improve frontend coverage#19505

Description

@mshr-h
mshr-h
opened on May 4, 2026

This is my brain dump for a possible improvement to the Relax ONNX frontend tests.

Currently, many ONNX frontend tests in Relax manually construct small ONNX models and compare the imported Relax program against ONNX Runtime. This makes the test suite somewhat ad-hoc. As coverage grows, it becomes easy to miss ONNX spec corner cases, opset-specific behavior, and edge cases that are already covered by the official ONNX test suite.

ONNX provides an official Backend Test suite, including node-level tests for operator behavior:

I think it would be worth exploring whether Relax's ONNX frontend tests can reuse those tests as a more systematic source of coverage.

The goal is not necessarily to expose TVM as a full ONNX backend. Rather, the idea is to use the ONNX Backend Tests as a testing mechanism for the importer:

  • improve coverage of ONNX operator semantics
  • reduce duplicated hand-written operator tests
  • catch edge cases that are easy to miss with random-input tests
  • align frontend correctness testing more closely with the ONNX specification
  • keep TVM-specific frontend tests only for behavior that ONNX Backend Tests do not cover, such as importer options, parameter handling, name sanitization, and Relax-specific integration

Contributions, suggestions, or alternative ideas would be very welcome.

This issue is intended to collect thoughts and discuss whether this direction makes sense before working out a detailed design.

cc @KJlaccHoeUM9l @junrushao

Activity

  1. Aharrypotter commented on May 5, 2026

    @Aharrypotter
    Aharrypotter
    on May 5, 2026
    Contributor

    Completely agree.

    This will really clean up the relax/frontend/onnx workflow. By hooking into the official suite, we can focus entirely on the actual lowering logic and avoid spending time hand-writing tests and making up fake data.

    I’d love to take this on and submit a PoC PR for the basic runner.

    cc @KJlaccHoeUM9l @junrushao

  2. Aharrypotter
    added a commit that references this issue on May 6, 2026

    [Relax][ONNX] Add ONNX Backend Test runner for frontend coverage

    38449b4

  3. Aharrypotter commented on May 6, 2026

    @Aharrypotter
    Aharrypotter
    on May 6, 2026
    Contributor

    I've started working on this and have a preliminary PoC ready.

    What it does:

    • Introduces a test runner that feeds ONNX Backend Test node-level cases into relax.frontend.onnx.from_onnx
    • Compiles the imported Relax module with the VM and compares against ONNX's expected protobuf outputs
    • Restricts the runner to NodeModel tests only (filters out real/simple/PyTorch model tests via BackendTest._test_items with layout validation)
    • Adds onnx_backend pytest marker and SKIP_SLOW_TESTS support

    Current status:

    • 2406 node tests collected
    • Representative subset (add, relu, gemm, conv, top_k, optional_get_element) passes; 1 xfail on test_top_k_cpu (dynamic k not yet supported)
    • Full suite will be run in CI; known xfails are documented per importer gap category

    Approach note: ONNX does not expose a public API to select only node tests, so the runner localizes _test_items access in one helper with assertions that fail collection if ONNX changes the internal layout.

    I'll submit a Draft PR tonight for early feedback on the integration strategy. The PR will intentionally not fix importer gaps—those will be addressed in follow-ups so each change stays reviewable.

    cc @KJlaccHoeUM9l @junrushao

  4. Aharrypotter
    added 2 commits that reference this issue on May 6, 2026

    [Relax][Frontend][ONNX] Add ONNX Backend Tests runner for systematic …

    22148b0

    [Relax][Frontend][ONNX] Add ONNX Backend Tests runner for systematic …

    15e28cd

  5. Aharrypotter commented on May 6, 2026

    @Aharrypotter
    Aharrypotter
    on May 6, 2026
    Contributor

    Update: PR #19515 is up with a working implementation.

    What it does

    • Wraps the official ONNX Backend Test Suite (node-level tests) around relax.frontend.onnx.from_onnx
    • Implements onnx.backend.base.Backend interface: import → DecomposeOpsForInference() → LegalizeOps() → detach_params() → tvm.compile() → VM execution
    • Compares VM outputs against ONNX's protobuf reference outputs

    Current numbers

    • 72 operators with 388 test cases, all passing.
    • No xfail markers — only operators where every ONNX node test passes are included
    • Uncovered operators (cast, reduce, reshape, resize, quantization, etc.) are excluded via allowlist and can be added incrementally

    Key design points

    • NodeModel tests only; real/simple/PyTorch model tests filtered out
    • Existing test_frontend_onnx.py untouched — backend tests complement, not replace, TVM-specific tests

    Feedback welcome.

  6. mshr-h
    added a commit that references this issue on May 13, 2026

    [Relax][ONNX] Add ONNX Backend Tests for systematic frontend coverage (…

    b473185
  7. Aharrypotter
    added 2 commits that reference this issue on Jun 28, 2026

    [Relax][ONNX] Support dynamic k in Trilu and expand backend allowlist

    94ea7a4

    [Relax][ONNX] Support dynamic k in Trilu and expand backend allowlist

    2b1cba2
  8. Aharrypotter
    added 2 commits that reference this issue on Aug 11, 2026

    [Relax][ONNX] Add CastLike support and unblock Relu backend tests

    563c9a7

    [Relax][ONNX] Support dynamic k in Trilu and expand backend allowlist

    720af08
  9. Aharrypotter
    added 2 commits that reference this issue on Sep 1, 2026

    [Relax][ONNX] Add CastLike support and unblock Relu backend tests

    a8f58ef

    [Relax][ONNX] Support dynamic k in Trilu and expand backend allowlist

    6428deb
  10. Aharrypotter
    added 2 commits that reference this issue on Sep 8, 2026

    [Relax][ONNX] Add CastLike support and unblock Relu backend tests

    a1be24e

    [Relax][ONNX] Support dynamic k in Trilu and expand backend allowlist

    93fb9e7
  11. tlopex
    added a commit that references this issue on Sep 11, 2026

    [Relax][ONNX] Add CastLike support and dynamic-k Trilu to expand back…

    8f328e8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions