Repository navigation
Providing reproduction code for failed test cases #379
Description
Activity
Thanks @mtsokol !
This is a great idea.
A couple of quick comments:- hypothesis does indeed sometimes make figuring out the inputs quite hard!
- the current
ph.assert_*helpers go to incredible lengths to give useful diagnostics. Maybe we could improve specific helpers? You mentiontest_repeat, do you happen to remember other cases with less-than-useful diagnostics? - As an example where the full output is not that helpful: https://github.com/data-apis/array-api-tests/actions/runs/15072644093/job/42372581562
- You mention
xp.reshape, but https://github.com/search?q=repo%3Adata-apis%2Farray-api-tests%20reshape&type=code shows what, three calls?
the current
ph.assert_*helpers go to incredible lengths to give useful diagnostics. Maybe we could improve specific helpers? You mentiontest_repeat, do you happen to remember other cases with less-than-useful diagnostics?I can take a look at the tests that I've fixed in the past and provide a few more next week!
You mention
xp.reshape, but https://github.com/search?q=repo%3Adata-apis%2Farray-api-tests%20reshape&type=code shows what, three calls?In
array-api-teststhat's correct. But the situation that I described originated in the Hypothesis package (sorry for omission). Indef do_draw, that is used for drawing samples, Hypothesis callsreshape:When
xp.reshapewas temporarily broken it caused 148 errors for us in CI, you can searchreshapeword there.I can take a look at the tests that I've fixed in the past and provide a few more next week!
Thanks! It'd be very helpful.
In array-api-tests that's correct. But the situation that I described originated in the Hypothesis package (sorry for omission). In def do_draw, that is used for drawing samples, Hypothesis calls reshape:
That's a good point. Not much we can do about it at the array-api-tests level though.
Looking at thathypothesis/extra/array_apimodule, it usesself.xp.reshape,self.xp.asarray,self.xp.zeros,self.xp.isnan. If any of these is broken, then indeed, so is the whole ofarray-api-tests.
That's unfortunate, but I don't know if there's an easy way out. An "obvious" solution to generate test arrays with numpy andxp.asarraythem will not work for non-CPU devices.
When dlpack 1.0 is mature enough and is available everywhere, we can possibly updatehypothesis/extra/array_apito generate numpy arrays andxp.from_dlpackthem, but then a new array library would need to start fromdlpack.Would it make sense to require
reshape,asarray,zeros,isnanto run the test suite? These ones could be considered essential.You mean add a note to this effect to the "Interpreting errors" section of the README?
https://github.com/data-apis/array-api-tests/blob/master/README.mdAs a usability check: would this be useful to you when you had a issues with
xp.reshape? If yes, sure, please send a PR! If not, then what would be?You mean add a note to this effect to the "Interpreting errors" section of the README? https://github.com/data-apis/array-api-tests/blob/master/README.md
Yes, I think a note listing which functions are essential to run most of the test suite would be useful.
As a usability check: would this be useful to you when you had a issues with
xp.reshape? If yes, sure, please send a PR! If not, then what would be?Here I think my answer is also "yes", but I'm not sure how it would impact execution time checking it for each test, and that's a bit noisy change for the repository:
ESSENTIAL_FUNCS_MISSING = any(not hasattr(xp, name) for name in ["reshape", "asarray", "zeros", "isnan"]) ensure_essentials = pytest.mark.skipif( ESSENTIAL_FUNCS_MISSING, reason="Essential function are missing from the namespace: ...", ) # in all relevant files: @ensure_essentials def test_function(): ...
As long as it's a couple of
hasattrchecks, it's not a problem (anyhypothesiswork dwarfs that, by far).Basically, what you talk about is #51 --- and if we do it, then each test should get its decorator with the actual list of its dependencies.
(Nevermind the "low priority" label on that issue; this is I think a useful thing to do).OTOH, if the main pain point is with the hidden dependencies of
hypothesis.array_api, then a single test probably suffices?When developing ndonnx, we sometimes encountered issues with these "essential" functions, too. However, as the package matured, we passed that bar and have not had issues in this regard in a long time. I think our development process would have been a bit easier if the test suite had forced us to implement these essential functions first and explicitly.
- added a commit that references this issue
on Jun 6, 2025 - added a commit that references this issue
on Aug 12, 2025 - added a commit that references this issue
on Oct 27, 2025 Speaking of "essential" functions, the work on MLX (data-apis/array-api-compat#450) highlighted
__complex__and__array_namespace_info__. The lack of the former generates walls of errors, and without the latter the test suite just does not run.
Hi all!
Here's one idea on how to potentially improve interacting with failing tests in
array-api-testssuite which @ev-br and I discussed this week.I've been involved in the introduction of
array-api-testssuite in a few repositories and, from my personal experience, the activity that took the most of the time was figuring out root causes of failed tests.Each failure shows the stack trace (or multiple stack traces, if there were e.g. 3 distinct failures for a given test), but at times they are merely related to the actual root cause. For instance, in NumPy we have an
xfail:which originally was reported by the test suite with an error:
I had to manually recreate the function call with the exact inputs to understand which edge case we hit. And for each test the error message was either accurate, like missing keyword argument, or irrelevant - which was mostly related to array scalars, Python builtins, or 0-D arrays as inputs or outputs.
From my point of view, one possible improvement of this process could be something like a CLI option
--with-repro-snippets, where each failing test is accompanied by a copy-paste line that exactly reproduces it.So for
test_repeatI would get:The
array-api-testssuite would compose it withf"xp.{func_name}({inputs}, {kwargs})"when a function call fails.This way, when I run a test suite for the first time and get e.g. 50 failures, I can iterate a bit faster by skipping the "reproduce failing code snippet" step. WDYT? Please share your thoughts!
More of a nitpick but some Array API functions are used in multiple tests, like
reshape. I think it’s used primarily for setting up inputs for tests. When that one function is missing from the namespace, a large part of the test suite fails with more cryptic error messages. I think that there could be ahasattr(xp, "reshape")decorator for tests and if a function for setting inputs is missing, the error says so.