Skipped flaky tests

Several tests are completely skipped right now because they are "flaky".

test_reshape
test_std
test_var
test_remainder

This is a pretty high priority issue because these functions are effectively completely untested, even though they appear to be tested.

Tests should be written in such a way that they aren't flaky, for instance, by using high numerical tolerances (or if necessary, avoiding values testing entirely).

Note that health checks for timeouts should just be skipped, and health checks for filtering too much should be fixed by fixing the strategy.

EDIT:

test_count_nonzero is not skipped but is flaky on JAX. Some discussion is at add a test for count_nonzero #347 (comment)

Looking at test_std, https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_statistical_functions.py#L262, it does not seem to attempt any value testing. Then what is flaky, assert_dtype or assert_keepdimable_shape?

I've no idea what's flaky with any of these. The first order of business would to remove that decorator and figure out why the test was failing. It's also possible that some of these were only flaky with certain libraries.

Also, it's possible the flakyness was fixed and the skip was never removed. It looks like skip for std was added in #233 (with no explanation) if you want to check previous versions.

At best, if the test seems to be passing, we can just remove the skip and see if any upstream failures are found. Like I mentioned in another issue, it's really easy to just revert changes here if they break stuff since we don't even have releases, so I wouldn't be too worried about that.

test_reshape is fixed in gh-319

Caught a test_std failure with array_api_compat.numpy:

array_api_tests/test_statistical_functions.py::test_std FAILED                   [100%]

    @given(
>       x=hh.arrays(
            dtype=hh.real_floating_dtypes,
            shape=hh.shapes(min_side=1),
            elements={"allow_nan": False},
        ).filter(lambda x: math.prod(x.shape) >= 2),
        data=st.data(),
    )

array_api_tests/test_statistical_functions.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def inconsistent_generation():
>       raise FlakyStrategyDefinition(
            "Inconsistent data generation! Data generation behaved differently "
            "between different runs. Is your data generation depending on external "
            "state?"
        )
E       hypothesis.errors.FlakyStrategyDefinition: Inconsistent data generation! Data generation behaved differently between different runs. Is your data generation depending on external state?

../../miniforge3/envs/array-api-tests/lib/python3.11/site-packages/hypothesis/internal/conjecture/datatree.py:52: FlakyStrategyDefinition
-------------------------------------- Hypothesis --------------------------------------
You can add @seed(146745493194750825545715057348996307346) to this test or run pytest with --hypothesis-seed=146745493194750825545715057348996307346 to reproduce this failure.
=================================== warnings summary ===================================
array_api_tests/test_statistical_functions.py: 59 warnings
  /home/br/miniforge3/envs/array-api-tests/lib/python3.11/site-packages/numpy/_core/_methods.py:227: RuntimeWarning: Degrees of freedom <= 0 for slice
    ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================== short test summary info ================================
FAILED array_api_tests/test_statistical_functions.py::test_std - hypothesis.errors.FlakyStrategyDefinition: Inconsistent data generation! Data gener...
===================== 1 failed, 7 deselected, 59 warnings in 3.14s =====================
(array-api-tests) br@gonzales:~/repos/array-api-tests$

I can reproduce that with

ARRAY_API_TESTS_MODULE=array_api_compat.numpy pytest --disable-warnings array_api_tests/test_statistical_functions.py -k std -v --hypothesis-seed=146745493194750825545715057348996307346 --max-examples=10000

I can't tell what is causing it. None of the strategies seem to be that unusual. The only thing I see that's a little different from the other tests is that the input array is filtered to have at least 2 elements, but that shouldn't be causing this error.

Unfortunately, hypothesis makes it quite hard to tell what's going on with this error. The only thing I can suggest would be to refactor the input strategies, e.g., to use shared instead of data.draw. Otherwise, we may want to report this upstream on the hypothesis repo, and see if the hypothesis devs can offer any advice. It may also just be a bug in hypothesis.

#417 removed skips from test_{std,var} on the grounds of not being able to catch the problem anymore, with ~50k examples. Let's see how this goes.

For test_remainder: on numpy, the following seems to pass:

$ ARRAY_API_TESTS_MODULE=array_api_compat.numpy pytest --disable-warnings array_api_tests/test_operators_and_elementwise_functions.py::test_remainder --max-examples=50_000 --hypothesis-verbosity=verbose --hypothesis-show-statistics

With torch,

$ ARRAY_API_TESTS_SKIP_DTYPES=uint16,uint32,uint64 ARRAY_API_TESTS_MODULE=array_api_compat.torch pytest --disable-warnings array_api_tests/test_operators_and_elementwise_functions.py::test_remainder --max-examples=50_000 --hypothesis-verbosity=verbose --hypothesis-show-statistics

fails with what looks like torch not worrying about extreme edge cases:

In [3]: from operator import __imod__

In [4]: __imod__(torch.as_tensor([[1.3659e+75, 1.3659e+75, 1.3659e+75, 1.3659e+75],
   ...:                    [1.3659e+75, 1.3659e+75, 1.3659e+75, 1.3659e+75]], dtype=torch.float64), torch.as_tensor(7.5979e-234, dtype=torch.float64))
Out[4]: 
tensor([[nan, nan, nan, nan],
        [nan, nan, nan, nan]], dtype=torch.float64)

Caught a test_std failure:

$ ARRAY_API_TESTS_MODULE=array_api_strict pytest array_api_tests/test_statistical_functions.py::test_std --hypothesis-seed=219838806059063697412923626075150989669  --hypothesis-verbosity=verbose --hypothesis-show-statistics
========================================================== test session starts ===========================================================
platform linux -- Python 3.12.0, pytest-8.3.5, pluggy-1.5.0
hypothesis profile 'array-api-tests-with-verbose-verbosity' -> verbosity=Verbosity.verbose, deadline=timedelta(milliseconds=800)
Environment variables:
----------------------
ARRAY_API_TESTS_MODULE = array_api_strict

Array API Tests Module: array_api_strict (2.5). API Version: 2025.12. Enabled Extensions: fft, linalg
rootdir: /home/br/repos/array-api-tests
configfile: pytest.ini
plugins: json-report-1.5.0, xdist-3.6.1, hypothesis-6.151.9, metadata-3.1.1
collected 1 item                                                                                                                         

array_api_tests/test_statistical_functions.py F                                                                                    [100%]

================================================================ FAILURES ================================================================
________________________________________________________________ test_std ________________________________________________________________

    @given(
>       x=hh.arrays(
            dtype=hh.real_floating_dtypes,
            shape=hh.shapes(min_side=1),
            elements={"allow_nan": False},
        ).filter(lambda x: math.prod(x.shape) >= 2),
        data=st.data(),
    )

array_api_tests/test_statistical_functions.py:342: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesis.internal.conjecture.datatree.TreeRecordingObserver object at 0x7f2bfdd21790>, status = Status.VALID
interesting_origin = None

    def conclude_test(
        self, status: Status, interesting_origin: InterestingOrigin | None
    ) -> None:
        """Says that ``status`` occurred at node ``node``. This updates the
        node if necessary and checks for consistency."""
        if status == Status.OVERRUN:
            return
        i = self._index_in_current_node
        node = self._current_node
    
        if i < len(node.values) or isinstance(node.transition, Branch):
>           raise FlakyStrategyDefinition(_FLAKY_STRAT_MSG)
E           hypothesis.errors.FlakyStrategyDefinition: Inconsistent data generation! Data generation behaved differently between different runs. Is your data generation depending on external state?

../../miniforge3/envs/array-api/lib/python3.12/site-packages/hypothesis/internal/conjecture/datatree.py:1151: FlakyStrategyDefinition
--------------------------------------------------------------- Hypothesis ---------------------------------------------------------------
Trying example: test_std(
    x=Array([-3.90673361e-187, -6.32527546e+016,
           -7.87355643e+153], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0, 'keepdims': True}
Trying example: test_std(
    x=Array([[ 1.5062885e-37,  1.8446743e+19,
             5.7786753e+16, -1.5921071e+19,
            -1.2016160e+19],
           [ 1.8446743e+19,  3.0445084e+18,
            -2.4831233e+16, -1.8446743e+19,
             1.8446743e+19],
           [ 1.2614472e+11,  0.0000000e+00,
             6.8658580e+18, -3.9193472e+18,
             4.8324746e+18],
           [-2.7490075e+15,  1.3366337e+16,
             1.0775137e+19, -1.4527134e+19,
             4.1176775e+16]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): 1
Draw 2 (correction): 2.2250738585072014e-308
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 1,
 'correction': 2.2250738585072014e-308,
 'keepdims': False}
Trying example: test_std(
    x=Array([-1.42474968e-128,  4.28206007e+016,
           -6.66166116e+016, -4.76688681e-242,
           -4.32457834e+016], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0.0}
Trying example: test_std(
    x=Array([ 7.194606e+18, -7.359926e+18], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0, 'keepdims': True}
Trying example: test_std(
    x=Array([[4.2207707e+15, 4.2207707e+15,
            4.2207707e+15, 4.2207707e+15,
            4.2207707e+15],
           [4.2207707e+15, 4.2207707e+15,
            4.2207707e+15, 4.2207707e+15,
            4.2207707e+15],
           [4.2207707e+15, 4.2207707e+15,
            4.2207707e+15, 4.2207707e+15,
            4.2207707e+15]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): tuple([])
Draw 2 (correction): 4.373461038448735e-69
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([]),
 'correction': 4.373461038448735e-69,
 'keepdims': False}
Trying example: test_std(
    x=Array([ 5.9524794e+18,  1.6011213e+19,
           -9.0977149e+18,  4.9439566e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): tuple([0])
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([0]), 'correction': 0, 'keepdims': False}
Trying example: test_std(
    x=Array([ 4.97675410e-163, -6.92609909e+016], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([])
Draw 2 (correction): 1.9999999999999998
Draw 3 (keepdims): True
Draw 4 (kw): {'axis': tuple([]),
 'correction': 1.9999999999999998,
 'keepdims': True}
Trying example: test_std(
    x=Array([-3.61005667e+016, -1.17463800e-035,
           -1.24035428e+016, -1.22750917e+109], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([(lambda x: x if x < ndim else x - 2 * ndim)(1)])
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([(lambda x: x if x < ndim else x - 2 * ndim)(1)]),
 'correction': 0.0,
 'keepdims': False}
Trying example: test_std(
    x=Array([[5.4563526e+17, 5.4563526e+17,
            5.4563526e+17],
           [5.4563526e+17, 5.4563526e+17,
            5.4563526e+17],
           [5.4563526e+17, 5.4563526e+17,
            5.4563526e+17],
           [5.4563526e+17, 5.4563526e+17,
            5.4563526e+17],
           [5.4563526e+17, 1.7190879e+19,
            6.6623889e+16]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 1.4104778187510527
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 1.4104778187510527}
Trying example: test_std(
    x=Array([-2.1053489e+18, -1.4356998e+16,
           -2.7968239e+01], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'axis': -1, 'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([-2.1053489e+18, -1.4356998e+16,
           -2.7968239e+01], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0.0}
Trying example: test_std(
    x=Array([-2.1053489e+18,  0.0000000e+00,
           -2.7968239e+01], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([ 0.0000000e+00, -2.1053489e+18,
           -2.7968239e+01], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Trying example: test_std(
    x=Array([[-2.1053489e+18,  0.0000000e+00],
           [-2.7968239e+01, -2.1053489e+18],
           [-2.1053489e+18, -2.1053489e+18]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([[-2.1053489e+18,  0.0000000e+00],
           [-2.7968239e+01, -2.1053489e+18],
           [-2.1053489e+18, -2.1053489e+18]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([ 3.3464317e+16,  1.6628357e+19,
           -9.8246015e+18,  2.1862309e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([ 3.3464317e+16,  1.6628357e+19,
           -9.8246015e+18,  2.1862309e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([ 3.3464317e+16,  3.3464317e+16,
           -9.8246015e+18,  2.1862309e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([ 3.3464317e+16,  3.3464317e+16,
           -9.8246015e+18,  2.1862309e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([ 3.3464317e+16, -9.8246015e+18,
           -9.8246015e+18,  2.1862309e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([-9.824601e+18, -9.824601e+18,
           -9.824601e+18,  2.186231e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([-9.824601e+18, -9.824601e+18,
           -9.824601e+18,  2.186231e+16], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 6.35731839e+016,
            -1.69105909e+016],
           [-3.60854688e-113,
            -2.78250221e+016]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0}
Trying example: test_std(
    x=Array([[ 6.35731839e+16, -1.69105909e+16],
           [ 0.00000000e+00, -1.95965779e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 6.35731839e+16, -1.69105909e+16],
           [ 0.00000000e+00, -1.95965779e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0.],
           [0., 0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([[0.],
           [0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([[ 6.4399831e+16,  1.1733861e+16,
             7.3173401e+15],
           [ 7.3489473e+15, -3.4536455e+16,
             1.8414277e+19]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0, 'keepdims': False}
Trying example: test_std(
    x=Array([[ 6.4399831e+16,  1.1733861e+16,
             7.3173401e+15],
           [ 7.3489473e+15, -3.4536455e+16,
             1.8414277e+19]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0, 'keepdims': False}
Trying example: test_std(
    x=Array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[-2.0000100e+00, -2.4705337e+16,
            -7.8850107e+18,  1.8446743e+19,
            -3.1100245e+16],
           [-6.2392041e+16, -1.5205350e+16,
             9.9999997e-06, -2.4705337e+16,
             1.5884370e+19],
           [-2.4705337e+16, -2.4705337e+16,
             1.8446743e+19, -8.3473774e+18,
            -2.4594892e+16]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.5
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.5}
Trying example: test_std(
    x=Array([[-2.0000100e+00,  1.8446743e+19,
            -7.8850107e+18,  1.8446743e+19,
            -3.1100245e+16],
           [-6.2392041e+16, -1.5205350e+16,
             9.9999997e-06,  1.8446743e+19,
             1.5884370e+19],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19, -8.3473774e+18,
            -2.4594892e+16]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.5
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.5}
Trying example: test_std(
    x=Array([[-6.2392041e+16,  1.8446743e+19,
            -7.8850107e+18,  1.8446743e+19,
            -3.1100245e+16],
           [-6.2392041e+16, -1.5205350e+16,
             9.9999997e-06,  1.8446743e+19,
             1.5884370e+19],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19, -8.3473774e+18,
            -2.4594892e+16]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.5
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.5}
Trying example: test_std(
    x=Array([[ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19],
           [-3.1100245e+16,  1.8446743e+19,
            -1.5205350e+16,  9.9999997e-06],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19],
           [-3.1100245e+16,  1.8446743e+19,
            -1.5205350e+16,  0.0000000e+00],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19],
           [-3.1100245e+16,  1.8446743e+19,
            -1.5205350e+16,  0.0000000e+00],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19],
           [ 1.8446743e+19,  1.8446743e+19,
             1.8446743e+19,  1.8446743e+19]], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([ 3.2980374e+18, -4.3380572e+18,
           -1.5989240e+16,  1.0000000e+01], dtype=array_api_strict.float32),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -2.38549134e+016,
             7.82976241e+006,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple(
    [(lambda x: x if x < ndim else x - 2 * ndim)(3),
     (lambda x: x if x < ndim else x - 2 * ndim)(2)],
)
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'axis': tuple(
     [(lambda x: x if x < ndim else x - 2 * ndim)(3),
      (lambda x: x if x < ndim else x - 2 * ndim)(2)],
 ),
 'correction': 0.0,
 'keepdims': True}
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
             7.82976241e+006,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple(
    [(lambda x: x if x < ndim else x - 2 * ndim)(3),
     (lambda x: x if x < ndim else x - 2 * ndim)(2)],
)
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'axis': tuple(
     [(lambda x: x if x < ndim else x - 2 * ndim)(3),
      (lambda x: x if x < ndim else x - 2 * ndim)(2)],
 ),
 'correction': 0.0,
 'keepdims': True}
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
             7.82976241e+006,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([(lambda x: x if x < ndim else x - 2 * ndim)(3), 0])
Draw 2 (correction): 0.0
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
             7.82976241e+006,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([0])
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([0]), 'correction': 0.0}
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0.0}
Trying example: test_std(
    x=Array([[-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([[ 0.00000000e+000,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [ 0.00000000e+000,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153],
           [-1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153,
            -1.74353291e+153]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Trying example: test_std(
    x=Array([[ 3.40282347e+038,
            -6.10674098e+016,
             3.38165033e+016],
           [-0.00000000e+000,
             1.33975003e-204,
            -1.86694487e+153],
           [-1.29778410e-177,
            -4.58938835e+016,
             1.34078079e+154],
           [-4.34140398e+016,
            -7.80051643e+015,
             1.28887764e-092]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0}
Trying example: test_std(
    x=Array([[ 3.40282347e+038,
            -6.10674098e+016,
             3.38165033e+016],
           [-0.00000000e+000,
             1.33975003e-204,
            -1.86694487e+153],
           [-1.29778410e-177,
            -4.58938835e+016,
             1.34078079e+154],
           [-4.34140398e+016,
            -7.80051643e+015,
             1.28887764e-092]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0}
Trying example: test_std(
    x=Array([[-5.26434399e+016,
            -6.10674098e+016,
             3.38165033e+016],
           [-0.00000000e+000,
             1.33975003e-204,
            -5.26434399e+016],
           [-5.26434399e+016,
            -4.58938835e+016,
             1.34078079e+154],
           [-4.34140398e+016,
            -7.80051643e+015,
             1.28887764e-092]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[-5.26434399e+016,
            -6.10674098e+016,
             3.38165033e+016],
           [-0.00000000e+000,
             1.33975003e-204,
            -5.26434399e+016],
           [-5.26434399e+016,
             1.33975003e-204,
             1.34078079e+154],
           [-4.34140398e+016,
            -7.80051643e+015,
             1.28887764e-092]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0., 0.],
           [0., 0., 0.],
           [0., 0., 0.],
           [0., 0., 0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0., 0.],
           [0., 0., 0.],
           [0., 0., 0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 3.16476029e+016],
           [-1.71461900e+016],
           [ 3.10588722e-122],
           [ 2.50000000e-001],
           [-5.93568753e+023]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): -1
Draw 2 (correction): 0.1390448613343254
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': -1, 'correction': 0.1390448613343254, 'keepdims': False}
Trying example: test_std(
    x=Array([[ 3.16476029e+016],
           [-1.71461900e+016],
           [ 3.10588722e-122],
           [ 1.49214950e-052],
           [-5.93568753e+023]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0}
Trying example: test_std(
    x=Array([[ 3.16476029e+016],
           [-1.71461900e+016],
           [ 3.10588722e-122],
           [ 1.49214950e-052],
           [-1.71461900e+016]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0}
Trying example: test_std(
    x=Array([[ 3.16476029e+16],
           [-1.71461900e+16],
           [-1.71461900e+16],
           [ 1.49214950e-52],
           [-1.71461900e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0}
Trying example: test_std(
    x=Array([[ 3.16476029e+16],
           [-1.71461900e+16],
           [-1.71461900e+16],
           [ 1.49214950e-52],
           [-1.71461900e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0}
Trying example: test_std(
    x=Array([[ 3.16476029e+16],
           [-1.71461900e+16],
           [-1.71461900e+16],
           [-1.71461900e+16],
           [-1.71461900e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0}
Trying example: test_std(
    x=Array([[-1.714619e+16],
           [ 0.000000e+00],
           [-1.714619e+16],
           [-1.714619e+16],
           [-1.714619e+16]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 2.35324393e+005],
           [-1.15937323e-307],
           [ 6.18435362e+016],
           [ 4.57694061e+055],
           [-3.40282347e+038]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple(
    [(lambda x: x if x < ndim else x - 2 * ndim)(2),
     (lambda x: x if x < ndim else x - 2 * ndim)(3)],
)
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple(
     [(lambda x: x if x < ndim else x - 2 * ndim)(2),
      (lambda x: x if x < ndim else x - 2 * ndim)(3)],
 ),
 'correction': 0.0,
 'keepdims': False}
Trying example: test_std(
    x=Array([[ 2.35324393e+005],
           [-1.15937323e-307],
           [ 6.18435362e+016],
           [ 4.57694061e+055],
           [-3.40282347e+038]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): 0
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': 0, 'correction': 0.0}
Trying example: test_std(
    x=Array([[ 2.35324393e+005],
           [-1.15937323e-307],
           [ 6.18435362e+016],
           [ 4.57694061e+055],
           [-3.40282347e+038]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([[ 2.35324393e+005],
           [-1.15937323e-307],
           [ 6.18435362e+016],
           [ 4.57694061e+055],
           [-3.40282347e+038]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[ 2.35324393e+005,
            -1.15937323e-307,
             6.18435362e+016,
             4.57694061e+055,
            -3.40282347e+038],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[ 2.35324393e+005,
            -1.15937323e-307,
             2.35324393e+005,
             4.57694061e+055,
            -3.40282347e+038],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055],
           [ 4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055,
             4.57694061e+055]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[ 8.08088588e+015,
            -1.75872646e+105],
           [ 8.08088588e+015,
            -1.86545736e+010],
           [-3.21812270e+015,
            -6.94512442e+015]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0
Draw 3 (keepdims): False
Draw 4 (kw): {'correction': 0}
Trying example: test_std(
    x=Array([[8.08088588e+15, 8.08088588e+15],
           [0.00000000e+00, 8.08088588e+15],
           [8.08088588e+15, 8.08088588e+15]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([[8.08088588e+15, 8.08088588e+15],
           [8.08088588e+15, 8.08088588e+15],
           [8.08088588e+15, 8.08088588e+15]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0.],
           [0., 0.],
           [0., 0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0}
Trying example: test_std(
    x=Array([[0., 0.],
           [0., 0.],
           [0., 0.]], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): True
Draw 4 (kw): {'axis': None, 'correction': 0.0, 'keepdims': True}
Trying example: test_std(
    x=Array([0., 0., 0.], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([0., 0., 0.], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): None
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': None, 'correction': 0.0, 'keepdims': False}
Trying example: test_std(
    x=Array([ 6.58839642e+153, -1.01045996e+153,
            6.16928203e+016,  1.42352149e-168,
            0.00000000e+000], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([0])
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([0]), 'keepdims': False}
Trying example: test_std(
    x=Array([ 6.58839642e+153, -1.01045996e+153,
            6.16928203e+016,  1.42352149e-168,
            0.00000000e+000], dtype=array_api_strict.float64),
    data=data(...),
)
Draw 1 (axis): tuple([0])
Draw 2 (correction): 0.0
Draw 3 (keepdims): False
Draw 4 (kw): {'axis': tuple([0]), 'correction': 0.0}
============================================================ warnings summary ============================================================
array_api_tests/test_statistical_functions.py::test_std
  /home/br/miniforge3/envs/array-api/lib/python3.12/site-packages/numpy/_core/_methods.py:222: RuntimeWarning: Degrees of freedom <= 0 for slice
    ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= Hypothesis Statistics ==========================================================
======================================================== short test summary info =========================================================
FAILED array_api_tests/test_statistical_functions.py::test_std - hypothesis.errors.FlakyStrategyDefinition: Inconsistent data generation! Data generation behaved differently between different runs. ...
====================================================== 1 failed, 1 warning in 0.81s ======================================================

Navigation Menu

Repository navigation

Skipped flaky tests #300

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Skipped flaky tests #300

Description

Activity

ev-br commented on Nov 17, 2024

asmeurer commented on Nov 18, 2024

asmeurer commented on Nov 18, 2024

ev-br commented on Nov 23, 2024

ev-br commented on Nov 25, 2024

asmeurer commented on Nov 25, 2024

asmeurer commented on Nov 25, 2024

ev-br commented on Feb 8, 2026

ev-br commented on Feb 8, 2026

ev-br commented on Feb 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions