matrix_power issues

Hi, I'm working with Ivy where we are wrapping various ML frameworks and trying to align them with the Array API standard https://github.com/unifyai/ivy. The first problem I'm having is slight numerical issues. For example, when using TensorFlow I have gotten the error:

x = ivy.array([[2.5418657e+12, 2.5418657e+12, 2.5418657e+12],
           [2.5418657e+12, 2.5418657e+12, 2.5418657e+12],
           [2.5418657e+12, 2.5418657e+12, 2.5418657e+12]], dtype=float32)
y = ivy.array([[2.541866e+12, 2.541866e+12, 2.541866e+12],
           [2.541866e+12, 2.541866e+12, 2.541866e+12],
           [2.541866e+12, 2.541866e+12, 2.541866e+12]], dtype=float32)
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

For PyTorch, I have gotten the error:

x = ivy.array([[0., 0.],
           [0., 0.]], dtype=float32)
y = ivy.array([[ 0.,  0.],
           [ 0., -0.]], dtype=float32)

>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

These types of failures don't appear each time I run the tests, only appearing every few runs. I wonder if something similar to NumPy.allclose would be better for value checking, as matrix_power seems like a function susceptible to these very small errors.

Also, when testing against JAX I get a failure caused by the following falsifying example:

Falsifying example: test_matrix_power(
    x=ivy.array([[[1., 1.],
                [1., 1.]]], dtype=float32), n=-1,
)

which is strange given that here x isn't invertible and n is negative, which it says in test_matrix_power shouldn't occur.

The exactly equal thing is a known problem which I plan to fix soon. See #44. The matrix_power generating a non-invertible matrix for n=-1 is a bug. It's supposed to avoid doing that, so I'll need to take a look as to why that is happening.

@honno

I'm not able to reproduce the matrix_power n=-1 issue. The code should only be generating invertible matrices when n < 0. Can you run the test with -k matrix_power -s --hypothesis-verbosity=verbose and paste the output somewhere (@honno do you have any other suggestions on how to get figure out what is going on here?).

I should add that the NumPy implementation raises an exception for matrix_power(asarray([[[1., 1.], [1., 1.]]]), -1). So if the test can generate this example then NumPy should be failing as well, but I'm not able to get it to fail, even after running thousands of examples.

I'll also just add to make sure you are using the latest versions of both the array_api_tests and hypothesis.

I can have a look at this later, just to clarify this example

For PyTorch, I have gotten the error:

x = ivy.array([[0., 0.],
           [0., 0.]], dtype=float32)
y = ivy.array([[ 0.,  0.],
           [ 0., -0.]], dtype=float32)

>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

It's failing because the 4th zero is positive in x and negative in y. For many Array API functions the sign of a zero shouldn't matter... but there are cases where we do care! We don't do this in test_matrix_power though do we? I don't think we even use y as a variable name anywhere—did someone make a modification for an internal test suite that ivy uses?

@mattbarrett98

The first two failures are coming from testing exact equality in stacking (I think, @mattbarrett98 didn't show the full stack trace), which is something we do want to test differently. A difference between positive and negative zero can be something that is affected by the numerical details of a specific algorithm. I don't know what the actual inputs were here (again, @mattbarrett98 it would be helpful to provide that detail), but there aren't any cases in linalg where such things are specified.

Use assert_allclose() in the linalg tests for float inputs

Thanks I think this should resolve the first 2 issues! As for the +-0, in what situations do we care about them being different and why?

For the invertible matrix issue, yes we test against numpy also and this has never had this issue. I wonder if this is because in numpy it throws an exception when trying to invert the given matrix. However jax.numpy actually gives you a result when asking it to invert that same non-invertible matrix?

As for the +-0, in what situations do we care about them being different and why?

From the top of my head: manipulation functions because those shouldn't change the actual elements of an array, and some elementwise functions because of special cases.

This is a stack trace for the non-invertible matrix error:

=================================== FAILURES ===================================
______________________________ test_matrix_power _______________________________

    @pytest.mark.xp_extension('linalg')
>   @given(
        # Generate any square matrix if n >= 0 but only invertible matrices if n < 0
        x=matrix_power_n.flatmap(lambda n: invertible_matrices() if n < 0 else
                                 xps.arrays(dtype=xps.floating_dtypes(),
                                            shape=square_matrix_shapes)),
        n=matrix_power_n,
    )

ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:358: in test_matrix_power
    _test_stacks(func, x, res=res, true_val=true_val)
ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:81: in _test_stacks
    assert_equal(res_stack, decomp_res_stack)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = ivy.array([[ 2., -1.],
           [-1.,  1.]], dtype=float32)
y = ivy.array([[ inf, -inf],
           [-inf,  inf]], dtype=float32)

    def assert_exactly_equal(x, y):
        """
        Test that the arrays x and y are exactly equal.
    
        If x and y do not have the same shape and dtype, they are not considered
        equal.
    
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

ivy/ivy_tests/test_array_api/array_api_tests/array_helpers.py:179: AssertionError
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_matrix_power(
    x=ivy.array([[[1., 1.],
                [1., 1.]]], dtype=float32), n=-1,
)

This is the commit where the test failed. At the time, we were testing against this commit.

Perhaps the commits since 28th March have fixed this?

My guess is it has to do with the fact that it's Jax, which works differently from the other array libraries due to it being graph based. I don't think we've used Jax much in the test suite so far, so it's not surprising if there are some issues.

Either way, I'll take a look to see if I can reproduce the issue.

I was able to reproduce the matrix_power issue. I'm still not clear what is going on, but it appears to be fixed by #101.

Commented the inv function due to singular matrices being generated i…

I believe this should be fixed. Let me know if you still see these failures.

Navigation Menu

Repository navigation

matrix_power issues #117

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

matrix_power issues #117

Description

Activity

asmeurer commented on Apr 25, 2022

asmeurer commented on Apr 25, 2022

asmeurer commented on Apr 25, 2022

asmeurer commented on Apr 25, 2022

honno commented on Apr 25, 2022

asmeurer commented on Apr 25, 2022

mattbarrett98 commented on Apr 26, 2022

honno commented on Apr 26, 2022

djl11 commented on Apr 27, 2022

asmeurer commented on Apr 27, 2022

asmeurer commented on Apr 28, 2022

asmeurer commented on Nov 14, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions