Skip to content

[ROCM] Failed topi unittests for rocm backend #552

Closed
Closed
[ROCM] Failed topi unittests for rocm backend#552
Labels
type: bug

Description

@masahi
masahi
opened on Oct 13, 2017

[Update 10/23] All failing test cases were resolved.

[Update 10/20] Currently 3 tests in topi unittests are failing.
See the summary here.

test_topi_conv2d_nchw.py

3 out of 12 test cases fail.

case verify_conv2d_nchw(1, 3, 224, 64, 7, 3, 2)


$ python test_topi_conv2d_nchw.py 
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=0

(mismatch 99.98687454346238%)
 x: array([[[[ 17.740227,  25.133751,  23.486025, ...,  25.639071,  24.148636,
           21.255533],
         [ 24.602934,  33.843094,  35.173992, ...,  39.136459,  36.724586,...
 y: array([[[[ 16.973913,  25.437023,  24.094303, ...,  27.416625,  25.55851 ,
           22.340623],
         [ 25.350509,  35.822084,  36.533454, ...,  38.809231,  36.673988,...

case verify_conv2d_nchw(1, 128, 28, 128, 3, 1, 1)

$ python test_topi_conv2d_nchw.py 
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 128, 28, 28), 'float32', 1, 1, (128, 128, 3, 3))
testing  rocm  backend
Memory access fault by GPU node-1 on address 0x1000. Reason: Page not present or supervisor privilege.
Aborted (core dumped)

case verify_conv2d_nchw(1, 256, 14, 256, 3, 1, 1)

$ python test_topi_conv2d_nchw.py 
Use memoize topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl(5, (1, 256, 14, 14), 'float32', 1, 1, (256, 256, 3, 3))
testing  rocm  backend
Memory access fault by GPU node-1 on address 0x5fc4e1000. Reason: Page not present or supervisor privilege.
Aborted (core dumped)

test_topi_depthwise_conv2d.py

2 out of 15 test cases fail.

case depthwise_conv2d_with_workload_nhwc(1, 728, 32, 1, 3, 1, "SAME")

$ python test_topi_depthwise_conv2d.py 
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=0

(mismatch 29.669793097527474%)
 x: array([[[[ 0.79067 ,  0.833293,  1.011379, ...,  0.855761,  0.685508,
           1.101951],
         [ 1.64011 ,  1.610993,  1.381752, ...,  1.928582,  1.182396,...
 y: array([[[[ 0.79067 ,  0.833293,  1.011379, ...,  0.855761,  0.685508,
           1.101951],
         [ 1.64011 ,  1.610993,  1.381752, ...,  1.928582,  1.182396,...

case depthwise_conv2d_with_workload_nhwc(1, 728, 32, 1, 3, 1, "VALID")

$ python test_topi_depthwise_conv2d.py 
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=0

(mismatch 27.69184981684981%)
 x: array([[[[ 2.178229,  1.748752,  2.032793, ...,  2.637433,  1.739626,
           1.027414],
         [ 1.45854 ,  1.976049,  1.973471, ...,  1.987515,  1.736224,...
 y: array([[[[ 2.178229,  1.748752,  2.032793, ...,  2.637433,  1.739626,
           1.027414],
         [ 1.45854 ,  1.976049,  1.973471, ...,  1.987515,  1.736224,...

test_topi_pooling.py

1 out of 8 test cases fails.

case verify_pool(1, 256, 32, 2, 2, [0, 0], 'avg')

$ python test_topi_pooling.py 
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=0

(mismatch 50.0%)
 x: array([[[[ 0.383371,  0.234747,  0.431097, ...,  0.498689,  0.369492,
           0.783951],
         [ 0.551467,  0.405494,  0.410852, ...,  0.414632,  0.581095,...
 y: array([[[[ 0.383371,  0.234747,  0.431097, ...,  0.498689,  0.369492,
           0.783951],
         [ 0.551467,  0.405494,  0.410852, ...,  0.414632,  0.581095,...

Activity

  1. tqchen
    added
    type: bug
    on Oct 13, 2017

  2. tqchen commented on Oct 13, 2017

    @tqchen
    tqchen
    on Oct 13, 2017
    Member

    cc @adityaatluri

  3. tqchen
    changed the title [-]Failed topi unittests for rocm backend[/-] [+][ROCM] Failed topi unittests for rocm backend[/+] on Oct 13, 2017

  4. tqchen commented on Oct 15, 2017

    @tqchen
    tqchen
    on Oct 15, 2017
    Member

    #558

  5. masahi commented on Oct 15, 2017

    @masahi
    masahi
    on Oct 15, 2017
    MemberAuthor

    The segfault bug was resolved by #558

  6. masahi commented on Oct 20, 2017

    @masahi
    masahi
    on Oct 20, 2017
    MemberAuthor

    Math function is supported with #570, 2 failing tests resolved.
    But found another failing test in test_topi_pooling.py

    Updated to reflect the latest status.

  7. masahi commented on Oct 22, 2017

    @masahi
    masahi
    on Oct 22, 2017
    MemberAuthor

    verify_conv2d_nchw(1, 128, 28, 128, 3, 1, 1) and verify_conv2d_nchw(1, 256, 14, 256, 3, 1, 1) were failing because of too aggressive unrolling when unroll_explicit = True.

    They were resolved by #576 .

  8. masahi commented on Oct 23, 2017

    @masahi
    masahi
    on Oct 23, 2017
    MemberAuthor

    The rest of failing tests cases were due to a large thread block size.
    They were all resolved by #577.

    We still do not know why rocm backend cannot handle thread block of size 512 or 728.

  9. masahi commented on Oct 23, 2017

    @masahi
    masahi
    on Oct 23, 2017
    MemberAuthor

    All unit tests are good now. Closing.
    nosetests output:

    $ nosetests -v topi/tests/python/
test_topi_basic.test_util ... ok
test_topi_basic.test_ewise ... ok
test_topi_broadcast.test_broadcast_to ... [09:44:50] src/runtime/opencl/opencl_device_api.cc:195: Initialize OpenCL platform 'AMD Accelerated Parallel Processing'
[09:44:50] src/runtime/opencl/opencl_device_api.cc:215: opencl(0)='gfx803' cl_device_id=0x7f03c8727720
ok
test_topi_broadcast.test_broadcast_binary ... ok
test_topi_clip.test_clip ... ok
test_topi_conv2d.test_conv2d ... ok
test_topi_conv2d_hwcn.test_conv2d_hwcn ... [09:44:57] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:57] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:57] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:58] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:58] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:58] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:58] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:44:58] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:02] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:02] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:02] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:02] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:09] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:09] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:09] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:10] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:14] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:14] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:14] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:15] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:15] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:15] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:15] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:15] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:18] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:18] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:19] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:19] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:25] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:25] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:25] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
[09:45:25] src/pass/vectorize_loop.cc:303: Detect vector condition in Vectorized Loop, scalarizing...
ok
test_topi_conv2d_nchw.test_conv2d_nchw ... ok
test_topi_dense.test_dense ... ok
test_topi_depthwise_conv2d.test_depthwise_conv2d ... ok
test_topi_depthwise_conv2d_back_input.test_topi_depthwise_conv2d_backward_input_nhwc ... ok
test_topi_depthwise_conv2d_back_weight.test_topi_depthwise_conv2d_backward_weight_nhwc ... ok
test_topi_dilate.test_dilate ... ok
test_topi_pooling.test_pool ... ok
test_topi_pooling.test_global_pool ... ok
test_topi_reduce.test_reduce_map ... [09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03c36af960)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f03c3bba530)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp, 0x7f03c36aa150)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03c38e3cf0)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f03b61214b0)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03b9dd8890)
[09:49:04] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f03b5b0db20)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp, 0x7f03c3bba530)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03c38ff430)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f03c8be7ee0)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03c3613670)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f0378e244a0)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp, 0x7f03c1938520)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.repl, 0x7f03c38b7820)
[09:49:05] src/schedule/bound.cc:98: not in feed graph consumer = compute(compute_red_temp.rf, 0x7f03c3613670)
ok
test_topi_relu.test_relu ... ok
test_topi_relu.test_leaky_relu ... ok
test_topi_softmax.test_softmax ... ok
test_topi_softmax.test_log_softmax ... ok
test_topi_transform.test_expand_dims ... ok
test_topi_transform.test_tranpose ... ok
test_topi_transform.test_reshape ... ok
test_topi_transform.test_squeeze ... ok
test_topi_transform.test_concatenate ... ok
test_topi_transform.test_split ... ok

----------------------------------------------------------------------
Ran 26 tests in 261.625s

OK
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_clip.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_conv2d.verify_conv2d.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_conv2d_hwcn.verify_hwcn.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_conv2d.verify_con2d_nchw.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_dense.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_depthwise_conv2d.nchw.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_depthwise_conv2d.nhwc.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_depthwise_conv2d_backward_input.nhwc.get_ref_data.pkl
Save memoize result to .pkl_memoize_py3/topi.tests.test_topi_depthwise_conv2d_backward_weight.nhwc.get_ref_data.pkl
  10. masahi
    closed this as completedon Oct 23, 2017
  11. zxybazh
    added 2 commits that reference this issue on Jan 7, 2022

    Squashed commit

    f98a8e9

    Squashed commit

    05f3aba
  12. junrushao
    added 2 commits that reference this issue on Jan 27, 2022

    Hotfix for software pipeline (apache#552)

    f998006

    Squashed commit

    3c6882a
  13. Hzfengsy
    added a commit that references this issue on Feb 2, 2022

    Squashed commit

    fc54ebb
  14. Hzfengsy
    added a commit that references this issue on Feb 12, 2022

    Squashed commit

    d440d79
  15. Hzfengsy
    added 2 commits that reference this issue on Feb 19, 2022

    Squashed commit

    17018cc

    Squashed commit

    623c0a6
  16. junrushao
    added 3 commits that reference this issue on Feb 20, 2022

    Squashed commit

    1d8cc54

    Squashed commit

    893555f

    Squashed commit

    46a98f1
  17. zxybazh
    added 5 commits that reference this issue on Feb 21, 2022

    Squashed commit

    eb8def7

    Squashed commit

    6f04005

    Squashed commit

    ed92495

    Squashed commit

    fc2f258

    Squashed commit

    0af0f71
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions