Skip to content

compatability between mxnet, tvm and nnvm #518

Closed
Closed
compatability between mxnet, tvm and nnvm#518

Description

@kaishijeng
kaishijeng
opened on Oct 7, 2017

I have mxnet 0.11.0 installed and mxnet is running OK by itself
However, I got the following error of running python2 deploy_model_on_rasp.py from nnvm tutorial:

[10:06:53] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [10:06:53] /home/sky/2TB/src/mxnet/dmlc-core/include/dmlc/././any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE requested=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f3d0eb8bd3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(ZZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_iENKUlPN4dmlc3anyEE_clESR+0x136) [0x7f3d213177d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op13UpdateAttrMapERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt8functionIFvPN4dmlc3anyEEE+0xcc) [0x7f3d2243013c]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_i+0x190) [0x7f3d213075b0]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(+0x1f77c3) [0x7f3d212727c3]
[bt] (5) /lib64/ld-linux-x86-64.so.2(+0x106ba) [0x7f3d26c206ba]
[bt] (6) /lib64/ld-linux-x86-64.so.2(+0x107cb) [0x7f3d26c207cb]
[bt] (7) /lib64/ld-linux-x86-64.so.2(+0x158e2) [0x7f3d26c258e2]
[bt] (8) /lib64/ld-linux-x86-64.so.2(+0x10564) [0x7f3d26c20564]
[bt] (9) /lib64/ld-linux-x86-64.so.2(+0x14da9) [0x7f3d26c24da9]

terminate called after throwing an instance of 'dmlc::Error'
what(): [10:06:53] /home/fc/2TB/src/mxnet/dmlc-core/include/dmlc/././any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE requested=N4nnvm5OpMapISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS6_EES9_EEEE

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f3d0eb8bd3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(ZZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_iENKUlPN4dmlc3anyEE_clESR+0x136) [0x7f3d213177d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op13UpdateAttrMapERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt8functionIFvPN4dmlc3anyEEE+0xcc) [0x7f3d2243013c]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm2Op8set_attrISt8functionIFbRKNS_9NodeAttrsEPSt6vectorINS_6TShapeESaIS7_EESA_EEEERS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKT_i+0x190) [0x7f3d213075b0]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.0-py2.7.egg/mxnet/libmxnet.so(+0x1f77c3) [0x7f3d212727c3]
[bt] (5) /lib64/ld-linux-x86-64.so.2(+0x106ba) [0x7f3d26c206ba]
[bt] (6) /lib64/ld-linux-x86-64.so.2(+0x107cb) [0x7f3d26c207cb]
[bt] (7) /lib64/ld-linux-x86-64.so.2(+0x158e2) [0x7f3d26c258e2]
[bt] (8) /lib64/ld-linux-x86-64.so.2(+0x10564) [0x7f3d26c20564]
[bt] (9) /lib64/ld-linux-x86-64.so.2(+0x14da9) [0x7f3d26c24da9]

Aborted (core dumped)

Any idea why this happen?

Activity

  1. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    can you check if you can run unittest cases of nnvm compiler alone?

  2. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    if you have older version of tvm, it is likely that you will need to rebuild and install tvm with the same version as in nnvm repo.

  3. kaishijeng commented on Oct 7, 2017

    @kaishijeng
    kaishijeng
    on Oct 7, 2017
    Author

    How do I run unittest cases of nnvm compiler?
    My tvm is the latest one from github

  4. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    under nnvm, do

    nosetests -v tests/python/unittest 
nosetests -v tests/python/compiler

  5. kaishijeng commented on Oct 7, 2017

    @kaishijeng
    kaishijeng
    on Oct 7, 2017
    Author

    Below is what I got:

    test_graph.test_json_pass ... ok
    test_graph.test_json_pass_with_attr ... ok
    test_graph.test_graph_json_attr ... ok
    test_graph.test_list_args ... ok
    test_graph.test_infer_shape ... ok
    test_graph.test_infer_shape_known_partial ... ok
    test_graph.test_infer_type ... ok
    test_graph.test_plan_memory ... ok
    test_graph.test_print_graph_ir ... ok
    test_infer_shape.test_dense ... ok
    test_infer_shape.test_concatenate ... ok
    test_infer_shape.test_expand_dims ... ok
    test_infer_shape.test_split ... ok
    test_infer_shape.test_batchnorm ... ok
    test_infer_shape.test_flatten ... ok
    test_infer_shape.test_conv2d ... ok
    test_infer_shape.test_conv2d_transpose ... ok
    test_infer_shape.test_max_pool2d ... ok
    test_infer_shape.test_global_pool2d ... ok
    test_infer_shape.test_reshape ... ok
    test_infer_shape.test_transpose ... ok
    test_infer_shape.test_broadcast_to ... ok
    test_infer_shape.test_broadcast_binary ... ok
    test_infer_shape.test_reduce ... ok
    test_symbol.test_dense ... ok
    test_symbol.test_compose ... ok
    test_symbol.test_default_input ... [10:43:26] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [10:43:26] src/core/symbolic.cc:309: Not enough argument to call operator elemwise_add

    Stack trace returned 10 entries:
    [bt] (0) /usr/local/lib/python2.7/dist-packages/tvm-0.1.0-py2.7-linux-x86_64.egg/tvm/libtvm.so(_ZN4dmlc15LogMessageF atalD1Ev+0x3c) [0x7fab29378d3c]
    [bt] (1) /home/sky/2TB/src/nnvm/python/nnvm/../../lib/libnnvm_compiler.so(ZN4nnvm6Symbol7ComposeERKN4dmlc10array_vie wIPKS0_EERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4hashISE_ESt8equal_toISE_ESaIS t4pairIKSE_S4_EEERSK+0x10ef) [0x7fab2873eccf]
    [bt] (2) /home/sky/2TB/src/nnvm/python/nnvm/../../lib/libnnvm_compiler.so(NNSymbolCompose+0x2e0) [0x7fab28729470]
    [bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fab3f0a1e40]
    [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7fab3f0a18ab]
    [bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7fab3f2b13df]
    [bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7fab3f2b5d82]
    [bt] (7) /usr/bin/python(PyObject_Call+0x43) [0x4b0cb3]
    [bt] (8) /usr/bin/python(PyEval_EvalFrameEx+0x5faf) [0x4c9faf]
    [bt] (9) /usr/bin/python(PyEval_EvalCodeEx+0x255) [0x4c2765]

    ok
    test_symbol.test_copy ... ok
    test_symbol.test_op_name ... ok
    test_top_level1.test_dense ... ok
    test_top_level1.test_concatenate_split ... ok
    test_top_level1.test_expand_dims ... ok
    test_top_level1.test_unary ... ok
    test_top_level1.test_batchnorm ... ok
    test_top_level2.test_conv2d ... ok
    test_top_level2.test_max_pool2d ... ok
    test_top_level3.test_reshape ... ok
    test_top_level3.test_scalar_op ... ok
    test_top_level3.test_leaky_relu ... ok
    test_top_level4.test_binary_broadcast ... ok
    test_top_level4.test_broadcast_to ... ok

    Ran 41 tests in 0.098s

    OK

  6. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    I do not have a quite clear clue on what is happening in mxnet side. You can try to upgrade mxnet using the latest pip version, that is how we build docker image to test mxnet frontend https://github.com/dmlc/nnvm/blob/master/tests/ci_build/Dockerfile.gpu

  7. kaishijeng commented on Oct 7, 2017

    @kaishijeng
    kaishijeng
    on Oct 7, 2017
    Author

    If I import mxnet before nnvm, the issue seems to disappear.

    However, I got a different error below which says llvm is not enabled:

    TVM: Initializing cython mode...
    [11:16:58] /home/sky/2TB/src/tvm/dmlc-core/include/dmlc/logging.h:308: [11:16:58] /home/sky/2TB/src/tvm/src/codegen/codegen.cc:27: Check failed: bf != nullptr Tar get llvm is not enabled

    Stack trace returned 10 entries:

  8. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    this is mainly because tvm need to build with llvm support, modify config.mk to do that

  9. kaishijeng commented on Oct 7, 2017

    @kaishijeng
    kaishijeng
    on Oct 7, 2017
    Author

    I installed llvm 5.0/clang 5.0 and is able to pass this error now.
    However, another error occurs:

    Intrinsic name not mangled correctly for type arguments! Should be: llvm.fmuladd.f32
    float (float, float, float)* @llvm.fmuladd.f32.f32.f32
    LLVM ERROR: Broken function found, compilation aborted!

  10. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    This seems to be an error that was fixed in later version of LLVM5, try llvm from here http://apt.llvm.org/

  11. tqchen commented on Oct 7, 2017

    @tqchen
    tqchen
    on Oct 7, 2017
    Member

    The problem of old llvm should also be fixed by #519

  12. kaishijeng commented on Oct 8, 2017

    @kaishijeng
    kaishijeng
    on Oct 8, 2017
    Author

    It works OK with PR#519
    By the way, how do I measure execution time on Raspberry PI and how to compare with a regular mxnet mobilenet on PI?

    Thanks,

  13. tqchen commented on Oct 8, 2017

    @tqchen
    tqchen
    on Oct 8, 2017
    Member

    You can use the RPC module which comes with remote executor, see https://github.com/dmlc/nnvm/blob/master/examples/benchmark/rasp_imagenet_bench.py and
    http://nnvm.tvmlang.org/tutorials/deploy_model_on_rasp.html#sphx-glr-tutorials-deploy-model-on-rasp-py

    For MXNet reference, we do need to do it manually by scripting a python script

  14. tqchen commented on Oct 8, 2017

    @tqchen
    tqchen
    on Oct 8, 2017
    Member

    I am going to close this issue as the original problem has been resolved. Feel free to open another one if you have other questions

  15. tqchen
    closed this as completedon Oct 8, 2017

  16. 23 remaining items

  17. zxybazh
    added 2 commits that reference this issue on Jan 7, 2022

    Squashed commit

    d1fce89

    Squashed commit

    05f3aba
  18. junrushao
    added a commit that references this issue on Jan 27, 2022

    Squashed commit

    3c6882a
  19. Hzfengsy
    added a commit that references this issue on Feb 2, 2022

    Squashed commit

    fc54ebb
  20. Hzfengsy
    added a commit that references this issue on Feb 12, 2022

    Squashed commit

    d440d79
  21. Hzfengsy
    added 2 commits that reference this issue on Feb 19, 2022

    Squashed commit

    17018cc

    Squashed commit

    623c0a6
  22. junrushao
    added 3 commits that reference this issue on Feb 20, 2022

    Squashed commit

    1d8cc54

    Squashed commit

    893555f

    Squashed commit

    46a98f1
  23. zxybazh
    added 5 commits that reference this issue on Feb 21, 2022

    Squashed commit

    eb8def7

    Squashed commit

    6f04005

    Squashed commit

    ed92495

    Squashed commit

    fc2f258

    Squashed commit

    0af0f71
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions