[Tutorial] tutorial for tensorize by yzhliu · Pull Request #1774 · apache/tvm

yzhliu · 2018-09-26T02:16:44Z

@cowanmeg @anijain2305 @ZihengJiang @tqchen Pls help to review.

anijain2305

Thanks @yzhliu for this tutorial. This is very helpful. I had so many questions when I was trying to use tensorize the first time. This tutorial is answering many of those questions.

anijain2305 · 2018-09-26T05:32:53Z

+#
+N, M, L = 1024, 512, 64
+A = tvm.placeholder((N, L), name='A')
+B = tvm.placeholder((M, L), name='B')


Seems matrix B is transposed here. If not, A should be NxL and B should be LxM.

Yes, it's A * B^T, maybe define it more clearly beforehand? or it is better to do A * B?

I think a better definition will suffice here. I think it is necessary to have this here because it becomes easier later to map GEMV

anijain2305 · 2018-09-26T05:33:39Z

+B = tvm.placeholder((M, L), name='B')
+k = tvm.reduce_axis((0, L), name='k')
+C = tvm.compute((N, M), lambda i, j:
+                tvm.sum(A[i, k] * B[j, k], axis=k), name='C')


If the matrix B is not supposed to be transposed, it will be B[k, j]

anijain2305 · 2018-09-26T05:33:53Z

+=========
+**Author**: `Yizhi Liu <https://github.com/yzhliu>`_
+
+This is an introduction material on how to do tensorization in TVM.


do --> perform

anijain2305 · 2018-09-26T05:37:16Z

+as well as extend TVM to support new hardware architectures.
+
+In this tutorial, we will demonstrate the usage of tensorization.
+Notice that the examples are more for showing the functionality,


Rewrite suggestion - The purpose of this tutorial is to show the functionality and usage of tensorize instead of providing an efficient solution.

anijain2305 · 2018-09-26T05:38:18Z

+######################################################################
+# Schedule the Matmul
+# -------------------
+# Now suppose we had an accelerator supports matrix-vector multiplication (GEMV) as a hardware primitive,


Rewrite suggestion - Now, suppose we have an accelerator that

anijain2305 · 2018-09-26T05:48:00Z

+# Before scheduling the tensorization, we need to first define the intrinsic function for GEMV.
+# It includes two parts, the first is computing definition of GEMV,
+# TVM uses it to match the computing pattern in the original Matmul schedule.
+# The second is the definition of how to execute GEMV on the device.


It might be more concrete to say the second portion is done in "intrin_func" if we are avoiding writing comments in the code.

anijain2305 · 2018-09-26T05:53:35Z

+# Here :code:`tvm.decl_tensor_intrin` declares how to execute the computation :code:`c.op`.
+# Our implementation simply takes the inputs and outputs,
+# converts them to pointers and emit an external function call.
+# Note that tensorization requires user to specify :code:`offset_factor`,


Thanks for adding the offset_factor description. It seems important but I was not able to fully understand the description. Maybe an example would be helpful here.

yeah it is still not very clear given the description here. Actually offset_factor hasn't been used in current TVM for loading optimization, only CHECK(elem_offset % offset_factor == 0) is inserted into the codegen. I'm thinking about adding more details in the Buffer doc.

anijain2305 · 2018-09-26T05:56:00Z

+b = np.random.uniform(size=get_const_tuple(B.shape)).astype(dtype)
+c = tvm.nd.array(np.zeros(get_const_tuple(C.shape), dtype=dtype), ctx)
+func(tvm.nd.array(a, ctx), tvm.nd.array(b, ctx), c)
+np.testing.assert_array_almost_equal(c.asnumpy(), np.dot(a, b.T), decimal=2)


Is numpy.testing.assert_allclose any better?

anijain2305 · 2018-09-26T06:00:23Z

+                                        cc.access_ptr("w"), aa.access_ptr("r"), bb.access_ptr("r"),
+                                        m, l, bb.strides[0]))
+            return ib.get()
+        return _instrin(0), _instrin(1), _instrin(2)


Here, I think we need more details here. We can rewrite the code to have 3 functions - reduce_reset, reduce_update and body. This will make things clearer. I think this is important for first-time users. I had the same questions when I was trying to use tensorize.

anijain2305 · 2018-09-26T06:00:45Z

+b = np.random.uniform(size=get_const_tuple(B.shape)).astype(dtype)
+c = tvm.nd.array(np.zeros(get_const_tuple(C.shape), dtype=dtype), ctx)
+func(tvm.nd.array(a, ctx), tvm.nd.array(b, ctx), c)
+np.testing.assert_array_almost_equal(c.asnumpy(), np.dot(a, b.T), decimal=2)


ZihengJiang · 2018-09-26T08:16:12Z

            ib = tvm.ir_builder.create()
            x = ib.pointer("float32")
-            with ib.for_range(1, 10, name="i") as i:
+            with ib.for_range(0, 10, name="i") as i:


here is an i-1 in below expression

merrymercy · 2018-09-26T15:35:07Z

@@ -0,0 +1,276 @@
+"""
+Use Tensorize to Leverage Hardware Intrinsics
+=========


align the text and "===="

cowanmeg · 2018-09-28T00:55:46Z

+# which can take arbitrary size of reduce axis, but another axis needs to be no larger than 16.
+# Thus we break down the matmul loops to make the innermost loops a (16x64) GEMV.
+#
+s = tvm.create_schedule(C.op)


Adding another print of the IR here might be nice for comparison with the IR with the loops split.

cowanmeg · 2018-09-28T00:58:01Z

+# - within the inner most two loops, the index :code:`i` is fixed,
+# the access to the matrix :code:`A` only varies by :code:`k`,
+# which makes the access pattern of :code:`A` a "vector".
+# In order to leverage our "virtual" hardware's GEMV instruction,


Maybe use 'hypothetical hardware', if you're trying to emphasize this is for a imaginary backend.

cowanmeg · 2018-09-28T01:01:50Z

+# Define GEMV Tensorization Intrinsic
+# -----------------------------------
+# Before scheduling the tensorization, we need to first define the intrinsic function for GEMV.
+# It includes two parts, the first is computing definition of GEMV,


I would reword to:
It includes two parts, the first is a compute definition of GEMV.

yzhliu · 2018-09-30T00:19:58Z

Thanks @anijain2305 @ZihengJiang @merrymercy @cowanmeg Pls review again.

tqchen · 2018-10-01T20:55:53Z

@anijain2305 @ZihengJiang @merrymercy @cowanmeg please https://docs.tvm.ai/contribute/code_review.html#approve-and-request-changes-explicitly

anijain2305

Looks good to me.

anijain2305 · 2018-10-01T21:04:42Z

+#
+N, M, L = 1024, 512, 64
+A = tvm.placeholder((N, L), name='A')
+B = tvm.placeholder((M, L), name='B')


I think a better definition will suffice here. I think it is necessary to have this here because it becomes easier later to map GEMV

ZihengJiang

Other parts look good to me

ZihengJiang · 2018-10-01T22:54:13Z

+# For now :code:`bb.strides[0] == l`,
+# but later we will see how they can differ with more complicated schedules.
+#
+# Note that we use :code:`tvm.var("ldw")` as the first stride dimension for :code:`B`.


BTW, I am curious whether 'ldw' has any meaning.

I change it to s1 for stride1 instead.

tqchen · 2018-10-02T03:18:55Z

@yzhliu can you act on the final review comments?

tqchen · 2018-10-04T01:14:12Z

Thanks @yzhliu @cowanmeg @ZihengJiang @anijain2305 @merrymercy this is now merged!

zhenhuaw-me · 2019-02-26T06:36:51Z

Hi, have we considered writing tutorial with jupyter notebook? I notice that most tutorials are interactive. By interactive I mean, read a document segment, run a code segment, check the result of that code segment, and go on. Jupyter notebook seems very good at this.

yidawang · 2019-02-26T06:46:18Z

It is a Jupyter notebook. For this one, check out https://docs.tvm.ai/tutorials/language/tensorize.html#sphx-glr-tutorials-language-tensorize-py, go to the end of the page, you will see Download Jupyter notebook: tensorize.ipynb

zhenhuaw-me · 2019-02-26T06:49:26Z

Yes. I have seen the web page. Thank you :) Yida Wang <notifications@github.com> 于2019年2月26日周二下午2:46写道：

…

It is a Jupyter notebook. For this one, check out https://docs.tvm.ai/tutorials/language/tensorize.html#sphx-glr-tutorials-language-tensorize-py, go to the end of the page, you will see Download Jupyter notebook: tensorize.ipynb — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1774 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEtTjXH-iVykEMeYCQaW9cNh03KC-givks5vRNg9gaJpZM4W5x5P> .

-- 以上，祝好王振华

yzhliu added status: need review beginner-friendly labels Sep 26, 2018

yzhliu mentioned this pull request Sep 26, 2018

[TUTORIAL] Tutorial for tensorize #1379

Closed

anijain2305 reviewed Sep 26, 2018

View reviewed changes

ZihengJiang reviewed Sep 26, 2018

View reviewed changes

merrymercy reviewed Sep 26, 2018

View reviewed changes

cowanmeg reviewed Sep 28, 2018

View reviewed changes

tqchen added status: need update need update based on feedbacks and removed beginner-friendly labels Sep 29, 2018

yzhliu added 2 commits September 29, 2018 15:56

tutorial for tensorize

938e156

address the review comments

d422f6c

yzhliu force-pushed the tutorial-tensorize branch from c1c563b to d422f6c Compare September 29, 2018 23:46

anijain2305 approved these changes Oct 1, 2018

View reviewed changes

cowanmeg approved these changes Oct 1, 2018

View reviewed changes

ZihengJiang reviewed Oct 1, 2018

View reviewed changes

ldw -> s1

c25415a

apache deleted a comment from ZihengJiang Oct 2, 2018

yzhliu commented Oct 2, 2018

View reviewed changes

Comment thread tutorials/language/tensorize.py

tqchen merged commit 147f3ad into apache:master Oct 4, 2018

tqchen added status: accepted and removed status: need review status: need update need update based on feedbacks labels Oct 4, 2018

tqchen mentioned this pull request Oct 4, 2018

[TUTORIAL] Tutorial for inline micro-kernel ASM #1811

Closed

yzhliu deleted the tutorial-tensorize branch November 5, 2018 07:12

FrozenGene pushed a commit to FrozenGene/tvm that referenced this pull request Dec 27, 2018

[Tutorial] tutorial for tensorize (apache#1774)

e198c15

Conversation

yzhliu commented Sep 26, 2018

Uh oh!

anijain2305 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZihengJiang Sep 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzhliu commented Sep 30, 2018

Uh oh!

tqchen commented Oct 1, 2018

Uh oh!

anijain2305 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZihengJiang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tqchen commented Oct 2, 2018

Uh oh!

Uh oh!

tqchen commented Oct 4, 2018

Uh oh!

zhenhuaw-me commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yidawang commented Feb 26, 2019

Uh oh!

zhenhuaw-me commented Feb 26, 2019 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ZihengJiang Sep 26, 2018 •

edited

Loading

zhenhuaw-me commented Feb 26, 2019 •

edited

Loading