[AutoTVM]Core functionality for Graph tuner by kevinthesun · Pull Request #2184 · apache/tvm

kevinthesun · 2018-11-28T01:05:57Z

This is the core function for graph tuner: #1585

Tasks yet to be done:

Move implementation of layout_transform into topi. @yzhliu working on it.
Modify x86 autotvm tutorial to include graph tuning step.
Support Relay model as input.

We can start reviewing for core parts. @tqchen @merrymercy @eqy @yzhliu @masahi

kevinthesun · 2018-12-22T00:07:55Z

@yzhliu Do we have a time line for layout_transform op?

kevinthesun · 2019-03-08T22:25:15Z

@tqchen @merrymercy @eqy @masahi @yzhliu All tasks are done and this PR is ready to be reviewed.

masahi · 2019-03-12T07:35:05Z

I want to try this on my box, @kevinthesun can you tell me how long should I expect tuning would take roughly? Say, I want to tune vgg16 or resnet50 on a single desktop. Assuming kernel level tuning is done.

yzhliu · 2019-03-12T07:54:35Z

@masahi The graph-tuner itself would be fairly fast (the time I tried resnet it took less than a minute, probably seconds, on a server cpu). When I say "itself", it means the performance for each layer has already tuned (and recorded) by tensor-level AutoTVM. But I think there exists some bug in current Relay's AlterOpLayout https://discuss.tvm.ai/t/relay-alter-op-layout-pass-regression/1870 though. @kevinthesun Correct me if I'm wrong.

masahi · 2019-03-12T08:22:53Z

In tune_relay_x86.py, I can see that we first tune each layer individually using standard AutoTVM workflow. At this stage, I assume the tuned layout is fixed (all conv ops are NCHW16c or NCHW8c depending on vector length). But for graph tuning, we also need timing record for NCHW64c, NCHW32c ... right? Does tune_graph(...) itself launch tensor level tuning jobs, in addition to solving a combinatorial problem?

I have a good idea of how long tune_kernels(...) would take, but not for tune_graph(...). If my understanding is correct, tune_graph should take a very long time if I start from scratch.

def tune_and_evaluate(tuning_opt):
    # extract workloads from relay program
    print("Extract tasks...")
    net, params, data_shape, out_shape = get_network(model_name, batch_size)
    tasks = autotvm.task.extract_from_program(net, target=target,
                                              params=params, ops=(relay.op.nn.conv2d,))

    # run tuning tasks
    print("Tuning...")
    tune_kernels(tasks, **tuning_opt)
    tune_graph(net, data_shape, log_file, graph_opt_sch_file)
    ...

kevinthesun · 2019-03-12T17:22:24Z

@masahi All tensor level jobs are done by autotvm. In this tutorial, all combinations defined in search space are covered.

masahi · 2019-03-12T22:38:48Z

 from tvm.relay import testing
 from tvm.autotvm.tuner import XGBTuner, GATuner, RandomTuner, GridSearchTuner
+from tvm.autotvm.graph_tuner import DPTuner, PBQPTuner
+from tvm.autotvm.graph_tuner.utils import relay_get_conv2d_NCHWc_AVX_workload, infer_conv2d_layout_shape_avx


It seems relay_get_conv2d_NCHWc_AVX_workload -> get_conv2d_NCHWc_AVX_workload

masahi · 2019-03-12T22:42:02Z

+    cfg = get_config()
+    cfg.add_flop(-1)
+    data = args[0]
+    out = topi.nn.layout_transform(*args)


There is no topi.nn.layout_transform. We have topi.layout_transform instead, here.

But I still get the following error. Maybe the API changed recently?

TypeError: layout_transform() takes 3 positional arguments but 6 were given

yes top.nn.layout_transform was changed recently.

masahi · 2019-03-12T22:53:05Z

@kevinthesun do you mean the tuning for non standard layouts such as NCHW64c, NCHW32c is supposed to happen during tune_kernels(tasks, **tuning_opt)?

I ran this tutorial yesterday with resnet 18, but the log indicates there are only 12 tasks. It doesn't seem it is trying many layouts exhaustively.

$ python tune_relay_x86.py 
Extract tasks...
Tuning...
[Task  1/12]  Current/Best:  406.15/ 549.47 GFLOPS | Progress: (400/400) | 1045.37 s Done.
[Task  2/12]  Current/Best:  235.75/ 568.55 GFLOPS | Progress: (360/360) | 981.12 s Done.
[Task  3/12]  Current/Best:  123.06/ 604.78 GFLOPS | Progress: (648/648) | 1841.81 s Done.
[Task  4/12]  Current/Best:  128.86/ 544.66 GFLOPS | Progress: (576/576) | 1552.07 s Done.
[Task  5/12]  Current/Best:  213.31/ 679.53 GFLOPS | Progress: (768/768) | 2152.06 s Done.
[Task  6/12]  Current/Best:  369.23/ 604.23 GFLOPS | Progress: (672/672) | 1808.16 s Done.
[Task  7/12]  Current/Best:   26.45/ 679.44 GFLOPS | Progress: (784/784) | 2194.49 s Done.
[Task  8/12]  Current/Best:   86.61/ 734.09 GFLOPS | Progress: (252/252) | 722.79 s Done.
[Task  9/12]  Current/Best:  140.74/ 630.58 GFLOPS | Progress: (784/784) | 2047.28 s Done.
[Task 10/12]  Current/Best:  103.17/ 635.58 GFLOPS | Progress: (672/672) | 1760.41 s Done.
[Task 11/12]  Current/Best:  108.43/ 652.48 GFLOPS | Progress: (576/576) | 1516.26 s Done.
[Task 12/12]  Current/Best:   23.81/ 547.43 GFLOPS | Progress: (360/360) | 942.57 s Done.

kevinthesun · 2019-03-14T17:44:58Z

@masahi This progress graph shows there are 12 conv2d workloads to be tuned. For each workload, there should be a bunch of different schedules tried. You can check the resulting log file to see different config for each workload.

masahi · 2019-03-14T23:05:13Z

Ah, I looked at the autotvm template definition for x86 conv, and realized that different layouts are tuned through ic_bn and oc_bn tuning parameters. Thanks, that clears up my confusion.

eqy · 2019-03-14T23:23:33Z

Ah, I looked at the autotvm template definition for x86 conv, and realized that different layouts are tuned through ic_bn and oc_bn tuning parameters. Thanks, that clears up my confusion.

Yes, that was also a confusing part of the organization to me as well :). I think this currently means that we do not really have a way to just tune one specific layout.

kevinthesun · 2019-03-20T00:03:38Z

@yzhliu The failing tests are not related to this PR, and I can successfully pass on my local machine.

kevinthesun · 2019-05-09T23:07:08Z

@tqchen test_rank_zero and test_rank_zero_bound_checkers are failing, which should be unrelated to this PR. I can reproduce the failing tests locally while running all python unit tests. However, if I only run test_codegen_llvm.py, those two tests can pass. Do you have any insight about this kind of behavior?

yzhliu · 2019-05-12T03:22:47Z

+RULE_OUT_NODE_NAMES = ["Tuple", "TupleGetItem", "batch_flatten", "transpose", "reshape",
+                       "multibox_prior", "multibox_transform_loc", "where",
+                       "non_max_suppression", "strided_slice"]
+INVALID_LAYOUT_TIME = 1000000000.0


add comment for time unit

yzhliu · 2019-05-12T03:35:01Z

+        """
+        self._node_list = []
+        self._layout_transform_dict = {}
+        self._layout_transform_matrix_dict = {}


would layout_transform_perf_records and layout_transform_interlayer_cost be better name?

yzhliu · 2019-05-12T03:44:22Z

+    node_dict : dictionary from tvm.relay.Expr to int
+        Dictionary to record node index
+
+    node_list : list of dictionary


could you elaborate what's in the dictionary?

yzhliu · 2019-05-12T03:51:20Z

+                        # Only replace the first input tensor
+                        input_workload = input_node["workloads"][0]
+                        first_tensor = input_workload[1]
+                        dtype = first_tensor[-1]


suggest to add comment describe the content of these lists, the index looks confusing, (and makes me panic about out-of-bound)

Added explanations in docstring of expr2graph

yzhliu · 2019-05-12T03:52:50Z

+                        dtype = first_tensor[-1]
+                        new_shape = tuple([val.value for val in node_entry["types"][0].shape])
+                        actual_workload = (input_workload[0],) + \
+                                          ((new_shape + (dtype,)),) + input_workload[2:]


why is a new shape generated?

The elemwise-like node shape might not be necessarily the same as input nodes. For example, a pooling following a conv2d, befoer going into an elemwise node.

yzhliu · 2019-05-12T04:49:03Z

+        if key not in out:
+            raise RuntimeError("%s not in output dictionary." % str(key))
+    if not os.path.isfile(log_file):
+        raise RuntimeError("No log file with name %s exists." % log_file)


can we assert some results instead of just ensure there's an output? e.g., assert out[key] == xxx, or assert max(out) == xxx

yzhliu · 2019-05-12T04:49:46Z

+    for op_name, node in zip(op_name_list, node_list):
+        if op_name != node["op"]:
+            raise RuntimeError("Node operator mismatch: expecting %s but got %s"
+                               % (str(op_name), str(node["op"])))


can we assert the order?

yzhliu · 2019-05-12T04:50:27Z

+    out = has_multiple_inputs(node_list, node_idx, input_names)
+    if out != expected_result:
+        raise RuntimeError("Output mismatch: expecting checking %s to be %s but got %s."
+                           % (node_list[node_idx]["op"], str(expected_result), str(out)))


assert out == expected_result, "Output mismatch ..."

yzhliu · 2019-05-12T04:51:16Z

+    input_names = ["data"]
+    out = get_direct_ancestor(node_list, visited_dict, target_ops, 5, input_names)
+    if len(out) != 2 or out != [2, 0]:
+        raise RuntimeError("Output mismatch: expecting [2, 0] but got %s." % str(out))


assert out == [2, 0]

yzhliu · 2019-05-12T04:53:42Z

+    expected_out = [records[3][0].config, records[1][0].config, records[2][0].config]
+    if expected_out != out:
+        raise RuntimeError("Output mismatch: expecting %s but got %s"
+                           % (str(expected_out), str(out)))


assert expected_out == out, xxx

kevinthesun · 2019-05-19T05:48:40Z

@yzhliu comments addressed.

yzhliu

The overall looks good to me. While any idea how could we address the ci failure?

yzhliu · 2019-05-29T23:36:16Z

Thanks!

* Add graph tuning * Add tests * Fix tests * Fix pylint * Small fix for docstring * Minor fix * Support fetching workload from relay expr * Simplify benchmark layout transformation * Add relay support * Fix infer layout func name * Refactor internal data representation * Fix issues * Add PBQP solver * Fix layout transform check * Add PBQPTuner test * Fix lint * Update tutorial * Fix tutorial * Fix lint * Add relay test * Remove nnvm since nnvm graph can be converted to relay function * Modify benchmark layout wrt new layout_transform api * Fix lint * Update docstring for DP tuner * Refactor traverse graph * Support graph tuning for multiple target operators * Fix fetching workloads * Add x86 depthwise_conv2d infer_layout * Fix x86 depthwise_conv2d autotvm * Fix PBQP tuner * Fix DP tuner * Generate dummy layout transform record * Update tutorial * Modify layout records name * Add ASF header * Add ASF header for testing files * Fix test * Fix topi fetching * Some refactors * Fix lint * Fix tutorial * Rename test files * Fix doc typo * Add test case note link

kevinthesun force-pushed the GraphTuner branch from c4fe94e to 64eaa46 Compare December 6, 2018 19:37

yzhliu self-assigned this Dec 11, 2018

icemelon added the status: WIP label Dec 21, 2018

kevinthesun force-pushed the GraphTuner branch 2 times, most recently from 028c984 to d2b9380 Compare January 22, 2019 23:56

yzhliu mentioned this pull request Feb 19, 2019

[DEV] TVM v0.6 Roadmap #2623

Closed

9 of 28 tasks

kevinthesun force-pushed the GraphTuner branch 2 times, most recently from 6ba6936 to deaaaf4 Compare March 7, 2019 20:59

kevinthesun changed the title ~~[WIP][AutoTVM]Core functionality for Graph tuner~~ [AutoTVM]Core functionality for Graph tuner Mar 7, 2019

kevinthesun force-pushed the GraphTuner branch from e7cab53 to 5fd4fa5 Compare March 8, 2019 22:10

yzhliu added status: need review and removed status: WIP labels Mar 12, 2019

kevinthesun force-pushed the GraphTuner branch 3 times, most recently from 43137bd to f88cba9 Compare March 12, 2019 19:52

masahi reviewed Mar 12, 2019

View reviewed changes

kevinthesun force-pushed the GraphTuner branch 2 times, most recently from f067487 to a729da7 Compare March 19, 2019 23:21

Wang and others added 6 commits May 9, 2019 15:03

Update tutorial

1a2937f

Modify layout records name

80f205b

Add ASF header

1a44730

Add ASF header for testing files

035a9d2

Fix test

2164dd4

Fix topi fetching

9058822

yzhliu reviewed May 12, 2019

View reviewed changes

yzhliu added the status: need update need update based on feedbacks label May 12, 2019

Some refactors

da65b65

kevinthesun force-pushed the GraphTuner branch from 416d392 to da65b65 Compare May 13, 2019 23:14

kevinthesun added 2 commits May 13, 2019 16:46

Fix lint

a19d349

Fix tutorial

0b51758

yzhliu reviewed May 20, 2019

View reviewed changes

Comment thread python/tvm/autotvm/graph_tuner/pbqp_tuner.py Outdated

kevinthesun added 2 commits May 28, 2019 12:26

Rename test files

da51cf1

Fix doc typo

6e563b7

loiccordone approved these changes May 29, 2019

View reviewed changes

yzhliu approved these changes May 29, 2019

View reviewed changes

Comment thread tests/python/unittest/test_graph_tuner_core.py

Add test case note link

0516a5b

yzhliu merged commit c8a0f52 into apache:master May 29, 2019

yzhliu added status: accepted and removed status: need review status: need update need update based on feedbacks labels May 29, 2019

kevinthesun deleted the GraphTuner branch June 13, 2019 20:10

loiccordone mentioned this pull request Jun 25, 2019

#3353 (frontend API change) breaks Graph tuning in AutoTVM #3432

Closed

tqchen mentioned this pull request Nov 8, 2019

[RELEASE][DRAFT] TVM v0.6 Release candidate #4259

Closed

Conversation

kevinthesun commented Nov 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinthesun commented Dec 22, 2018

Uh oh!

kevinthesun commented Mar 8, 2019

Uh oh!

masahi commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yzhliu commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

masahi commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinthesun commented Mar 12, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masahi commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinthesun commented Mar 14, 2019

Uh oh!

masahi commented Mar 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eqy commented Mar 14, 2019

Uh oh!

kevinthesun commented Mar 20, 2019

Uh oh!

kevinthesun commented May 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinthesun commented May 19, 2019

Uh oh!

yzhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yzhliu commented May 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kevinthesun commented Nov 28, 2018 •

edited

Loading

masahi commented Mar 12, 2019 •

edited

Loading

yzhliu commented Mar 12, 2019 •

edited

Loading

masahi commented Mar 12, 2019 •

edited

Loading

masahi commented Mar 12, 2019 •

edited

Loading

masahi commented Mar 14, 2019 •

edited

Loading