Update all ci- containers to reflect main

@jroesch

This is a tracking issue for the process of updating the TVM ci- containers to reflect the following PRs:

It will also unblock PRs:

[ETHOSN] Removed support for 20.08 version of the driver stack. #7858
Bumped Ubuntu version to 18.04 for ci_gpu #7970 (it's possible we could just include this PR in this update)

Steps:

Let's use this tracking issue to record challenges we face in updating the drivers.

@jroesch can you please note down where you were with this?

cc @tqchen @u99127 @d-smirnov @leandron @tristan-arm

So last time I tried to update the images but ran into issues where tests no longer passed under updated GPU and CPU images (see docker pull tlcpack/ci-gpu:v0.73-t3).

I had chased down the image building, there are a few important things.

First disable caching in order to make sure you pull the freshest apt as if you have built before you can have a stale layer which pulled an apt-registry which is no longer valid.

Second there were some updates to the drivers which caused breakage, specifically this test no longer works because it incorrectly believes that CUDA is on (even though there is no GPU):

https://github.com/apache/tvm/blob/main/tests/cpp/build_module_test.cc#L84

These needs to be patched to actually check for the GPU's existence.

There were also changes intended to fix Rust CI which should included in master today allowing us to turn Rust CI back on.

@tkonolige

@tkonolige and i will attempt to update to 18.04 (e.g. include #7970) with these updates.

#8031 contains the updated docker images

@tkonolige

@tkonolige is kindly rebuilding the containers. @tkonolige , can you document the main hash used following the steps above?

Testing: https://ci.tlcpack.ai/job/tvm/job/ci-docker-staging/87/

Built from: 55471632d97bb9d0d4535bd977ca4fb7e7cdcc28

retesting: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/88/pipeline

Mark zephyr install world-writable in docker image to unblock apache#…

Mark zephyr install world-writable in docker image to unblock #7995. (#…

we accidentally tested the wrong thing; so re-re-testing here: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/89/pipeline/6

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/90/pipeline

seems like the last run was unexpectedly unable to access to the gpu in "Frontend : GPU" phase. retrying with printing nvidia-smi before running those tests: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/91/pipeline/

@tkonolige

apologies for the light updates here. we determined that the "Frontend : GPU" tests get into a state where either the GPU hardware is inaccessible after a while or TVM's existence check is wrong. Since we didn't change the CUDA version used here--we just updated to 18.04--the theory is that there is some interoperability problem between CUDA running in the containers (at 10.0) and the CUDA driver loaded on the docker host side (either 10.2 or 11.0, depending which CI node you hit).

@tkonolige and I have spent the last couple days running on a test TVM CI cluster using the same AMI (which has only CUDA 11.0). With CUDA 10.0 (ci-gpu) and 11.0 (host), we ran into another similar-looking bug during the GPU unit tests:

[ RUN      ] BuildModule.Heterogeneous
[22:11:58] /workspace/src/target/opt/build_cuda_on.cc:89: Warning: cannot detect compute capability from your device, fall back to compute_30.
unknown file: Failure
C++ exception with description "[22:11:58] /workspace/src/runtime/cuda/cuda_device_api.cc:117: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (e == cudaSuccess || e == cudaErrorCudartUnloading) is false: CUDA: no CUDA-capable device is detected

We then upgraded ci-gpu to use CUDA 11.0, and this test seemed to pass all the way to the end of the GPU integration tests, modulo a tolerance issue:

tests/python/contrib/test_cublas.py::test_batch_matmul FAILED
// ...
>       np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, verbose=True)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-05, atol=1e-07
E       
E       Mismatched elements: 2875175 / 3866624 (74.4%)
E       Max absolute difference: 0.00541687
E       Max relative difference: 0.00015383
E        x: array([[[29.647408, 31.88966 , 33.90233 , ..., 34.673954, 32.908764,
E                31.219051],
E               [30.993076, 30.78019 , 33.67124 , ..., 36.1395  , 29.176218,...
E        y: array([[[29.646427, 31.889557, 33.900528, ..., 34.673126, 32.90791 ,
E                31.21726 ],
E               [30.991737, 30.780437, 33.67001 , ..., 36.139397, 29.174744,...

we'll try and push this CUDA 11.0 ci-gpu container through the test CI cluster to see how far we can get. feel free to comment if there are concerns updating to CUDA 11.0.