Skip to content

Cascading failure on bots when interrupted when performing git operations #34595

Closed
Closed
Cascading failure on bots when interrupted when performing git operations#34595
Labels
A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Description

@alexcrichton
alexcrichton
opened on Jul 1, 2016

This is basically a best guess as to what's happening on the bots, but from time to time it appears that the build directory gets entirely corrupted with error messages like:

extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rust-std-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rustc-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-22\cargo-nightly-x86_64-pc-windows-msvc.tar.gz
Synchronizing submodule url for 'src/compiler-rt'
Synchronizing submodule url for 'src/jemalloc'
Synchronizing submodule url for 'src/liblibc'
Synchronizing submodule url for 'src/llvm'
Synchronizing submodule url for 'src/rt/hoedown'
Synchronizing submodule url for 'src/rust-installer'
fatal: Needed a single revision


command did not execute successfully: "git" "submodule" "update"
expected success, got: exit code: 1


Makefile:23: recipe for target 'all' failed
>> rustjob: found 0 remaining processes
Unable to find current revision in submodule path 'src/llvm'
make: *** [all] Error 1

This happened on a MSVC rustbuild bot, but I believe I've seen this before on basically all bots (but I feel the windows bots are affected more often).

My best guess as to what's happening here is:

  • A build is running, and it's updating the LLVM submodule
  • The build is canceled for any number of a few reasons
  • The LLVM update process is interrupted, leaving the git directory in a corrupt state
  • All future builds attempt to use this corrupt directory and fail

I... don't actually know what the best solution here is. May as well track it though!

Activity

  1. alexcrichton
    added
    A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)
    on Jul 1, 2016

  2. alexcrichton commented on Jul 1, 2016

    @alexcrichton
    alexcrichton
    on Jul 1, 2016
    MemberAuthor

    FWIW the fix now is to log into the bots and delete the obj directory and restart everything.

  3. alexcrichton commented on Jul 1, 2016

    @alexcrichton
    alexcrichton
    on Jul 1, 2016
    MemberAuthor

    Correction, fix is to remove src/llvm, not obj ...

  4. alexcrichton commented on Jul 20, 2016

    @alexcrichton
    alexcrichton
    on Jul 20, 2016
    MemberAuthor

    This unfortunately has happened quite a bit recently. I believe it's isolated to rustbuild currently because the ./configure step (which is where the makefiles deal with submodules) is an uninterruptible step in buildbot. That at least means I think the fix only needs to be in rustbuild!

    The only fix here that I can think of is that we detect this failure, then detect where it failed, blow away the entire subdirectory if it exists, then try the whole operation again. That way it can automatically recover from a corrupt directory.

    Would be a great way to jump into rustbuild if anyone's interested!

  5. alexcrichton
    added
    T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
    on Jul 20, 2016

  6. alexcrichton commented on Dec 30, 2016

    @alexcrichton
    alexcrichton
    on Dec 30, 2016
    MemberAuthor

    Buildbots are almost gone now, so closing.

  7. alexcrichton
    closed this as completedon Dec 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions