Socket disconnect followed by segfault on macos/arm64 #191

Activity

1. ryanofsky commented on Jul 24, 2025

   

   ryanofsky

   on Jul 24, 2025

   Last edited by ryanofsky

   CollaboratorAuthor

   More actions

   From these logs it seems like there are abrupt socket disconnects followed by segfaults and other errors here.

   It seems like in the first example, the client is connecting and calling getbalance and then disconnecting before waiting for the getbalance response, and this causes the server to crash because it not handling the disconnect properly. The problem with abrupt disconnects causing crashes is known and should be fixed with bitcoin/bitcoin#32345. That PR should cause the server not crash in this case.

   It is not clear what is causing the client to disconnect, though. And the client errors don't seem to make that much sense.

   The first client error "timeout on transient error [...] Peer disconnected" only happens when the client is trying to connect to the server before it sends any RPC method request. So it does not seem consistent with the server log. Maybe it is actually from a different client than the one that send the getbalances request?

   The second client error "null Own<> dereference" seems more promising. Maybe that is the client which sent the getbalances request, and suddenly crashed. If so, getting a stack trace from that client could reveal what the problem is.

   The third client error "Could not connect to the server 127.0.0.1:8332" is actually an RPC error not an IPC error. But there is an IPC "Peer disconnected" error immediately before it, which I think would have to come from a different bitcoin-cli invocation? I don't see how a single bitcoin-cil invocation could result in both of these errors, since the first error should cause it to exit.

   I guess overall I'm confused about some of this behavior, and also don't understand why there are multiple different client errors and only a single server error. Unclear if the server crashed different times with the same error, or if all of these clients were just to connecting to the same server around the time it crashed. [EDIT: rereading the OP it sounds like the server just crashed multiple times and the clients showed varying error messages each time it crashed]

   I do think bitcoin/bitcoin#32345 should remove a lot of instability and noise here and make any remaining issues easier to resolve.
2. ryanofsky commented on Jul 24, 2025

   

   ryanofsky

   on Jul 24, 2025

   CollaboratorAuthor

   More actions

   Actually I think I'm misreading the server logs and there might not even a sudden disconnect causing the server to crash. The two lines beginning the log could be a clean disconnect. The getbalances call after could be coming from another client if -getinfo calls are being made in succession. Could help to have more logs, or a stack trace from the server to clarify.
3. pinheadmz commented on Jul 28, 2025

   

   pinheadmz

   on Jul 28, 2025

   More actions

   I was making many client calls to get the crashes, so that might explain why there seem to be multiple clients. I'll try to capture a stack.
4. pinheadmz commented on Jul 28, 2025

   

   pinheadmz

   on Jul 28, 2025

   More actions

   Got something after pummeling with RPC requests:

   Process 50097 stopped
   * thread #3, name = 'b-capnp-loop', stop reason = EXC_BAD_ACCESS (code=1, address=0x795848873540be60)
       frame #0: 0x0000000100c6c934 bitcoin-node`std::__1::__function::__value_func<void (bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>)>::operator bool[abi:ne180100](this=0x795848873540be48) const at function.h:463:75
      460        std::swap(__f_, __f.__f_);
      461    }
      462
   -> 463    _LIBCPP_HIDE_FROM_ABI explicit operator bool() const _NOEXCEPT { return __f_ != nullptr; }
      464
      465  #  ifndef _LIBCPP_HAS_NO_RTTI
      466    _LIBCPP_HIDE_FROM_ABI const std::type_info& target_type() const _NOEXCEPT {
   Target 0: (bitcoin-node) stopped.
   (lldb) bt
   * thread #3, name = 'b-capnp-loop', stop reason = EXC_BAD_ACCESS (code=1, address=0x795848873540be60)
     * frame #0: 0x0000000100c6c934 bitcoin-node`std::__1::__function::__value_func<void (bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>)>::operator bool[abi:ne180100](this=0x795848873540be48) const at function.h:463:75
       frame #1: 0x0000000100c6c8d0 bitcoin-node`std::__1::function<void (bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>)>::operator bool[abi:ne180100](this=0x795848873540be48) const at function.h:886:93
       frame #2: 0x0000000100c6cb48 bitcoin-node`mp::Logger& mp::operator<<<char const (&) [2]>(logger=0x000000016ff12130, value={\x83N\xc6\x01\x01\0\0\0}) at proxy-io.h:113:13
       frame #3: 0x0000000100c6c68c bitcoin-node`mp::EventLoop::log(this=0x795848873540bd30) at proxy-io.h:180:16
       frame #4: 0x0000000101141b50 bitcoin-node`void mp::serverDestroy<mp::ProxyServer<ipc::capnp::messages::Rpc>>(server=0x0000600007535220) at proxy-types.h:580:41
       frame #5: 0x0000000101141aa4 bitcoin-node`mp::ProxyServer<ipc::capnp::messages::Rpc>::~ProxyServer(this=0x0000600007535220, vtt=0x0000000101ec3d50) at rpc.capnp.proxy-types.c++:8:58
       frame #6: 0x0000000101141ce0 bitcoin-node`mp::ProxyServer<ipc::capnp::messages::Rpc>::~ProxyServer(this=0x0000600007535220) at rpc.capnp.proxy-types.c++:8:56
       frame #7: 0x0000000101141d44 bitcoin-node`mp::ProxyServer<ipc::capnp::messages::Rpc>::~ProxyServer(this=0x0000600007535220) at rpc.capnp.proxy-types.c++:8:56
       frame #8: 0x0000000100dd48ac bitcoin-node`kj::_::HeapDisposer<mp::ProxyServer<ipc::capnp::messages::Rpc>>::disposeImpl(this=0x0000000101eb7400, pointer=0x0000600007535220) const at memory.h:557:60
       frame #9: 0x00000001063812e0 libcapnp-rpc.1.1.0.dylib`capnp::LocalClient::~LocalClient() + 188
       frame #10: 0x000000010637c3a8 libcapnp-rpc.1.1.0.dylib`capnp::LocalClient::~LocalClient() + 20
       frame #11: 0x000000010637e924 libcapnp-rpc.1.1.0.dylib`kj::_::AttachmentPromiseNode<kj::_::Tuple<kj::Own<capnp::LocalClient, std::nullptr_t>, kj::Own<capnp::CallContextHook, std::nullptr_t>>>::~AttachmentPromiseNode() + 100
       frame #12: 0x0000000106512384 libkj-async.1.1.0.dylib`kj::_::ForkHubBase::fire() + 80
       frame #13: 0x000000010650fef4 libkj-async.1.1.0.dylib`kj::EventLoop::turn() + 128
       frame #14: 0x000000010651071c libkj-async.1.1.0.dylib`kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) + 464
       frame #15: 0x000000010172fe8c bitcoin-node`kj::Promise<unsigned long>::wait(this=0x000000016ff12c20, waitScope=0x0000600002300458, location=(fileName = "ipc/libmultiprocess/src/mp/proxy.cpp", function = "loop", lineNumber = 196, columnNumber = 35)) at async-inl.h:1357:3
       frame #16: 0x000000010172f810 bitcoin-node`mp::EventLoop::loop(this=0x0000000148804098) at proxy.cpp:196:68
       frame #17: 0x0000000100c5c190 bitcoin-node`ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(this=0x0000600001609408)::'lambda'()::operator()() const at protocol.cpp:91:21
       frame #18: 0x0000000100c5c008 bitcoin-node`decltype(std::declval<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>()()) std::__1::__invoke[abi:ne180100]<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__f=0x0000600001609408) at invoke.h:344:25
       frame #19: 0x0000000100c5bfac bitcoin-node`void std::__1::__thread_execute[abi:ne180100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__t=size=2, (null)=__tuple_indices<> @ 0x000000016ff12f77) at thread.h:199:3
       frame #20: 0x0000000100c5b92c bitcoin-node`void* std::__1::__thread_proxy[abi:ne180100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>(__vp=0x0000600001609400) at thread.h:208:3
       frame #21: 0x00000001895a2f94 libsystem_pthread.dylib`_pthread_start + 136
   
5. pinheadmz commented on Jul 28, 2025

   

   pinheadmz

   on Jul 28, 2025

   More actions

   To be totally clear, I pummeled using watch ... -getinfo from 4 separate tmux panes, while sync was around 99%. I let that run for several minutes, then cancelled all the watch processes, took a breath, and executed one single -getinfo request -- for some reason that produced the same crash twice in lldb.

   👍React with 👍1Reacted by Ryan Ofsky
6. ryanofsky commented on Jul 28, 2025

   

   ryanofsky

   on Jul 28, 2025

   CollaboratorAuthor

   More actions

   Thanks! The bitcoin-node stack trace is consistent with bugs fixed in #160 and bitcoin/bitcoin#32345. An annotated version of the stack trace shown below shows this is probably happening because the server.m_context.connection pointer is null, which would happen if the connection is closed suddenly before #160. If you merge into bitcoin/bitcoin#32345 into your branch, the server crash should stop happening.

   It's still not clear what clients may be doing to trigger this crash though. If one of them is segfaulting before the server crash, that might explain this, and it could help to see the client stack trace. Otherwise it's not really clear. bitcoin/bitcoin#32345 might fix the client problems or might not, would have to test to find out.

   Annotated stack trace from #191 (comment):

       frame #2: 0x0000000100c6cb48 bitcoin-node`mp::Logger& mp::operator<<<char const (&) [2]>(logger=0x000000016ff12130, value={\x83N\xc6\x01\x01\0\0\0}) at proxy-io.h:113:13
   
      110      template <typename T>
      111      friend Logger& operator<<(Logger& logger, T&& value)
      112      {
   >  113          if (logger.m_fn) logger.m_buffer << std::forward<T>(value);
      114          return logger;
      115      }
       
       frame #3: 0x0000000100c6c68c bitcoin-node`mp::EventLoop::log(this=0x795848873540bd30) at proxy-io.h:180:16
   
      177      Logger log()
      178      {
      179          Logger logger(false, m_log_fn);
   >  180          logger << "{" << LongThreadName(m_exe_name) << "} ";
      181          return logger;
      182      }
       
       frame #4: 0x0000000101141b50 bitcoin-node`void mp::serverDestroy<mp::ProxyServer<ipc::capnp::messages::Rpc>>(server=0x0000600007535220) at proxy-types.h:580:41
   
      577  template <typename Server>
      578  void serverDestroy(Server& server)
      579  {
   >  580      server.m_context.connection->m_loop.log() << "IPC server destroy " << typeid(server).name();
      581  }
7. pinheadmz commented on Aug 7, 2025

   

   pinheadmz

   on Aug 7, 2025

   More actions

   Couldn't reproduce the crash bitcoin/bitcoin#32345 so I think you can close this since the fixes seem to be merged already in libmultiprocess
8. fanquake

   closed this as completedon Aug 21, 2025
9. bitcoin-core

   locked and limited conversation to collaborators on Aug 21, 2026