bitcoin-cli uncaught kj::ExceptionImpl with #32297

@pinheadmz

Originally posted by @pinheadmz in #182

Not sure if my observation belongs here or in a new issue. Testing bitcoin/bitcoin#32297 I tried to use socat to inspect the IPC request:

socat -v UNIX-LISTEN:/Users/matthewzipkin/.bitcoin/regtest/node.sock STDOUT

and then make a bitcoin-cli request even though bitcoin-node was not running:

bccli -regtest getblockcount

error:
libc++abi: terminating due to uncaught exception of type kj::ExceptionImpl: /opt/homebrew/include/kj/memory.h:258: failed: expected ptr != nullptr; null Own<> dereference
stack: 1070007cb 10489843b 10494a81f 10494a74b 104951f4f 104951def 104951d93 104951d33 104951c73 104951c17 10494f59b 1048acccb 1048ac9d7 104ddc727 104ddc077 10488ab97 10488aa0f 10488a8e7 104889977 1a01b6f93 1a01b1d33
Abort trap: 6
Stepping through with lldb I found the error in ConnectStream():

libmultiprocess/include/mp/proxy-io.h

Lines 559 to 580 in 0198397

//! Given stream file descriptor, make a new ProxyClient object to send requests

//! over the stream. Also create a new Connection object embedded in the

//! client that is freed when the client is closed.

template <typename InitInterface>

std::unique_ptr<ProxyClient<InitInterface>> ConnectStream(EventLoop& loop, int fd)

{

typename InitInterface::Client init_client(nullptr);

std::unique_ptr<Connection> connection;

loop.sync([&] {

auto stream =

loop.m_io_context.lowLevelProvider->wrapSocketFd(fd, kj::LowLevelAsyncIoProvider::TAKE_OWNERSHIP);

connection = std::make_unique<Connection>(loop, kj::mv(stream));

init_client = connection->m_rpc_system->bootstrap(ServerVatId().vat_id).castAs<InitInterface>();

Connection* connection_ptr = connection.get();

connection->onDisconnect([&loop, connection_ptr] {

loop.log() << "IPC client: unexpected network disconnect.";

delete connection_ptr;

});

});

return std::make_unique<ProxyClient<InitInterface>>(

kj::mv(init_client), connection.release(), /* destroy_connection= */ true);

}

What I believe happens is, we get a file descriptor way earlier in the stack (in bitcoin: src/ipc/interfaces.cpp::connectAddress() but by the time we get to ConnectStream() the listener has disconnected (or, just isn't bitcoin) and then things start going wrong in ProxyClientBase()

It could be useful to see the backtrace from lldb, but depending on where the exception is being thrown, I think #32345 could help with this by turning the kj::Exception into an ipc::Exception that could be more easily caught in bitcoin-cli. It would make sense for the CallIPC method in bitcoin/bitcoin#32297 to catch this exception and treat it the same as the connection failing.

bt:

(lldb) bt
* thread #2, name = 'b-capnp-loop', stop reason = signal SIGABRT
  * frame #0: 0x00000001a017ea60 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001a01b6c20 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001a00c3a30 libsystem_c.dylib`abort + 180
    frame #3: 0x00000001a016dd08 libc++abi.dylib`abort_message + 132
    frame #4: 0x00000001a015dfa4 libc++abi.dylib`demangling_terminate_handler() + 320
    frame #5: 0x000000019fdfc1e0 libobjc.A.dylib`_objc_terminate() + 160
    frame #6: 0x00000001a016d0cc libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #7: 0x00000001a0170348 libc++abi.dylib`__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) + 88
    frame #8: 0x00000001a017028c libc++abi.dylib`__cxa_throw + 308
    frame #9: 0x00000001028262a0 libkj.1.1.0.dylib`kj::ExceptionCallback::RootExceptionCallback::onFatalException(kj::Exception&&) + 60
    frame #10: 0x0000000102825184 libkj.1.1.0.dylib`kj::throwFatalException(kj::Exception&&, unsigned int) + 60
    frame #11: 0x00000001028214cc libkj.1.1.0.dylib`kj::_::Debug::Fault::fatal() + 148
    frame #12: 0x00000001028207cc libkj.1.1.0.dylib`kj::_::inlineRequireFailure(char const*, int, char const*, char const*, char const*) + 108
    frame #13: 0x00000001000b843c bitcoin-cli`kj::Own<capnp::ClientHook, std::nullptr_t>::operator->(this=0x0000600002120050) at memory.h:258:28
    frame #14: 0x000000010016a820 bitcoin-cli`capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> capnp::Capability::Client::newCall<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults>(this=0x0000600002120050, interfaceId=9815814193794562661, methodId=0, sizeHint=Maybe<capnp::MessageSize> @ 0x000000016fe866b0, hints=(noPromisePipelining = false, onlyPromisePipeline = false)) at capability.h:1113:19
    frame #15: 0x000000010016a74c bitcoin-cli`ipc::capnp::messages::Init::Client::constructRequest(this=0x0000600002120048, sizeHint=Maybe<capnp::MessageSize> @ 0x000000016fe86880) at init.capnp.c++:500:10
    frame #16: 0x0000000100171f50 bitcoin-cli`void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(this=0x000000016fdfd358)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()::operator()() const at proxy-types.h:620:24
    frame #17: 0x0000000100171df0 bitcoin-cli`decltype(std::declval<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>>()()) std::__1::__invoke[abi:ne180100]<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()&>(__f=0x000000016fdfd358) at invoke.h:344:25
    frame #18: 0x0000000100171d94 bitcoin-cli`std::__1::__invoke_of<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()&>::type std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>::operator()[abi:ne180100]<>(this=0x000000016fdfd2a0) const at reference_wrapper.h:64:12
    frame #19: 0x0000000100171d34 bitcoin-cli`decltype(std::declval<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>>()()) std::__1::__invoke[abi:ne180100]<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>&>(__f=0x000000016fdfd2a0) at invoke.h:344:25
    frame #20: 0x0000000100171c74 bitcoin-cli`void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ne180100]<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>&>(__args=0x000000016fdfd2a0) at invoke.h:419:5
    frame #21: 0x0000000100171c18 bitcoin-cli`std::__1::__function::__alloc_func<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>, std::__1::allocator<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>>, void ()>::operator()[abi:ne180100](this=0x000000016fdfd2a0) at function.h:169:12
    frame #22: 0x000000010016f59c bitcoin-cli`std::__1::__function::__func<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>, std::__1::allocator<std::__1::reference_wrapper<void mp::clientInvoke<mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>>(mp::ProxyClientBase<ipc::capnp::messages::Init, interfaces::Init>&, capnp::Request<ipc::capnp::messages::Init::ConstructParams, ipc::capnp::messages::Init::ConstructResults> (ipc::capnp::messages::Init::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::init_fields::ThreadMap, 19>>&&)::'lambda'()>>, void ()>::operator()(this=0x000000016fdfd298) at function.h:311:10
    frame #23: 0x00000001000ccccc bitcoin-cli`std::__1::__function::__value_func<void ()>::operator()[abi:ne180100](this=0x000000016fdfd298) const at function.h:428:12
    frame #24: 0x00000001000cc9d8 bitcoin-cli`std::__1::function<void ()>::operator()(this=0x000000016fdfd298) const at function.h:981:10
    frame #25: 0x00000001005fc728 bitcoin-cli`void mp::Unlock<std::__1::unique_lock<std::__1::mutex>, std::__1::function<void ()> const&>(lock=0x000000016fe86e88, callback=0x000000016fdfd298) at util.h:146:5
    frame #26: 0x00000001005fc078 bitcoin-cli`mp::EventLoop::loop(this=0x000000012be04c88) at proxy.cpp:200:13
    frame #27: 0x00000001000aab98 bitcoin-cli`ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(this=0x0000600003425568)::'lambda'()::operator()() const at protocol.cpp:91:21
    frame #28: 0x00000001000aaa10 bitcoin-cli`decltype(std::declval<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>()()) std::__1::__invoke[abi:ne180100]<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__f=0x0000600003425568) at invoke.h:344:25
    frame #29: 0x00000001000aa8e8 bitcoin-cli`void std::__1::__thread_execute[abi:ne180100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(__t=size=2, (null)=__tuple_indices<> @ 0x000000016fe86f77) at thread.h:199:3
    frame #30: 0x00000001000a9978 bitcoin-cli`void* std::__1::__thread_proxy[abi:ne180100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>(__vp=0x0000600003425560) at thread.h:208:3
    frame #31: 0x00000001a01b6f94 libsystem_pthread.dylib`_pthread_start + 136

Thanks, so from the stack trace it doesn't seem like any changes to bitcoin/bitcoin#32297 need to be made to fix this because the failing IPC call is happening when the Init object is being constructed in ConnectStream calling down to the ProxyClientBase<Init> constructor.

So bitcoin/bitcoin#32345 might be sufficient by itself to fix this, or if not the clientInvoke function needs to be improved more to handle exceptions coming from this place.

Looking at stack trace more, it does seem like a new fix will be required to prevent this crash. The line that is throwing is:

libmultiprocess/include/mp/proxy-types.h

Line 620 in 0198397

auto request = (proxy_client.m_client.*get_request)(nullptr);

So probably need to add a try / catch around this line and turn it into an exception similar to 56fff76 from #160

For what it's worth, I rebased bitcoin/bitcoin#32297 on top of bitcoin/bitcoin#32345 and the edge case of ipcconnecting bitcoin-cli to a non-bitcoin socket failed gracefully:

$ bccli -regtest getblockcount
error: timeout on transient error: IPC client method called after disconnect.

Probably bitcoin-node is not running or not listening on a unix socket. Can be started with:

    bitcoin-node -chain=regtest -ipcbind=unix

Thanks for testing and that's interesting. I'm a little surprised bitcoin/bitcoin#32345 would fix this issue based on where the exception seemed to be coming from in the stack trace.

And either way, I would like to add unit test to cover the case of client connecting to a socket that just disconnects.

bitcoin/bitcoin#32345 is merged but it still seems like a good idea to add a test to covers connecting to a socket that just disconnects without replying, so will keep this open

WIP: Add test coverage for ConnectStream

@ryanofsky

I've been working on this issue and have pushed a commit that adds test coverage: xyzconstant@bb60535.

I've added tests for four scenarios:

Attempting to use an fd associated with a server that has already been disconnected results in a crash with: "IPC client method called after disconnect".
Attempting to use an fd associated with a server that disconnects immediately after the first recv(), the outcome will either be crashing with "IPC client method called after disconnect" or "IPC client method call interrupted by disconnect." This behavior seems to depend on whether the recv() call successfully reads the entirety of what is likely Capnp’s bootstrap request or only a portion of it.
Attempting to use an fd associated with a server that disconnects after a second recv(), the process crashes with "IPC client method call interrupted by disconnect."
Attempting to use an fd associated with a server that indefinitely calls recv() in a while loop results in the test getting stuck at this line:

libmultiprocess/include/mp/proxy-types.h

Line 754 in 8412fcd

thread_context.waiter->wait(lock, [&done]() { return done; });

It’s important to note that all of these cases call FooInterface::initThreadMap() to invoke clientInvoke(), as ConnectStream() on its own won't trigger any libmultiprocess request if there's not a construct() method available in the Init interface.

I'll look into the 4th scenario further and see if there's an elegant solution that covers all of these scenarios (if needed).

I would appreciate your feedback on this, @ryanofsky.

I would appreciate your feedback on this, @ryanofsky.

Thanks for the tests! Followed up in #298

WIP: Add test coverage for ConnectStream

Merge #298: Fix error handling when creating clients (`mp::ConnectStr…

Navigation Menu

Repository navigation

bitcoin-cli uncaught kj::ExceptionImpl with #32297 #183

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	//! Given stream file descriptor, make a new ProxyClient object to send requests
	//! over the stream. Also create a new Connection object embedded in the
	//! client that is freed when the client is closed.
	template <typename InitInterface>
	std::unique_ptr<ProxyClient<InitInterface>> ConnectStream(EventLoop& loop, int fd)
	{
	typename InitInterface::Client init_client(nullptr);
	std::unique_ptr<Connection> connection;
	loop.sync([&] {
	auto stream =
	loop.m_io_context.lowLevelProvider->wrapSocketFd(fd, kj::LowLevelAsyncIoProvider::TAKE_OWNERSHIP);
	connection = std::make_unique<Connection>(loop, kj::mv(stream));
	init_client = connection->m_rpc_system->bootstrap(ServerVatId().vat_id).castAs<InitInterface>();
	Connection* connection_ptr = connection.get();
	connection->onDisconnect([&loop, connection_ptr] {
	loop.log() << "IPC client: unexpected network disconnect.";
	delete connection_ptr;
	});
	});
	return std::make_unique<ProxyClient<InitInterface>>(
	kj::mv(init_client), connection.release(), /* destroy_connection= */ true);
	}

bitcoin-cli uncaught kj::ExceptionImpl with #32297 #183

Description

Activity

ryanofsky commented on Jun 16, 2025

pinheadmz commented on Jun 16, 2025

ryanofsky commented on Jun 16, 2025

ryanofsky commented on Jun 16, 2025

pinheadmz commented on Jun 17, 2025

ryanofsky commented on Jun 17, 2025

ryanofsky commented on Aug 23, 2025

xyzconstant commented on Jun 24, 2026

ryanofsky commented on Jul 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions