You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Sink] Configurable rank-local routing and processing sink #130
Add one native RecordSink between the completed D2H RecordEnvelope stream and the existing destination sinks. It matches each record's producer output_id to configured routes, shares one immutable input across the matching routes, runs each route as a forward, stateless, or stateful pipeline, and submits each result to that route's destination sink.
RecordDescriptor (native) carries output_id; bind_hook reports output name to id so routes can be configured by name.
New routing sink: route table, shared immutable input, forward executor inline on the record worker, stateless worker pool, stateful owner workers over a fixed K x N state region with same-key ordering.
Processors registered by name at build time (DMI_REGISTER_*_PROCESSOR); stateful processors also register key, key range, and slot spec functions.
Output records validated against their registered layout before submission.
Destinations are existing RecordSink objects, leased by the routing sink; one RecordSchema covers input and output layouts and is validated against the ClickHouse stage at construction.
Unmatched records forward unchanged to a default destination. Full route queues block the record worker, matching the ClickHouse insert queue today.
Checked flush drains route queues then flushes each destination; first failure latched and surfaced.
Counters: per-route queue depth, backlog, processed, emitted, failed; per-destination records and bytes; state slot validity.
Remote or storage-side workflows, cross-rank aggregation, runtime plugin ABI, route chaining, GPU preprocessing changes, D2H scheduling, and the concrete signal processors (Q/K spectral, expert health, pathway, TwoNN), which plug in as registered processors.
Verification
Raw-only, processed-only, and raw-plus-processed fan-out; shared input lifetime; stateless concurrency and same-key serialization; overload blocking; flush, shutdown, and failure propagation; parity of a raw-only route with the direct ClickHouse sink; end-to-end through ClickHouse and the capture sink.
Activity