Context
Kafka Connect currently writes to ClickHouse using a raw binary format that assumes the connector-side schema and the server-side table schema are fully in sync.
This can fail during schema evolution in replicated ClickHouse clusters:
- A customer adds a column to a replicated table, usually with a default value.
- The DDL takes a few seconds to propagate across all replicas.
- Kafka Connect reads/describes the schema from one replica.
- Kafka Connect writes a batch to another replica that may still have the previous schema.
- Because the connector sends raw binary bytes without column names/types, the receiving replica interprets the payload according to its local schema and the insert may fail.
Problem
The connector currently relies on positional binary encoding. During schema propagation lag, this creates ambiguity between:
- the schema used by Kafka Connect to serialize the batch,
- and the schema known by the ClickHouse replica receiving the insert.
Proposal
Add support for writing inserts using ClickHouse RowBinaryWithNamesAndTypes.
With this format, the connector sends column names and types in the insert header. This allows ClickHouse to interpret the incoming rows according to the schema declared by the client, rather than relying only on the receiving replica’s current table schema.
Expected benefits:
- Better handling when a new column exists in ClickHouse but the connector is still sending the previous schema.
- Safer behavior when columns are added with defaults.
- Clearer failure mode when the connector sends a column that does not yet exist on the target replica.
- Improved ability to detect schema mismatch and retry only when needed.
- Reduced risk of misinterpreting raw binary payloads during schema evolution windows.
Scope
Implement RowBinaryWithNamesAndTypes as an insert format option for Kafka Connect.
Recommended behavior:
- Prefer
RowBinaryWithNamesAndTypes for schema-aware inserts.
- Preserve the existing raw binary behavior unless/until we decide to change the default.
- Add configuration to enable the new format, or introduce it behind an existing insert-format abstraction if one already exists.
- Ensure the connector handles schema mismatch errors clearly and retriably where appropriate.
As a reference, please see our Flink connector (which already supports this format).
Context
Kafka Connect currently writes to ClickHouse using a raw binary format that assumes the connector-side schema and the server-side table schema are fully in sync.
This can fail during schema evolution in replicated ClickHouse clusters:
Problem
The connector currently relies on positional binary encoding. During schema propagation lag, this creates ambiguity between:
Proposal
Add support for writing inserts using ClickHouse
RowBinaryWithNamesAndTypes.With this format, the connector sends column names and types in the insert header. This allows ClickHouse to interpret the incoming rows according to the schema declared by the client, rather than relying only on the receiving replica’s current table schema.
Expected benefits:
Scope
Implement
RowBinaryWithNamesAndTypesas an insert format option for Kafka Connect.Recommended behavior:
RowBinaryWithNamesAndTypesfor schema-aware inserts.As a reference, please see our Flink connector (which already supports this format).