Skip to content

Implement RowBinaryWithNamesAndTypes support for schema evolution #782

Description

@BentsiLeviav

Context

Kafka Connect currently writes to ClickHouse using a raw binary format that assumes the connector-side schema and the server-side table schema are fully in sync.

This can fail during schema evolution in replicated ClickHouse clusters:

  1. A customer adds a column to a replicated table, usually with a default value.
  2. The DDL takes a few seconds to propagate across all replicas.
  3. Kafka Connect reads/describes the schema from one replica.
  4. Kafka Connect writes a batch to another replica that may still have the previous schema.
  5. Because the connector sends raw binary bytes without column names/types, the receiving replica interprets the payload according to its local schema and the insert may fail.

Problem

The connector currently relies on positional binary encoding. During schema propagation lag, this creates ambiguity between:

  • the schema used by Kafka Connect to serialize the batch,
  • and the schema known by the ClickHouse replica receiving the insert.

Proposal

Add support for writing inserts using ClickHouse RowBinaryWithNamesAndTypes.

With this format, the connector sends column names and types in the insert header. This allows ClickHouse to interpret the incoming rows according to the schema declared by the client, rather than relying only on the receiving replica’s current table schema.

Expected benefits:

  • Better handling when a new column exists in ClickHouse but the connector is still sending the previous schema.
  • Safer behavior when columns are added with defaults.
  • Clearer failure mode when the connector sends a column that does not yet exist on the target replica.
  • Improved ability to detect schema mismatch and retry only when needed.
  • Reduced risk of misinterpreting raw binary payloads during schema evolution windows.

Scope

Implement RowBinaryWithNamesAndTypes as an insert format option for Kafka Connect.

Recommended behavior:

  • Prefer RowBinaryWithNamesAndTypes for schema-aware inserts.
  • Preserve the existing raw binary behavior unless/until we decide to change the default.
  • Add configuration to enable the new format, or introduce it behind an existing insert-format abstraction if one already exists.
  • Ensure the connector handles schema mismatch errors clearly and retriably where appropriate.

As a reference, please see our Flink connector (which already supports this format).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions