CASSANDRA-21417: Fix non-printable characters in Gossiper log for TOKENS by Suhel0328 · Pull Request #5124 · apache/cassandra

Suhel0328 · 2026-09-07T06:09:26Z

toString() on EndpointState was printing the raw TOKENS bytes directly. TOKENS is a serialized token collection stored as an ISO-8859-1 string just to preserve the raw bytes, so printing it as text dumps non-printable control characters into gossip debug logs.

Now TOKENS renders as Value(<N tokens>,<version>), same shape as every other application state's Value(<value>,<version>). If it can't be deserialized for any reason, including a corrupt length prefix that could otherwise blow up with an OutOfMemoryError, it falls back to Value(<N undecodable bytes>,<version>) instead of printing raw bytes.

Added two tests in EndpointStateTest.java: testToStringDoesNotLeakRawTokenBytes and testToStringHandlesUndecodableTokensValue.

ant test -Dtest.name=EndpointStateTest passes, 4/4.

CASSANDRA-21417

AshenScribe · 2026-09-08T04:14:49Z

+     * never meant to be printed as text - doing so produces unreadable, control-character-laden output
+     * (see CASSANDRA-21417). All other states are rendered normally.
+     */
+    private static String appStateMapToString(Map<ApplicationState, VersionedValue> applicationState)


how about renaming to formatAppStateMapForLogging for a showing an explicit purpose?

AshenScribe · 2026-09-08T04:23:41Z

+            else
+                sb.append(entry.getValue());
+        }
+        return sb.append('}').toString();


wondering if we can use Streams API here.

AshenScribe · 2026-09-08T04:38:13Z

+     * TokenSerializer attempt a multi-gigabyte array allocation (see CASSANDRA-21417 discussion).
+     */
+    @Test
+    public void testToStringHandlesUndecodableTokensValue()


is this function's name self-explanatory? does it need the above comment?

AshenScribe · 2026-09-08T04:40:25Z

+     * the raw bytes losslessly, and toString() must not print that raw data (see CASSANDRA-21417).
+     */
+    @Test
+    public void testToStringDoesNotLeakRawTokenBytes()


how about renaming to testToStringSummarizesTokensInsteadOfPrintingRawBinaryData, is it self-containing?

AshenScribe · 2026-09-08T04:42:28Z

@Suhel0328 can you put a picture of the logs verifying the change on ticket?

AshenScribe · 2026-09-08T04:43:31Z

@Suhel0328 make the status of ticket to "Patch Available" by "Open Issue" -> "Patch Available".

AshenScribe · 2026-09-08T04:55:03Z

+        assertTrue(rendered.contains("RELEASE_VERSION=Value("));
+    }
+
+    /**


The current tests via EndpointState.toString() are correct, but I'd recommend extracting the app state formatting logic into a package-private method with dedicated tests.

we are testing toString() which includes heartbeat state, alive status, etc.

if toString() formatting changes for heartbeat or other fields, these tests might break even though the app state logic is fine

test edge cases for appStateMapToString alone

…d unit tests

Suhel0328 · 2026-09-08T07:08:23Z

Here is the screenshot of logs after the change:

Suhel0328 · 2026-09-08T07:09:48Z

Hi @AshenScribe, done changes as per the suggestion

AshenScribe · 2026-09-08T07:59:23Z

+            VersionedValue value = entry.getValue();
+            try
+            {
+                int numTokens = TokenSerializer.deserialize(DatabaseDescriptor.getPartitioner(),


DatabaseDescriptor.getPartitioner() is called for every iteration, extract it into a variable.

Extracted partitioner into a local variable before the stream loop.

AshenScribe · 2026-09-08T08:02:27Z

+            if (entry.getKey() != ApplicationState.TOKENS)
+                return entry.getKey() + "=" + entry.getValue();
+
+            VersionedValue value = entry.getValue();


can you make VersionedValue value final?

Added final

AshenScribe · 2026-09-08T08:02:56Z

+                int numTokens = TokenSerializer.deserialize(DatabaseDescriptor.getPartitioner(),
+                                                              new DataInputStream(new ByteArrayInputStream(value.toBytes())))
+                                                .size();
+                return entry.getKey() + "=Value(<" + numTokens + " tokens>," + value.version + ')';


numTokens is not used anywhere except once, please inline it.

I would prefer keeping numTokens as a separate variable here, it clearly documents what deserialize().size() returns, and inlining a multi-line deserialization call directly inside string concatenation gets pretty dense to read.

AshenScribe · 2026-09-08T08:46:03Z

the patch looks good to me on code. but the image is not a representative of this patch as opposed to the image on jira showing the issue.

To properly verify the fix, could you please provide a log snippet that includes the TOKENS state, showing it rendered as TOKENS=Value(,X) instead of the raw bytes?

Suhel0328 · 2026-09-08T09:39:10Z

the patch looks good to me on code. but the image is not a representative of this patch as opposed to the image on jira showing the issue.

To properly verify the fix, could you please provide a log snippet that includes the TOKENS state, showing it rendered as TOKENS=Value(,X) instead of the raw bytes?

The previous screenshot did already show it correctly (last line, TOKENS=Value(<16 tokens>,35) at Gossiper.java:2406), just easy to miss in a wide capture. Here's a clearer TOKENS rendered correctly, repeated across multiple lines:

DEBUG [GossipStage:1] 2026-09-08T09:19:54,946 Gossiper.java:2406 - Updated epstates for /127.0.0.1:38000: EndpointState: HeartBeatState = HeartBeat: generation = 1788859185, version = 0, AppStateMap = {SCHEMA=Value(00000000-0000-0000-0000-000000000002,17), DC=Value(datacenter1,0), RACK=Value(rack1,0), RELEASE_VERSION=Value(7.0.0-SNAPSHOT,0), RPC_ADDRESS=Value(127.0.0.1,0), NET_VERSION=Value(14,5), HOST_ID=Value(6d194555-f6eb-41d0-c000-000000000002,0), TOKENS=Value(<16 tokens>,0), INTERNAL_ADDRESS_AND_PORT=Value(127.0.0.1:38000,0), NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:40042,0), STATUS_WITH_PORT=Value(BOOT,4821400658802826711,0), SSTABLE_VERSIONS=Value(big-pa,7)}, isAlive = true
DEBUG [GossipStage:1] 2026-09-08T09:19:56,513 Gossiper.java:2406 - Updated epstates for /127.0.0.1:38000: EndpointState: HeartBeatState = HeartBeat: generation = 1788859185, version = 0, AppStateMap = {SCHEMA=Value(00000000-0000-0000-0000-000000000002,21), DC=Value(datacenter1,22), RACK=Value(rack1,23), RELEASE_VERSION=Value(7.0.0-SNAPSHOT,24), RPC_ADDRESS=Value(127.0.0.1,0), NET_VERSION=Value(14,5), HOST_ID=Value(6d194555-f6eb-41d0-c000-000000000002,26), TOKENS=Value(<16 tokens>,27), INTERNAL_ADDRESS_AND_PORT=Value(127.0.0.1:38000,28), NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:40042,29), STATUS_WITH_PORT=Value(NORMAL,-9180886933987773365,30), SSTABLE_VERSIONS=Value(big-pa,7)}, isAlive = true
DEBUG [main] 2026-09-08T09:19:51,511 Gossiper.java:2079 - Gossip looks settled. {localhost/127.0.0.1:38000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859185, version = 13, AppStateMap = {NET_VERSION=Value(14,5), SSTABLE_VERSIONS=Value(big-pa,7)}, isAlive = true, /127.0.0.1:37000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859012, version = 218, AppStateMap = {LOAD=Value(74541.0,172), SCHEMA=Value(00000000-0000-0000-0000-000000000002,30), DC=Value(datacenter1,31), RACK=Value(rack1,32), RELEASE_VERSION=Value(7.0.0-SNAPSHOT,33), NET_VERSION=Value(14,16), HOST_ID=Value(6d194555-f6eb-41d0-c000-000000000001,35), TOKENS=Value(<16 tokens>,36), RPC_READY=Value(true,44), INTERNAL_ADDRESS_AND_PORT=Value(127.0.0.1:37000,37), NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:39042,38), STATUS_WITH_PORT=Value(NORMAL,-8773856934580577011,39), SSTABLE_VERSIONS=Value(big-pa,18)}, isAlive = true}
DEBUG [main] 2026-09-08T09:19:52,512 Gossiper.java:2079 - Gossip looks settled. {localhost/127.0.0.1:38000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859185, version = 14, AppStateMap = {NET_VERSION=Value(14,5), SSTABLE_VERSIONS=Value(big-pa,7)}, isAlive = true, /127.0.0.1:37000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859012, version = 219, AppStateMap = {LOAD=Value(74541.0,172), SCHEMA=Value(00000000-0000-0000-0000-000000000002,30), DC=Value(datacenter1,31), RACK=Value(rack1,32), RELEASE_VERSION=Value(7.0.0-SNAPSHOT,33), NET_VERSION=Value(14,16), HOST_ID=Value(6d194555-f6eb-41d0-c000-000000000001,35), TOKENS=Value(<16 tokens>,36), RPC_READY=Value(true,44), INTERNAL_ADDRESS_AND_PORT=Value(127.0.0.1:37000,37), NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:39042,38), STATUS_WITH_PORT=Value(NORMAL,-8773856934580577011,39), SSTABLE_VERSIONS=Value(big-pa,18)}, isAlive = true}
DEBUG [main] 2026-09-08T09:19:53,513 Gossiper.java:2079 - Gossip looks settled. {localhost/127.0.0.1:38000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859185, version = 15, AppStateMap = {NET_VERSION=Value(14,5), SSTABLE_VERSIONS=Value(big-pa,7)}, isAlive = true, /127.0.0.1:37000=EndpointState: HeartBeatState = HeartBeat: generation = 1788859012, version = 220, AppStateMap = {LOAD=Value(74541.0,172), SCHEMA=Value(00000000-0000-0000-0000-000000000002,30), DC=Value(datacenter1,31), RACK=Value(rack1,32), RELEASE_VERSION=Value(7.0.0-SNAPSHOT,33), NET_VERSION=Value(14,16), HOST_ID=Value(6d194555-f6eb-41d0-c000-000000000001,35), TOKENS=Value(<16 tokens>,36), RPC_READY=Value(true,44), INTERNAL_ADDRESS_AND_PORT=Value(127.0.0.1:37000,37), NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:39042,38), STATUS_WITH_PORT=Value(NORMAL,-8773856934580577011,39), SSTABLE_VERSIONS=Value(big-pa,18)}, isAlive = true}

Both call sites (2406 and 2079) show TOKENS=Value(<16 tokens>,X) instead of raw bytes across all 5 lines here, from two separate real nodes gossiping with each other.

AshenScribe · 2026-09-08T11:22:25Z

@Suhel0328 Aah nice. it even shows TOKENS=Value(<16 tokens>,0) for a node with STATUS_WITH_PORT=BOOT as expected.

AshenScribe · 2026-09-08T11:28:36Z

+1 from my side.

AshenScribe · 2026-09-08T11:29:42Z

+    {
+        IPartitioner partitioner = DatabaseDescriptor.getPartitioner();
+        return applicationState.entrySet().stream().map(entry -> {
+            if (entry.getKey() != ApplicationState.TOKENS)


@smiklosovic as ApplicationState.TOKENS is "deprecated since version CEP-21", is there an alternate way?

imho just leave it like this

smiklosovic · 2026-09-08T13:22:09Z

+            if (entry.getKey() != ApplicationState.TOKENS)
+                return entry.getKey() + "=" + entry.getValue();
+
+            final VersionedValue value = entry.getValue();


I think all of this below is not necessary, we can just say TOKENS=Value(<hidden>) and be done with it, no deserialisation or similar, that can go away, just follow https://issues.apache.org/jira/secure/attachment/12755869/10330.txt

Updated! Removed the deserialization logic entirely and formatted TOKENS as Value(<hidden>) when present, or not present when missing, aligning directly with CASSANDRA-10330.

CASSANDRA-21417: Fix non-printable characters in Gossiper log for TOKENS

9505791

AshenScribe reviewed Sep 8, 2026

View reviewed changes

Refactor app state formatting to package-private method with dedicate…

bc4a99f

…d unit tests

AshenScribe reviewed Sep 8, 2026

View reviewed changes

Make value final and hoist getPartitioner() out of the stream loop

c20874d

AshenScribe reviewed Sep 8, 2026

View reviewed changes

smiklosovic reviewed Sep 8, 2026

View reviewed changes

Hide raw TOKENS in EndpointState logs per CASSANDRA-10330

5241f13

Uh oh!

Conversation

Suhel0328 commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AshenScribe Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshenScribe commented Sep 8, 2026

Uh oh!

AshenScribe commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Suhel0328 commented Sep 8, 2026

Uh oh!

Suhel0328 commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshenScribe Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshenScribe commented Sep 8, 2026

Uh oh!

Suhel0328 commented Sep 8, 2026

Uh oh!

AshenScribe commented Sep 8, 2026

Uh oh!

AshenScribe commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Suhel0328 commented Sep 7, 2026 •

edited

Loading

AshenScribe Sep 8, 2026 •

edited

Loading

AshenScribe Sep 8, 2026 •

edited

Loading

smiklosovic Sep 8, 2026 •

edited

Loading