All DataFrame Elementwise methods uncorrectly work with NULL values

Describe the bug
PrimitiveColumn ElementwiseEquals, ElementwizeNotEquals and other Elementwise method ignore validity information

For example. this unit test fails as it ElementWizeEquals return true for the first item ( as 0 == null):

[Fact]
public void TestElementWiseEquals()
{
    PrimitiveDataFrameColumn<int> intColumn1 = new PrimitiveDataFrameColumn<int>("Int1", new int?[] { 0, 2, 3 });
    PrimitiveDataFrameColumn<int> intColumn2 = new PrimitiveDataFrameColumn<int>("Int2", new int?[] { null, 2, 3 });

    var results = intColumn1.ElementwiseEquals(intColumn2);

    Assert.False(results[0]);
}

Triage: seems a bug, we should fix it, any help would be appreciated.

I'm happy to take a look.

The underlying reason for this behaviour appears to be that the comparison operation acts on ReadOnlyBuffer<T>, without first checking whether the buffer contains a valid data point.

Simple fix seems to be something like:

b = CompareIsValid(leftBitmap, rightBitmap)
v = arithmetic.CompareValues(left, right)
result = Combine(b, v)

Would need a careful eye on performance, of course. I'll investigate a bit further.

Add 857 contribution: dotnet/machinelearning#6820

Navigation Menu

Repository navigation

All DataFrame Elementwise methods uncorrectly work with NULL values #6820

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

All DataFrame Elementwise methods uncorrectly work with NULL values #6820

Description

Activity

rosebyte commented on Sep 29, 2026

adestr commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions