Skip to content

SparseEncodingDecoder allocates an (n+1) offsets table for a zero-patch VarBin column #340

Description

@dfa1

Problem

In SparseEncodingDecoder's VarBin path, the zero-patch case allocates a full n+1 offsets table to describe a column of n empty strings (reader/decode/SparseEncodingDecoder.java:270-274):

MemorySegment outOffsets = ctx.arena().allocate((n + 1) * 4L, 4);
if (numPatches == 0) {
    MemorySegment outBytes = ctx.arena().allocate(1);
    Array result = new VarBinOffsetArray(ctx.dtype(), n, outBytes, outOffsets, PType.I32);
    return withSparseValidity(ctx, result, fillValid, null, idxData, 0, n, offset);
}

Every entry in that table is zero — the array is n copies of the empty string (or, when fillValid is false, n nulls). A sparse column with no patches in the scanned range is the common case for a genuinely sparse column, and it costs (n + 1) * 4 bytes to say "nothing here".

Fix

VarBinConstantArray — added in #329 for exactly this shape — represents it in O(1):

if (numPatches == 0) {
    Array result = new VarBinConstantArray(ctx.dtype(), n, new byte[0]);
    return withSparseValidity(ctx, result, fillValid, null, idxData, 0, n, offset);
}

withSparseValidity is unchanged and still applies the fill validity, so the all-null case behaves as before. The outOffsets allocation then moves inside the numPatches > 0 branch where it is actually used.

Worth confirming the empty-byte[] case is exercised by VarBinConstantArrayTestgetBytes clones and getString decodes, both of which are trivially fine for a zero-length array, but the zero-length constant is a corner the #329 tests may not have covered.

Context

Found in a sweep for remaining eager materializations after #329 / 7e0d6e7 — this is the same encoding family the #329 carrier was built for, in a spot that didn't get updated. One of the smaller items in that sweep; see the sibling issues for the vortex.runend string expansion, vortex.sequence, and the vortex.dict primitive path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions