Problem
In SparseEncodingDecoder's VarBin path, the zero-patch case allocates a full n+1 offsets table to describe a column of n empty strings (reader/decode/SparseEncodingDecoder.java:270-274):
MemorySegment outOffsets = ctx.arena().allocate((n + 1) * 4L, 4);
if (numPatches == 0) {
MemorySegment outBytes = ctx.arena().allocate(1);
Array result = new VarBinOffsetArray(ctx.dtype(), n, outBytes, outOffsets, PType.I32);
return withSparseValidity(ctx, result, fillValid, null, idxData, 0, n, offset);
}
Every entry in that table is zero — the array is n copies of the empty string (or, when fillValid is false, n nulls). A sparse column with no patches in the scanned range is the common case for a genuinely sparse column, and it costs (n + 1) * 4 bytes to say "nothing here".
Fix
VarBinConstantArray — added in #329 for exactly this shape — represents it in O(1):
if (numPatches == 0) {
Array result = new VarBinConstantArray(ctx.dtype(), n, new byte[0]);
return withSparseValidity(ctx, result, fillValid, null, idxData, 0, n, offset);
}
withSparseValidity is unchanged and still applies the fill validity, so the all-null case behaves as before. The outOffsets allocation then moves inside the numPatches > 0 branch where it is actually used.
Worth confirming the empty-byte[] case is exercised by VarBinConstantArrayTest — getBytes clones and getString decodes, both of which are trivially fine for a zero-length array, but the zero-length constant is a corner the #329 tests may not have covered.
Context
Found in a sweep for remaining eager materializations after #329 / 7e0d6e7 — this is the same encoding family the #329 carrier was built for, in a spot that didn't get updated. One of the smaller items in that sweep; see the sibling issues for the vortex.runend string expansion, vortex.sequence, and the vortex.dict primitive path.
Problem
In
SparseEncodingDecoder's VarBin path, the zero-patch case allocates a fulln+1offsets table to describe a column ofnempty strings (reader/decode/SparseEncodingDecoder.java:270-274):Every entry in that table is zero — the array is
ncopies of the empty string (or, whenfillValidis false,nnulls). A sparse column with no patches in the scanned range is the common case for a genuinely sparse column, and it costs(n + 1) * 4bytes to say "nothing here".Fix
VarBinConstantArray— added in #329 for exactly this shape — represents it in O(1):withSparseValidityis unchanged and still applies the fill validity, so the all-null case behaves as before. TheoutOffsetsallocation then moves inside thenumPatches > 0branch where it is actually used.Worth confirming the empty-
byte[]case is exercised byVarBinConstantArrayTest—getBytesclones andgetStringdecodes, both of which are trivially fine for a zero-length array, but the zero-length constant is a corner the #329 tests may not have covered.Context
Found in a sweep for remaining eager materializations after #329 / 7e0d6e7 — this is the same encoding family the #329 carrier was built for, in a spot that didn't get updated. One of the smaller items in that sweep; see the sibling issues for the
vortex.runendstring expansion,vortex.sequence, and thevortex.dictprimitive path.