Repository navigation
Blosc silently fails with 2GB buffer #80
Description
>>> import numcodecs
>>> import numpy as np
>>> a = np.ones(1024**3, dtype=np.int8)
>>> a.nbytes
1073741824
>>> codec = numcodecs.Blosc()
>>> x = codec.encode(a)
>>> len(x)
4505616
>>> a = np.ones(2 * 1024**3, dtype=np.int8)
>>> x = codec.encode(a)
Input buffer size cannot exceed 2147483631 bytes
>>> codec.decode(x)
# Freezes interpreter
It looks like Blosc is raising this error and it's not being caught here because csize is declared as size_t, and so cannot be negative.
Any thoughts @alimanfoo? Should be an easy fix, but it's not entirely clear how to test this. Do you know of any other ways we can provoke errors in blosc which don't involve mallocing 2GB?
Activity
alimanfoo commentedon May 9, 2018
You could try corrupting a compressed buffer. I.e., create some compressed bytes via encode(), then remove some leading or trailing bytes then try to decode().