Skip to content

CASSANDRA-21094 Advise sequential access on descriptors owned by bulk SSTable scans and streaming - #5133

Open
cheeeee wants to merge 1 commit into
apache:trunkfrom
cheeeee:CASSANDRA-21094-trunk
Open

cheeeee wants to merge 1 commit into
apache:trunkfrom
cheeeee:CASSANDRA-21094-trunk

Conversation

@cheeeee

@cheeeee cheeeee commented Sep 9, 2026 •

Copy link
Copy Markdown

CASSANDRA-21094: compaction and streaming read SSTables sequentially but never tell the kernel, so
readahead starts at its default window and ramps up heuristically.

posix_fadvise(POSIX_FADV_SEQUENTIAL) doubles the readahead window of the open file description it is
called on. It therefore has to be issued on a descriptor that only the sequential reader uses; the
SSTable's shared data handle also serves point reads and must not be advised.

Changes:

  • SSTableReader.openDataReaderForBulkScan(mode): used by SSTableSimpleScanner and SSTableCursorReader
    (compaction, cleanup, scrub, offline tools). When the effective access mode is buffered (standard),
    it opens a private FileHandle for the scan, advises its descriptor as sequential, and closes it with
    the reader. The private handle does not share the SSTable's chunk cache or the writer's mmapped
    regions cache, so opening and closing it cannot invalidate cached chunks for other readers. Under
    mmap the file is not read through the descriptor's read()/readahead path and under direct I/O the
    page cache is bypassed, so in both cases the shared handle is reused unchanged and no advice is
    issued.
  • openDataReaderForScan() (user range reads via SSTableScanner) is unchanged: it keeps reusing the
    shared handle and is never advised.
  • Streaming: CassandraStreamWriter, CassandraCompressedStreamWriter and entire-SSTable ComponentContext
    each open their own channel per transfer and advise it, but only when a single section spans the
    whole file, since readahead past a section end would read bytes that are not sent.
  • NativeLibrary.trySetSequential(fd, path): posix_fadvise(fd, 0, 0, SEQUENTIAL) on Linux (the OS type
    Cassandra also uses as the fallback for unrecognised systems); a failure is logged at WARN with
    rate limiting and is otherwise ignored, as for trySkipCache. No-op on macOS and AIX.

Behaviour change: buffered bulk scans no longer read through the shared chunk cache. They neither
warm it nor evict its entries; compaction reads previously did both.

Testing:

  • SSTableReaderDataReaderTest (12): the private handle is opened only for bulk scans; closing it
    leaves the shared handle's cached chunks in place; a bulk scan of a freshly flushed SSTable under
    mmap works; direct mode on an uncompressed table falls back to the shared handle; user range reads
    are never advised; plus the existing reuse/close/leak cases.
  • NativeLibraryTest.testSetSequential (Byteman on the native call): a single advice with offset 0,
    length 0 and POSIX_FADV_SEQUENTIAL on the given descriptor.
  • Kernel-level check, Linux 6.19, ext4, 4 KiB pages: after a 1 MiB sequential read of a 64 MiB file
    the pages resident beyond the read position are 32 without advice and 96 with it (mincore); reading
    through a second, unadvised descriptor of the same file after advising the first shows 32, i.e. the
    advice does not leak between open file descriptions.
  • No throughput measurement was made; the ticket's expected gain on IOPS-constrained storage is not
    quantified here.

CASSANDRA-21094

@cheeeee
cheeeee force-pushed the CASSANDRA-21094-trunk branch from 13a50f6 to 4fe4da7 Compare September 10, 2026 01:25
Keep existing mmap regions and direct-I/O fallback semantics. Buffered scans
use independently owned uncached handles so closing a scan does not evict
foreground chunks. Advise descriptors actually used by buffered scans and
streaming reads; leave shared mmap and direct readers unadvised.

CASSANDRA-21094

Generated-by: Claude (Anthropic)
@cheeeee
cheeeee force-pushed the CASSANDRA-21094-trunk branch from 4fe4da7 to f728cc5 Compare September 11, 2026 22:50
@cheeeee cheeeee changed the title CASSANDRA-21094: Use POSIX_FADV_SEQUENTIAL for SSTable reads during compaction and streaming CASSANDRA-21094 Advise sequential access on descriptors owned by bulk SSTable scans and streaming Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant