Add int8 / byte-vector similarity support by r-devulap · Pull Request #709 · datastax/jvector

r-devulap · 2026-08-11T08:26:34Z

INT8 (Byte-Vector) Support for JVector

Overview

This branch adds full INT8 (byte-vector) support to JVector — from the math primitives all the way through graph building, on-disk storage, SIMD acceleration, and end-to-end tests.

Commits

1. `fd131a7` — Add random-access int8 vector value support

Introduces a new VectorValues super-interface that captures shared random-access behavior, then refactors RandomAccessVectorValues (float) to extend it, and adds RandomAccessByteVectorValues and its list-backed implementation ListRandomAccessByteVectorValues as the byte-vector counterpart. Also updates BuildScoreProvider to use the generalized thread-local supplier.

2. `c55574c` — Add byte-similarity methods to VectorUtilSupport / VectorUtil / DefaultVectorUtilSupport

Adds dotProduct, squaredL2, and cosine for byte[] / ByteSequence to the vector utility stack.

3. `9ba737a` — Add ByteVectorSimilarityFunction enum

Mirrors the float VectorSimilarityFunction enum; wraps the three byte similarity metrics with a compare() API and score normalization.

4. `5652b46` — Add BuildScoreProvider.byteVectorScoreProvider and searchProviderFor(ByteSequence)

Adds byteVectorScoreProvider() factory and searchProviderFor(ByteSequence) so the graph builder can score byte-vector candidates.

5. `389c391` — Wire byteVectorScoreProvider into GraphIndexBuilder

GraphIndexBuilder now accepts RandomAccessByteVectorValues and routes to the byte score provider, enabling full graph construction over int8 data.

6. `c6dfea0` — Vectorize ByteSequence similarity metrics with SIMD

Panama SIMD and AVX-512 native kernels for int8 dot-product, L2, and cosine; adds native benchmarks and C++ unit tests.

7. `3d4dada` — Add InlineByteVectors feature for native int8 on-disk storage

New FeatureId and InlineByteVectors class for serializing byte vectors into the graph index file format, plus OnDiskGraphIndex reader support.

8. `974f508` — Add INT8 end-to-end tutorial (Int8Example)

End-to-end walkthrough showing how to build, serialize, and query a byte-vector graph index; registered in TutorialRunner.

9. `b0ad5cd` — Add int8 test coverage for byte-vector similarity, RABVV, BSP, SIMD, graph build, and disk round-trip

Unit tests for ByteVectorSimilarityFunction, BuildScoreProvider, TestVectorizationProvider (SIMD), TestVectorGraph (build), TestOnDiskGraphIndex (round-trip), and ListRandomAccessByteVectorValues.

github-actions · 2026-08-11T08:26:44Z

r-devulap · 2026-08-12T04:31:32Z

To do:

Update all the docs/README to reflect these changes
Add release notes
Add BenchYAML results

jshook

This PR needs to be updated.

merge conflicts need to be addressed
the base commit this branch depends on should probably determine the target of the merge, rather than main, since there are direct dependencies.

r-devulap · 2026-08-17T14:24:29Z

This PR needs to be updated.

merge conflicts need to be addressed

Fixed, it is now rebased with the latest changes in main.

the base commit this branch depends on should probably determine the target of the merge, rather than main, since there are direct dependencies.

This is no longer a problem since it includes the commits in #708

r-devulap · 2026-08-18T07:52:29Z

Marking this as draft until I resolve the question of leveraging NVQ for INT8.

Extract shared random-access behavior into the generic VectorValues interface while preserving RandomAccessVectorValues for float vectors. Add RandomAccessByteVectorValues and its list-backed implementation to support native ByteSequence vectors without float32 conversion. Update BuildScoreProvider to use the generalized thread-local supplier.

…ltVectorUtilSupport Add three fundamental signed int8 vector similarity operations to the vectorization support layer so they participate in the same provider dispatch as float similarity. Bytes are treated as signed int8 (Java byte range -128..127). VectorUtilSupport — three new abstract methods: float dotProduct(ByteSequence<?> a, ByteSequence<?> b) float squareDistance(ByteSequence<?> a, ByteSequence<?> b) float cosine(ByteSequence<?> a, ByteSequence<?> b) VectorUtil — three new public static delegates: dotProduct(ByteSequence<?>, ByteSequence<?>) -> impl.dotProduct squareL2Distance(ByteSequence<?>, ByteSequence<?>) -> impl.squareDistance cosine(ByteSequence<?>, ByteSequence<?>) -> impl.cosine DefaultVectorUtilSupport — scalar loop implementations: dotProduct: accumulate (int)a.get(i) * (int)b.get(i), return as float. squareDistance: accumulate (diff * diff) for each signed byte difference. cosine: dot / sqrt(normA * normB) using per-element float promotion. PanamaVectorUtilSupport — scalar stub overrides identical to Default, so jvector-twenty compiles without requiring a SIMD implementation now. SIMD optimisation of byte similarity is a future concern.

New enum in jvector-base/.../vector/ parallel to VectorSimilarityFunction but operating on ByteSequence<?>, delegating to the VectorUtil byte methods from Sub-Task 2. Three variants with return values normalised to [0,1] matching VectorSimilarityFunction conventions (higher = more similar): EUCLIDEAN: 1 / (1 + squaredL2 / (n * 255^2)) Normalises by the maximum possible squared distance between two signed int8 vectors (255^2 per dimension) so the result stays in (0,1] regardless of dimension. DOT_PRODUCT: (1 + dot / (n * 127^2)) / 2 Normalises by the maximum possible dot product magnitude (127^2 per dimension) before applying the (1+x)/2 mapping so the result stays in [0,1] regardless of dimension or whether vectors are unit-norm. For already unit-norm int8 vectors (e.g. Cohere, OpenAI reduced-precision) prefer COSINE. COSINE: (1 + cosine(v1, v2)) / 2 Cosine is inherently bounded to [-1,1] so no extra normalisation is needed.

…ByteSequence) Add a new static factory byteVectorScoreProvider(RandomAccessByteVectorValues, ByteVectorSimilarityFunction) that performs exact byte×byte scoring with no float32 round-trip. The returned BuildScoreProvider implements isExact()=true, approximateCentroid(), searchProviderFor(ByteSequence), searchProviderFor(int), diversityProviderFor(int), and diversityScoreFunctionFor(int), using two independent threadLocalSupplier() handles for thread-safe concurrent builds. Also adds a default searchProviderFor(ByteSequence<?>) to the interface that throws UnsupportedOperationException, so existing float-based providers are unaffected.

Add builder(RandomAccessByteVectorValues, ByteVectorSimilarityFunction, int M) and builder(..., List<Integer> maxDegrees) factory overloads so callers can build a graph over int8 byte vectors without touching the builder core. Generalize build(RandomAccessVectorValues) to build(VectorValues<?>) and switch the parallel addGraphNode loop to use scoreProvider.searchProviderFor(node) directly, removing the float-only getVector path that was incompatible with byte-vector score providers. Add addGraphNode(int node, ByteSequence<?> vector) as a public single-node insertion entry point for byte vectors. Update TestVectorGraph to add explicit casts (RandomAccessVectorValues, VectorSimilarityFunction) on the null arguments so the compiler resolves the correct overload now that the new byte-vector builder() overloads exist.

Replace scalar loop implementations of dotProduct, squareDistance, and cosine for ByteSequence with Panama Vector API implementations in PanamaVectorUtilSupport, and native AVX-512/AVX2 kernels wired through NativeVectorUtilSupport -> NativeSimdOps JNI bindings. Java (Panama) path: - Widen signed bytes to int32 via B2I conversion, accumulate products in IntVector lanes, then reduce. - Dispatch on PREFERRED_BIT_SIZE: 512-bit: load 16 bytes (SPECIES_128) -> IntVector.SPECIES_512 256-bit: load 8 bytes (SPECIES_64) -> IntVector.SPECIES_256 128-bit: scalar fallback - cosine variants accumulate dot/norm products in long after reduction to avoid int32 overflow on large vectors. Native path (C++): - New kernels in jvector_simd_kernels.cpp and jvector_avx3_dl_kernels.cpp: dot_product_i8, euclidean_i8, cosine_i8 using Highway SIMD (AVX-512 / AVX2 dispatch). - Registered in jvector_simd_kernel_list.h and exported via jvector_simd.cpp. - Microbenchmarks added in bench_similarity_i8.cpp. - C++ unit tests added in test_similarity_i8.cpp using a prime-length (107-element) vector to exercise tail handling. Java tests: - TestVectorizationProvider.testSimilarityMetricsByte cross-checks SIMD results against scalar DefaultVectorUtilSupport baseline.

Introduces InlineByteVectors, a new Feature that stores signed int8 vectors inline in an OnDiskGraphIndex at 1 byte per component — 4x more compact than the float32 InlineVectors representation. Changes: - InlineByteVectors.java: new Feature implementation backed by VectorTypeSupport.writeByteSequence / readByteSequence - FeatureId: add INLINE_BYTE_VECTORS as ordinal 5 (backward-compatible) - AbstractGraphIndexWriter.Builder: accept INLINE_BYTE_VECTORS as the canonical source for the vector dimension in the file header - OnDiskGraphIndex.View: add getByteVector(int) to read a stored int8 vector from disk, and byteVectorRerankerFor(ByteSequence, ByteVectorSimilarityFunction) to wire byte-by-byte disk scoring directly into the search path

Demonstrates the full int8 pipeline using the siftsmall dataset: - Read siftsmall_base.fvecs and convert float32 vectors to signed int8 - Build a graph index with byte-by-byte scoring (no float32 round-trip) - Save the graph to disk using InlineByteVectors (1 byte/component) - Load the index from disk - Search with random int8 query vectors, scoring directly from disk byte vectors Also registers the tutorial under the 'int8' key in TutorialRunner.

…graph build, and disk round-trip

MarkWolters · 2026-09-08T17:29:38Z

+
+        int result = acc.reduceLanes(VectorOperators.ADD);
+        for (int i = limit; i < length; i++) {
+            result += a.get(aOff + i) * b.get(bOff + i);


ArrayByteSequence.slice(offset, length) (ArrayByteSequence.java:75-80) returns an ArraySliceByteSequence wrapping the original array with that offset stored. So when you call .slice(2, 3) on an ArrayByteSequence and then call .get(0), .get(1), .get(2), you get data[2], data[3], data[4] with the offset already resolved. This code calls a.get(aOff + i) where aOff = a.offset(). For a slice, that's a.get(2 + i), which internally becomes data.get(offset + (2 + i)) = data.get(2 + 2 + i) = data.get(4 + i), the offset is being applied twice.

AFAIK, this should always operate on a ArrayByteSequence and we are not calling .slice on it, right? Or am I missing something?

r-devulap requested review from MarkWolters, ashkrisk, jshook and tlwillke as code owners August 11, 2026 08:26

r-devulap linked an issue Aug 11, 2026 that may be closed by this pull request

Native INT8 (byte vector) HNSW build + search API #665

Open

r-devulap mentioned this pull request Aug 11, 2026

Native INT8 (byte vector) HNSW build + search API #665

Open

r-devulap force-pushed the int8-support branch from 7aec240 to 9ee1486 Compare August 11, 2026 08:29

jshook requested changes Aug 14, 2026

View reviewed changes

r-devulap force-pushed the int8-support branch from 9ee1486 to 4c789ac Compare August 17, 2026 14:22

r-devulap marked this pull request as draft August 18, 2026 07:51

r-devulap force-pushed the int8-support branch 2 times, most recently from 2515d75 to 09c4e3c Compare September 2, 2026 10:09

r-devulap force-pushed the int8-support branch from 09c4e3c to ecc9edd Compare September 7, 2026 04:12

r-devulap marked this pull request as ready for review September 7, 2026 04:17

r-devulap marked this pull request as draft September 7, 2026 04:21

r-devulap added 9 commits September 7, 2026 07:52

Add int8 test coverage for byte-vector similarity, RABVV, BSP, SIMD, …

b0ad5cd

…graph build, and disk round-trip

r-devulap force-pushed the int8-support branch from ecc9edd to b0ad5cd Compare September 8, 2026 07:59

r-devulap marked this pull request as ready for review September 8, 2026 08:04

MarkWolters reviewed Sep 8, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add int8 / byte-vector similarity support - #709

Add int8 / byte-vector similarity support#709
r-devulap wants to merge 9 commits into
mainfrom
int8-support

r-devulap commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026 •

edited by r-devulap

Loading

Uh oh!

r-devulap commented Aug 12, 2026 •

edited

Loading

Uh oh!

jshook left a comment

Uh oh!

r-devulap commented Aug 17, 2026

Uh oh!

r-devulap commented Aug 18, 2026 •

edited

Loading

Uh oh!

MarkWolters Sep 8, 2026

Uh oh!

r-devulap Sep 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

r-devulap commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

INT8 (Byte-Vector) Support for JVector

Overview

Commits

1. fd131a7 — Add random-access int8 vector value support

2. c55574c — Add byte-similarity methods to VectorUtilSupport / VectorUtil / DefaultVectorUtilSupport

3. 9ba737a — Add ByteVectorSimilarityFunction enum

4. 5652b46 — Add BuildScoreProvider.byteVectorScoreProvider and searchProviderFor(ByteSequence)

5. 389c391 — Wire byteVectorScoreProvider into GraphIndexBuilder

6. c6dfea0 — Vectorize ByteSequence similarity metrics with SIMD

7. 3d4dada — Add InlineByteVectors feature for native int8 on-disk storage

8. 974f508 — Add INT8 end-to-end tutorial (Int8Example)

9. b0ad5cd — Add int8 test coverage for byte-vector similarity, RABVV, BSP, SIMD, graph build, and disk round-trip

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited by r-devulap Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r-devulap commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jshook left a comment

Choose a reason for hiding this comment

Uh oh!

r-devulap commented Aug 17, 2026

Uh oh!

r-devulap commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarkWolters Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

r-devulap Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r-devulap commented Aug 11, 2026 •

edited

Loading

1. `fd131a7` — Add random-access int8 vector value support

2. `c55574c` — Add byte-similarity methods to VectorUtilSupport / VectorUtil / DefaultVectorUtilSupport

3. `9ba737a` — Add ByteVectorSimilarityFunction enum

4. `5652b46` — Add BuildScoreProvider.byteVectorScoreProvider and searchProviderFor(ByteSequence)

5. `389c391` — Wire byteVectorScoreProvider into GraphIndexBuilder

6. `c6dfea0` — Vectorize ByteSequence similarity metrics with SIMD

7. `3d4dada` — Add InlineByteVectors feature for native int8 on-disk storage

8. `974f508` — Add INT8 end-to-end tutorial (Int8Example)

9. `b0ad5cd` — Add int8 test coverage for byte-vector similarity, RABVV, BSP, SIMD, graph build, and disk round-trip

github-actions Bot commented Aug 11, 2026 •

edited by r-devulap

Loading

r-devulap commented Aug 12, 2026 •

edited

Loading

r-devulap commented Aug 18, 2026 •

edited

Loading

r-devulap Sep 9, 2026 •

edited

Loading