Skip to content

bench(manifest): add criterion benchmark for Manifest::parse_avro - #3468

Draft
kevinjqliu wants to merge 4 commits into
apache:mainfrom
kevinjqliu:kevinjqliu-manifest-parse-benchmark
Draft

bench(manifest): add criterion benchmark for Manifest::parse_avro#3468
kevinjqliu wants to merge 4 commits into
apache:mainfrom
kevinjqliu:kevinjqliu-manifest-parse-benchmark

Conversation

@kevinjqliu

@kevinjqliu kevinjqliu commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Adds the repo's first benchmark, crates/iceberg/benches/manifest_parse.rs (criterion), for Manifest::parse_avro on V2 data manifests with full stats for every column and an identity partition on a string column. It has two groups:

  • manifest_parse_avro: 1k to 100k entries with 12 columns.
  • manifest_parse_avro_by_columns: 10,000 entries with 9 to 610 columns, which makes the files about 1 MB to 100 MB. It reports MB/s. This group takes about 3 minutes and peaks at about 6 GB of memory.

The bench writes each manifest with ManifestWriter, then rewrites it the way Iceberg Java writes manifests by default, which is how most manifests are written: without first_row_id, content_offset, and content_size_in_bytes, with deflate, and with 64 KB blocks. Its writer schema differs from iceberg-rust's reader schema, which is the case #3354 sped up. No fixtures, Python, or JVM are needed. Run with cargo bench -p iceberg --bench manifest_parse.

Comparison with other Iceberg readers

A cross-language comparison can't run as a criterion bench, so the harness is below rather than in the repo. Iceberg Java 1.12.0 writes the manifests with default settings. Five readers parse each file from memory:

Each reader warms up, then parses each file at least 15 times (5 times in the file size sweep) and for at least 3 s. All readers run single threaded on an Apple M5 Max (JDK 17, Go 1.25, Python 3.13).

Time for Java, Rust after #3354, Go, Python, and Rust before #3354 to parse manifests by number of entries and by file size

By number of entries, Rust after #3354 parses 5.9x to 6.1x faster than Rust before it. It is 1.1x to 1.2x faster than Go and 1.7x to 2.6x faster than Python, and it takes 1.11x to 1.16x as long as Java.

By file size, Rust after #3354 parses 5.3x to 6.0x faster than Rust before it, and runs within 8% of Go and within 20% of Java. Python is the fastest reader from 5 MB on: at 98 MB it takes 0.82 s, against 1.97 s for Rust after #3354. Its Cython decoder returns the four count maps as a LazyDict that builds the dict on first access, and it keeps bounds as bytes, while iceberg-rust builds every HashMap and converts every bound to a Datum while parsing. Building the dicts adds 0.56 s at 98 MB, so Python is still faster.

By number of entries, median time per parse with the interquartile range:

Java (Iceberg Java 1.12.0) Rust after #3354 Go (iceberg-go 0.7.0) Python (PyIceberg 0.12.0, Cython) Rust before #3354
1,000 3.8 ms (3.8–3.9) 4.4 ms (4.3–4.4) 5.3 ms (5.2–5.4) 7.4 ms (7.2–7.5) 26.4 ms (26.2–26.9)
2,000 7.6 ms (7.6–7.7) 8.7 ms (8.6–9.0) 10.0 ms (9.9–10.2) 14.9 ms (14.7–15.2) 52.4 ms (52.1–52.6)
5,000 19.0 ms (18.9–19.1) 22.0 ms (21.9–22.4) 24.4 ms (24.1–24.6) 40.6 ms (39.1–46.7) 130.7 ms (130.2–131.4)
10,000 38.4 ms (38.1–40.5) 44.2 ms (44.0–44.4) 48.4 ms (47.4–49.6) 86.3 ms (82.9–88.8) 261.4 ms (260.9–263.1)
20,000 76.3 ms (75.9–76.7) 88.2 ms (87.6–88.6) 97.4 ms (95.7–98.4) 179.1 ms (169.9–185.6) 525.0 ms (521.9–527.1)
50,000 190.9 ms (190.0–197.7) 219.7 ms (218.7–224.2) 241.5 ms (239.8–242.8) 517.6 ms (502.4–533.4) 1.31 s (1.31–1.31)
100,000 390.6 ms (386.4–391.0) 435.8 ms (434.2–439.4) 477.7 ms (474.3–479.8) 1.13 s (1.11–1.17) 2.63 s (2.62–2.65)

By file size (10,000 entries), median time per parse with the interquartile range:

Java (Iceberg Java 1.12.0) Rust after #3354 Go (iceberg-go 0.7.0) Python (PyIceberg 0.12.0, Cython) Rust before #3354
1.0 MB 31.1 ms (30.9–31.3) 36.2 ms (36.0–36.6) 39.1 ms (38.6–39.5) 79.8 ms (77.7–83.2) 214.4 ms (213.9–215.4)
1.9 MB 50.8 ms (50.6–51.1) 58.6 ms (58.4–59.0) 62.3 ms (61.3–63.8) 88.6 ms (85.3–91.4) 343.5 ms (342.6–345.1)
4.9 MB 119.1 ms (118.5–122.8) 130.4 ms (130.2–131.1) 140.0 ms (138.5–140.9) 122.1 ms (118.2–126.1) 787.5 ms (787.1–790.1)
10.2 MB 222.5 ms (214.3–227.7) 237.0 ms (235.7–239.2) 251.1 ms (249.9–254.8) 160.9 ms (156.9–167.6) 1.43 s (1.43–1.44)
21.8 MB 511.0 ms (497.9–531.1) 430.9 ms (430.9–433.8) 454.1 ms (453.5–456.5) 234.4 ms (226.1–242.2) 2.48 s (2.47–2.49)
51.3 MB 1.11 s (1.02–1.21) 989.9 ms (985.8–1029.4) 1.03 s (1.03–1.04) 439.1 ms (437.8–444.4) 5.53 s (5.52–5.58)
98.1 MB 1.91 s (1.89–2.08) 1.97 s (1.93–1.97) 1.91 s (1.90–1.91) 824.0 ms (809.6–829.7) 10.51 s (10.47–10.63)

The criterion bench gives 51.7 ms at 10k entries and 2.23 s at 610 columns. Its stand-in uses apache-avro's default deflate level, and its schema attributes differ, so its files aren't the same size as Java's.

Harness

run.sh and run_size.sh write the manifests with Iceberg Java, time each reader, and plot the results. rust and rust-old are the same Rust project; their iceberg path dependency points at this branch and at ecba0b537, respectively. The Python environment has pyiceberg==0.12.0, pyarrow, and matplotlib. Each harness prints one reader,entries,ms row per timed run.

run.sh:

#!/bin/bash
# Times five readers on V2 data manifests that Iceberg Java writes with default settings.
set -euo pipefail
cd "$(dirname "$0")"
SIZES="1000 2000 5000 10000 20000 50000 100000"
CP="java/target/classes:$(ls java/lib/*.jar | tr '\n' ':')"
mkdir -p data
java -cp "$CP" bench.ManifestBench generate data $SIZES > /dev/null
FILES=$(for n in $SIZES; do echo data/manifest_$n.avro; done)
echo "reader,entries,ms" > samples.csv
java -Xms4g -Xmx4g -cp "$CP" bench.ManifestBench bench $FILES >> samples.csv
.venv/bin/python bench_pyiceberg.py $FILES >> samples.csv
go/manifest-bench $FILES >> samples.csv
rust-old/target/release/manifest-parse-bench iceberg-rust-before $FILES >> samples.csv
rust/target/release/manifest-parse-bench iceberg-rust-after $FILES >> samples.csv
.venv/bin/python plot.py samples.csv size_samples.csv manifest_read.png

run_size.sh:

#!/bin/bash
# Times five readers on 10,000-entry manifests from about 1 MB to 100 MB, made larger by adding columns.
set -euo pipefail
cd "$(dirname "$0")"
COLUMNS="9 17 44 82 145 330 610"
CP="java/target/classes:$(ls java/lib/*.jar | tr '\n' ':')"
mkdir -p data/wide
java -cp "$CP" bench.ManifestBench generate-wide data/wide 10000 $COLUMNS > /dev/null
export MIN_WARMUPS=1 MIN_RUNS=5
echo "reader,columns,bytes,ms" > size_samples.csv
for c in $COLUMNS; do
  f=data/wide/wide_$c.avro
  bytes=$(stat -f %z "$f")
  {
    java -Xms4g -Xmx4g -cp "$CP" bench.ManifestBench bench "$f"
    .venv/bin/python bench_pyiceberg.py "$f"
    go/manifest-bench "$f"
    rust-old/target/release/manifest-parse-bench iceberg-rust-before "$f"
    rust/target/release/manifest-parse-bench iceberg-rust-after "$f"
  } | awk -F, -v c="$c" -v b="$bytes" '{print $1 "," c "," b "," $3}' | tee -a size_samples.csv | awk -F, '{print $1, $2, $4}' | sort -u -k1,1 | head -5
done

ManifestBench.java (iceberg-core 1.12.0 and slf4j-nop):

package bench;

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.iceberg.DataFile;
import org.apache.iceberg.DataFiles;
import org.apache.iceberg.FileFormat;
import org.apache.iceberg.ManifestContent;
import org.apache.iceberg.ManifestFile;
import org.apache.iceberg.ManifestFiles;
import org.apache.iceberg.ManifestReader;
import org.apache.iceberg.ManifestWriter;
import org.apache.iceberg.Metrics;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.inmemory.InMemoryFileIO;
import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;

public class ManifestBench {
  // Column types repeat in this order; column 2 is the partition source.
  static final Type.PrimitiveType[] TYPES = {
    Types.LongType.get(),
    Types.StringType.get(),
    Types.IntegerType.get(),
    Types.LongType.get(),
    Types.FloatType.get(),
    Types.DoubleType.get(),
    Types.StringType.get(),
    Types.BooleanType.get(),
    Types.DateType.get(),
    Types.TimestampType.withoutZone(),
    Types.StringType.get(),
    Types.LongType.get()
  };
  // Serialized bound width per position in TYPES; strings use 16 ASCII bytes.
  static final int[] WIDTHS = {8, 0, 4, 8, 4, 8, 0, 1, 4, 8, 0, 8};

  static Schema schema(int columns) {
    List<Types.NestedField> fields = new ArrayList<>();
    for (int f = 1; f <= columns; f++) {
      fields.add(Types.NestedField.optional(f, f == 2 ? "category" : "c" + f, TYPES[(f - 1) % 12]));
    }
    return new Schema(fields);
  }

  static ByteBuffer bound(int f, long v) {
    int width = WIDTHS[(f - 1) % 12];
    if (width == 0) {
      return ByteBuffer.wrap(String.format("%016d", v).getBytes(StandardCharsets.US_ASCII));
    }
    long value = width == 8 ? v : v % (1L << (8 * width - 1));
    ByteBuffer buf = ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(value);
    return ByteBuffer.wrap(Arrays.copyOf(buf.array(), width));
  }

  static DataFile dataFile(PartitionSpec spec, int columns, int i) {
    Map<Integer, Long> columnSizes = new HashMap<>();
    Map<Integer, Long> valueCounts = new HashMap<>();
    Map<Integer, Long> nullCounts = new HashMap<>();
    Map<Integer, Long> nanCounts = new HashMap<>();
    Map<Integer, ByteBuffer> lower = new HashMap<>();
    Map<Integer, ByteBuffer> upper = new HashMap<>();
    for (int f = 1; f <= columns; f++) {
      columnSizes.put(f, 1000L + f + i);
      valueCounts.put(f, 10_000L + i);
      nullCounts.put(f, (long) (i % 7));
      if ((f - 1) % 12 == 4 || (f - 1) % 12 == 5) {
        nanCounts.put(f, 0L);
      }
      lower.put(f, bound(f, (long) i * 13 + f));
      upper.put(f, bound(f, (long) i * 17 + f + 1000));
    }
    Metrics metrics =
        new Metrics(10_000L + i, columnSizes, valueCounts, nullCounts, nanCounts, lower, upper);
    return DataFiles.builder(spec)
        .withPath(
            String.format(
                "s3://bucket/warehouse/db/table/data/category=c%d/%08d-0-6f1e1a2b-3c4d-5e6f-7a8b-9c0d1e2f3a4b.parquet",
                i % 10, i))
        .withFormat(FileFormat.PARQUET)
        .withPartitionPath("category=c" + (i % 10))
        .withRecordCount(10_000L + i)
        .withFileSizeInBytes(1_000_000L + i)
        .withMetrics(metrics)
        .withSplitOffsets(List.of(4L))
        .withSortOrder(SortOrder.unsorted())
        .build();
  }

  static void write(Path path, int columns, int entries) throws IOException {
    PartitionSpec spec =
        PartitionSpec.builderFor(schema(columns)).withSpecId(0).identity("category").build();
    Files.deleteIfExists(path);
    ManifestWriter<DataFile> writer =
        ManifestFiles.write(2, spec, org.apache.iceberg.Files.localOutput(path.toFile()), 1L);
    try (writer) {
      for (int i = 0; i < entries; i++) {
        writer.add(dataFile(spec, columns, i), 1L);
      }
    }
    System.out.printf("%s,%d,%d,%d%n", path, columns, entries, Files.size(path));
  }

  static ManifestFile manifestFile(String path, long length) {
    return new ManifestFile() {
      public String path() { return path; }
      public long length() { return length; }
      public int partitionSpecId() { return 0; }
      public ManifestContent content() { return ManifestContent.DATA; }
      public long sequenceNumber() { return 1; }
      public long minSequenceNumber() { return 1; }
      public Long snapshotId() { return 1L; }
      public Integer addedFilesCount() { return null; }
      public Long addedRowsCount() { return null; }
      public Integer existingFilesCount() { return null; }
      public Long existingRowsCount() { return null; }
      public Integer deletedFilesCount() { return null; }
      public Long deletedRowsCount() { return null; }
      public List<PartitionFieldSummary> partitions() { return null; }
      public ManifestFile copy() { return this; }
    };
  }

  static int read(ManifestFile manifest, InMemoryFileIO io) throws IOException {
    List<DataFile> files = new ArrayList<>();
    try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, io, null)) {
      for (DataFile file : reader) {
        files.add(file.copy());
      }
    }
    return files.size();
  }

  static void bench(List<Path> paths) throws IOException {
    int minWarmups = Integer.parseInt(System.getenv().getOrDefault("MIN_WARMUPS", "5"));
    int minRuns = Integer.parseInt(System.getenv().getOrDefault("MIN_RUNS", "15"));
    InMemoryFileIO io = new InMemoryFileIO();
    for (Path p : paths) {
      byte[] bytes = Files.readAllBytes(p);
      String location = "memory://" + p.toAbsolutePath();
      io.addFile(location, bytes);
      ManifestFile manifest = manifestFile(location, bytes.length);
      int entries = read(manifest, io);
      long start = System.nanoTime();
      int warmups = 0;
      while (warmups < minWarmups || System.nanoTime() - start < 3_000_000_000L) {
        read(manifest, io);
        warmups++;
      }
      start = System.nanoTime();
      for (int i = 0; i < 100 && (i < minRuns || System.nanoTime() - start < 3_000_000_000L); i++) {
        long t = System.nanoTime();
        read(manifest, io);
        System.out.printf("java,%d,%.3f%n", entries, (System.nanoTime() - t) / 1e6);
      }
    }
  }

  public static void main(String[] args) throws IOException {
    switch (args[0]) {
      case "generate" -> {
        // generate <dir> <columns> <entries...>
        int columns = Integer.parseInt(args[2]);
        for (int k = 3; k < args.length; k++) {
          int n = Integer.parseInt(args[k]);
          write(Path.of(args[1]).resolve("manifest_" + n + ".avro"), columns, n);
        }
      }
      case "generate-wide" -> {
        // generate-wide <dir> <entries> <columns...>
        int n = Integer.parseInt(args[2]);
        for (int k = 3; k < args.length; k++) {
          int columns = Integer.parseInt(args[k]);
          write(Path.of(args[1]).resolve("wide_" + columns + ".avro"), columns, n);
        }
      }
      default -> bench(Arrays.stream(args).skip(1).map(Path::of).toList());
    }
  }
}

go/main.go (github.com/apache/iceberg-go v0.7.0):

// Times iceberg-go's manifest reader on in-memory manifests.
package main

import (
	"bytes"
	"fmt"
	"os"
	"strconv"
	"time"

	"github.com/apache/iceberg-go"
)

func envInt(k string, d int) int {
	if v, ok := os.LookupEnv(k); ok {
		n, err := strconv.Atoi(v)
		if err != nil {
			panic(err)
		}
		return n
	}
	return d
}

func main() {
	minWarmups, minRuns := envInt("MIN_WARMUPS", 3), envInt("MIN_RUNS", 15)
	for _, path := range os.Args[1:] {
		data, err := os.ReadFile(path)
		if err != nil {
			panic(err)
		}
		manifest := iceberg.NewManifestFile(2, path, int64(len(data)), 0, 1).
			SequenceNum(1, 1).
			Content(iceberg.ManifestContentData).
			Build()
		read := func() int {
			entries, err := iceberg.ReadManifest(manifest, bytes.NewReader(data), true)
			if err != nil {
				panic(err)
			}
			return len(entries)
		}
		entries := read()
		start := time.Now()
		for i := 0; i < minWarmups || time.Since(start) < time.Second; i++ {
			read()
		}
		start = time.Now()
		for i := 0; i < 100 && (i < minRuns || time.Since(start) < 3*time.Second); i++ {
			t := time.Now()
			read()
			fmt.Printf("iceberg-go,%d,%.3f\n", entries, float64(time.Since(t).Nanoseconds())/1e6)
		}
	}
}

bench_pyiceberg.py:

"""Time PyIceberg's manifest reader (Cython Avro decoder) on in-memory manifests."""

import io
import os
import sys
import time

from pyiceberg.avro.decoder import new_decoder
from pyiceberg.io import FileIO, InputFile, InputStream, OutputFile
from pyiceberg.manifest import ManifestContent, ManifestFile


class MemoryInputFile(InputFile):
    def __init__(self, location: str, data: bytes):
        super().__init__(location)
        self._data = data

    def __len__(self) -> int:
        return len(self._data)

    def exists(self) -> bool:
        return True

    def open(self, seekable: bool = True) -> InputStream:
        return io.BytesIO(self._data)


class MemoryFileIO(FileIO):
    def __init__(self) -> None:
        super().__init__()
        self.files: dict[str, bytes] = {}

    def new_input(self, location: str) -> InputFile:
        return MemoryInputFile(location, self.files[location])

    def new_output(self, location: str) -> OutputFile:
        raise NotImplementedError

    def delete(self, location) -> None:
        raise NotImplementedError


def main(paths: list[str]) -> None:
    min_warmups, min_runs = int(os.environ.get("MIN_WARMUPS", 3)), int(os.environ.get("MIN_RUNS", 15))
    assert type(new_decoder(b"")).__name__ == "CythonBinaryDecoder"
    file_io = MemoryFileIO()
    for path in paths:
        with open(path, "rb") as f:
            data = f.read()
        file_io.files[path] = data
        manifest = ManifestFile.from_args(
            manifest_path=path,
            manifest_length=len(data),
            partition_spec_id=0,
            content=ManifestContent.DATA,
            sequence_number=1,
            min_sequence_number=1,
            added_snapshot_id=1,
        )
        read = lambda: manifest.fetch_manifest_entry(file_io, discard_deleted=True)  # noqa: E731
        entries = len(read())
        start = time.perf_counter()
        warmups = 0
        while warmups < min_warmups or time.perf_counter() - start < 1:
            read()
            warmups += 1
        start = time.perf_counter()
        i = 0
        while i < 100 and (i < min_runs or time.perf_counter() - start < 3):
            t = time.perf_counter()
            read()
            print(f"pyiceberg,{entries},{(time.perf_counter() - t) * 1e3:.3f}", flush=True)
            i += 1


if __name__ == "__main__":
    main(sys.argv[1:])

Rust Cargo.toml and src/main.rs:

[package]
name = "manifest-parse-bench"
version = "0.0.0"
edition = "2024"
publish = false

[dependencies]
iceberg = { path = "../iceberg-rust/crates/iceberg" }

[workspace]
use std::hint::black_box;
use std::time::{Duration, Instant};

use iceberg::spec::Manifest;

fn parse(bytes: &[u8]) -> usize {
    Manifest::parse_avro(black_box(bytes)).unwrap().entries().len()
}

fn main() {
    let env = |k: &str, d: usize| std::env::var(k).map_or(d, |v| v.parse().unwrap());
    let (min_warmups, min_runs) = (env("MIN_WARMUPS", 3), env("MIN_RUNS", 15));
    let mut args = std::env::args().skip(1);
    let label = args.next().unwrap();
    for path in args {
        let bytes = std::fs::read(&path).unwrap();
        let entries = parse(&bytes);
        let start = Instant::now();
        let mut i = 0;
        while i < min_warmups || start.elapsed() < Duration::from_secs(1) {
            parse(&bytes);
            i += 1;
        }
        let start = Instant::now();
        let mut i = 0;
        while i < 100 && (i < min_runs || start.elapsed() < Duration::from_secs(3)) {
            let t = Instant::now();
            parse(&bytes);
            println!("{label},{entries},{:.3}", t.elapsed().as_secs_f64() * 1e3);
            i += 1;
        }
    }
}

plot.py:

"""Plot the time each manifest reader takes to parse a whole manifest.

Usage: plot.py samples.csv size_samples.csv out.png
"""

import csv
import statistics
import sys
from collections import defaultdict

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.ticker import FixedLocator, FuncFormatter

# Okabe-Ito colors; both Rust versions share a color.
READERS = {
    "iceberg-rust-before": ("Rust before #3354", "#D55E00", "--", "x"),
    "pyiceberg": ("Python (PyIceberg 0.12.0, Cython)", "#0072B2", "-", "s"),
    "iceberg-go": ("Go (iceberg-go 0.7.0)", "#009E73", "-", "D"),
    "iceberg-rust-after": ("Rust after #3354", "#D55E00", "-", "o"),
    "java": ("Java (Iceberg Java 1.12.0)", "#E69F00", "-", "^"),
}


def load(path, x_of):
    samples = defaultdict(list)
    for row in csv.DictReader(open(path)):
        samples[(row["reader"], x_of(row))].append(float(row["ms"]) / 1000)
    return samples


def draw(ax, samples, gap):
    xs = sorted({x for _, x in samples})
    labels = []
    for reader, (label, color, ls, marker) in READERS.items():
        q = [statistics.quantiles(samples[(reader, x)], n=4) for x in xs]
        ax.plot(xs, [q2 for _, q2, _ in q], ls, color=color, marker=marker, ms=5, lw=2)
        ax.fill_between(xs, [q1 for q1, _, _ in q], [q3 for _, _, q3 in q], color=color, alpha=0.15, lw=0)
        labels.append([q[-1][1], q[-1][1], f"{label.split(' (')[0]}  {q[-1][1]:.2f} s", color])
    # Keep end labels at least `gap` of the tallest value apart, joined to their lines.
    min_gap = gap * max(y for y, *_ in labels)
    labels.sort()
    for below, above in zip(labels, labels[1:]):
        above[0] = max(above[0], below[0] + min_gap)
    for y, y_line, text, color in labels:
        ax.annotate(text, (xs[-1], y_line), xytext=(xs[-1] * 1.06, y), va="center", color=color,
                    fontsize=9.5, fontweight="bold",
                    arrowprops=dict(arrowstyle="-", color=color, lw=0.8, shrinkA=0, shrinkB=3, relpos=(0, 0.5)))
    ax.set_xlim(0, xs[-1] * 1.45)
    ax.set_ylim(bottom=0)
    ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g} s"))
    ax.set_ylabel("time to parse the whole manifest")
    ax.grid(True, alpha=0.3)
    ax.spines[["top", "right"]].set_visible(False)


by_entries = load(sys.argv[1], lambda r: int(r["entries"]))
by_size = load(sys.argv[2], lambda r: int(r["bytes"]) / 1e6)

fig, (left, right) = plt.subplots(1, 2, figsize=(16, 6.5))
draw(left, by_entries, 0.06)
left.xaxis.set_major_locator(FixedLocator([0, 10_000, 20_000, 50_000, 100_000]))
left.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v / 1000:g}k" if v else "0"))
left.set_xlabel("entries in the manifest (12 columns, 0.14 MB per 1k entries)")
left.set_title("By number of entries", loc="left", fontsize=11)

draw(right, by_size, 0.06)
right.xaxis.set_major_locator(FixedLocator([0, 10, 20, 50, 100]))
right.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g} MB" if v else "0"))
right.set_xlabel("manifest file size (10,000 entries; 9 to 610 columns)")
right.set_title("By file size", loc="left", fontsize=11)

fig.text(0.01, 0.975, "Time to parse an Iceberg manifest written by Iceberg Java with default settings",
         fontsize=13, va="top")
fig.text(0.01, 0.93, "V2 data manifest, deflate, 64 KB blocks, full stats for every column, identity partition. "
         "Parsed from memory, single thread, after warmup. Apple M5 Max. Lower is better.\n"
         "Line: median parse time (at least 15 parses per point by entries, 5 by file size); band: interquartile range.",
         fontsize=8.5, color="0.35", va="top")
fig.tight_layout(rect=(0, 0, 1, 0.87))
fig.savefig(sys.argv[3], dpi=140)

order = ["java", "iceberg-rust-after", "iceberg-go", "pyiceberg", "iceberg-rust-before"]
for title, samples, fmt in [("By number of entries", by_entries, lambda x: f"{x:,}"),
                            ("By file size (10,000 entries)", by_size, lambda x: f"{x:.1f} MB")]:
    print(f"{title}, median time per parse with the interquartile range:\n")
    print("| | " + " | ".join(READERS[r][0] for r in order) + " |")
    print("|--:|" + "--:|" * len(order))
    for x in sorted({x for _, x in samples}):
        cells = []
        for r in order:
            q1, q2, q3 = (v * 1000 for v in statistics.quantiles(samples[(r, x)], n=4))
            cells.append(f"{q2:.1f} ms ({q1:.1f}–{q3:.1f})" if q2 < 1000 else f"{q2 / 1000:.2f} s ({q1 / 1000:.2f}–{q3 / 1000:.2f})")
        print(f"| {fmt(x)} | " + " | ".join(cells) + " |")
    print()

Are these changes tested?

This PR only adds a benchmark. I ran:

  • cargo bench -p iceberg --bench manifest_parse: passed.
  • cargo clippy -p iceberg --all-targets --all-features -- -D warnings: passed.
  • cargo fmt --all -- --check, taplo fmt --check, and cargo machete: all passed.
  • cargo deny check licenses bans: passed on the first commit; the later commits don't change dependencies.

AI Disclosure

Written with GitHub Copilot (Claude Opus 5.5). Please check:

  • Whether the bench's rewrite is a fair stand-in for Java-written manifests. It matches Java's field set, codec, and block size, but not its deflate level or schema attributes.
  • Whether the readers do comparable work. Java's timing includes copy() of each DataFile. PyIceberg builds its count maps lazily and keeps bounds as bytes. iceberg-rust builds everything eagerly.
  • Whether the 610-column group is worth its runtime and memory in the in-repo bench.
  • Using criterion as a workspace dev-dependency with default-features = false (no plotters or rayon). It adds 11 crates to Cargo.lock and moves colored and winapi-util to the newer windows-sys already in the lock.

kevinjqliu and others added 4 commits October 15, 2026 15:26
@kevinjqliu
bench(manifest): add criterion benchmark for Manifest::parse_avro
746edd4 
Measures parsing V2 data manifests with 1k, 10k, and 50k entries, each
with stats for 12 columns and an identity partition on a string column.
The manifests are written in the bench with ManifestWriter. A second
case drops the data_file fields PyIceberg doesn't write, so the writer
schema differs from the reader schema.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu
bench(manifest): write Java-like manifests and cover up to 100k entries
4c7e7ee 
Rewrite the second case the way Iceberg Java writes V2 manifests, with
deflate and 64 KB blocks, and measure 1k to 100k entries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu
bench(manifest): bench only Java-default manifests
c426079 
Parse manifests laid out the way Iceberg Java writes them by default:
without the data_file fields it doesn't write, deflate, and 64 KB blocks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu
bench(manifest): parse manifests from 1 MB to 100 MB
d65772d 
Add a group that parses 10,000-entry manifests made larger by adding
columns with stats, from about 1 MB to 100 MB, and reports MB/s.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kevinjqliu