Skip to content

perf(c/driver/postgresql): speed up decimal128 COPY encoding - #4498

Merged
lidavidm merged 2 commits into
apache:mainfrom
BOKJUNSOO:optimize-postgresql-decimal128-copy
Jul 13, 2026
Merged

perf(c/driver/postgresql): speed up decimal128 COPY encoding#4498
lidavidm merged 2 commits into
apache:mainfrom
BOKJUNSOO:optimize-postgresql-decimal128-copy

Conversation

@BOKJUNSOO

@BOKJUNSOO BOKJUNSOO commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

This adds a fast path for PostgreSQL binary COPY encoding of Arrow
decimal128 values.

The existing path converts every Arrow decimal value to a decimal string, splits
that string into integer/fractional parts, then parses those parts back into
PostgreSQL's base-10000 numeric representation. For decimal128, we can avoid
that round trip: read the unscaled 128-bit integer directly, align it to
PostgreSQL's 4-decimal-digit groups, and write the numeric digits.

The fast path is guarded by __SIZEOF_INT128__. Platforms without native
unsigned __int128 support keep using the existing implementation.

decimal256 is unchanged.

Local benchmark

Environment:

  • macOS arm64
  • PostgreSQL 15 container
  • Release build
  • Python adbc_driver_postgresql.dbapi.adbc_ingest
  • table schema: v numeric
  • input: decimal128(38, 6), values Decimal(i) / Decimal(100)

Before, PyPI wheel adbc-driver-postgresql==1.11.0:

100k rows: ~2.166s, ~46k rows/s

Before, local unpatched Release build from the ADBC source tree:

100k rows: ~0.831s, ~120k rows/s

After this patch:

100k rows: ~0.058s, ~1.73M rows/s
200k rows: ~0.097s, ~2.05M rows/s
1M rows:   ~0.486s, ~2.06M rows/s

Testing

PKG_CONFIG_PATH=/opt/homebrew/lib/postgresql@14/pkgconfig \
  cmake -S c -B build-local \
  -DADBC_DRIVER_POSTGRESQL=ON \
  -DADBC_BUILD_SHARED=ON \
  -DADBC_BUILD_STATIC=OFF \
  -DADBC_BUILD_TESTS=ON \
  -DADBC_BUILD_BENCHMARKS=OFF \
  -DADBC_WITH_VENDORED_FMT=ON \
  -DADBC_WITH_VENDORED_NANOARROW=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build-local --target adbc-driver-postgresql-copy-test -j 8

ADBC_POSTGRESQL_TEST_URI='postgresql://postgres:password@127.0.0.1:9900/postgres?sslmode=disable' \
  build-local/driver/postgresql/adbc-driver-postgresql-copy-test

Result:

[==========] 73 tests from 5 test suites ran.
[  PASSED  ] 73 tests.

I also checked PostgreSQL roundtrips for scale 2/6/8/18, negative values,
small fractional values, and 38-digit decimal128 values.

@BOKJUNSOO
BOKJUNSOO requested a review from lidavidm as a code owner July 10, 2026 07:33
@lidavidm

Copy link
Copy Markdown
Member

CC @Mandukhai-Alimaa for review

@BOKJUNSOO
BOKJUNSOO force-pushed the optimize-postgresql-decimal128-copy branch from 4a44c14 to 38d7938 Compare July 10, 2026 09:08

@Mandukhai-Alimaa Mandukhai-Alimaa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears we need to use __extension__ __int128 or similar to make GCC happy: https://stackoverflow.com/a/54649367

@BOKJUNSOO

Copy link
Copy Markdown
Contributor Author

Thanks. Fixed in d929cfc by defining __extension__ using UInt128 = unsigned __int128; under the existing __SIZEOF_INT128__ guard and using the alias throughout the fast path. I verified the actual writer header with GCC 14 using -std=gnu++17 -Wpedantic -Werror; the PostgreSQL COPY test suite also passes (73 tests).

@lidavidm

Copy link
Copy Markdown
Member

@Mandukhai-Alimaa want to take a look as well?

@lidavidm

Copy link
Copy Markdown
Member

Oh wait, you already did.

@lidavidm
lidavidm merged commit 5403e57 into apache:main Jul 13, 2026
86 of 91 checks passed
@lidavidm lidavidm added this to the ADBC Libraries 24 milestone Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants