[C++][FlightRPC] Flight generates misaligned buffers

@lidavidm

Protobuf's wire format design + our zero-copy serializer/deserializer mean that buffers can end up misaligned. On some Arrow versions, this can cause segfaults in kernels assuming alignment (and generally violates expectations).

We should:

Possibly include buffer alignment in array validation
See if we can adjust the serializer to somehow pad things properly

See if we can do anything about this in the deserializer

Example:

import pyarrow as pa
import pyarrow.flight as flight

class TestServer(flight.FlightServerBase):
    def do_get(self, context, ticket):
        schema = pa.schema(
            [
                ("index", pa.int64()),
                ("int8", pa.float64()),
                ("int16", pa.float64()),
                ("int32", pa.float64()),
            ]
        )
        return flight.RecordBatchStream(pa.table([
            [0, 1, 2, 3],
            [0, 1, None, 3],
            [0, 1, 2, None],
            [0, None, 2, 3],
        ], schema=schema))


with TestServer() as server:
    client = flight.connect(f"grpc://localhost:{server.port}")
    table = client.do_get(flight.Ticket(b"")).read_all()
    for col in table:
        print(col.type)
        for chunk in col.chunks:
            for buf in chunk.buffers():
                if not buf: continue
                print("buffer is 8-byte aligned?", buf.address % 8)
            chunk.cast(pa.float32())

On Arrow 8


int64
buffer is 8-byte aligned? 1
double
buffer is 8-byte aligned? 1
buffer is 8-byte aligned? 1
double
buffer is 8-byte aligned? 1
buffer is 8-byte aligned? 1
double
buffer is 8-byte aligned? 1
buffer is 8-byte aligned? 1

On Arrow 7


int64
buffer is 8-byte aligned? 4
double
buffer is 8-byte aligned? 4
buffer is 8-byte aligned? 4
fish: Job 1, 'python ../test.py' terminated by signal SIGSEGV (Address boundary error)

Reporter: David Li / @lidavidm

_{Note: This issue was originally created as ARROW-16958. Please see the migration documentation for further details.}

@pitrou

Antoine Pitrou / @pitrou:
Ouch, our kernels should not assume alignment. What kind of instructions are generated that fail on misaligned data?

@lidavidm

David Li / @lidavidm:
I didn't look too closely since it was fixed in Arrow 8.0.0, I just haven't gotten to look yet whether it was an intentional or unintentional fix

@lidavidm

David Li / @lidavidm:
FWIW in Arrow 7 it's this sequence


   │0x7ffff56f06b7 <_ZN5arrow7compute8internal12_GLOBAL__N_112DoStaticCastIfdEEvPKvlllPv+119>       shr    $0x2,%r9                                                                   │
   │0x7ffff56f06bb <_ZN5arrow7compute8internal12_GLOBAL__N_112DoStaticCastIfdEEvPKvlllPv+123>       nopl   0x0(%rax,%rax,1)                                                           │
  >│0x7ffff56f06c0 <_ZN5arrow7compute8internal12_GLOBAL__N_112DoStaticCastIfdEEvPKvlllPv+128>       cvtpd2ps (%rsi,%rax,2),%xmm0                                                      │
   │0x7ffff56f06c5 <_ZN5arrow7compute8internal12_GLOBAL__N_112DoStaticCastIfdEEvPKvlllPv+133>       cvtpd2ps 0x10(%rsi,%rax,2),%xmm1                                                  │
   │0x7ffff56f06cb <_ZN5arrow7compute8internal12_GLOBAL__N_112DoStaticCastIfdEEvPKvlllPv+139>       add    $0x1,%rcx

so an SSE instruction. Backtrace is


#0  0x00007ffff56f06c0 in void arrow::compute::internal::(anonymous namespace)::DoStaticCast<float, double>(void const*, long, long, long, void*) ()
   from /home/lidavidm/miniconda3/envs/theseus_env/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.700
#1  0x00007ffff56fc17f in arrow::compute::internal::CastNumberToNumberUnsafe(arrow::Type::type, arrow::Type::type, arrow::Datum const&, arrow::Datum*) ()
   from /home/lidavidm/miniconda3/envs/theseus_env/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.700
#2  0x00007ffff5709c73 in arrow::compute::internal::CastFloatingToFloating(arrow::compute::KernelContext*, arrow::compute::ExecBatch const&, arrow::Datum*) ()
   from /home/lidavidm/miniconda3/envs/theseus_env/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.700

@lidavidm

David Li / @lidavidm:
Hmm, I wonder if I misdiagnosed…I thought it would be alignment, but GDB says the code dereferenced a null pointer.

Yifei Yang:
Hey, I guess I also got the similar issue. What I did is to use Flight to transfer an Arrow table and then feed it into Arrow's aggregate exec node to do hash-aggregate. It will crash at arrow::util::CheckAlignment(). However using the original table works well. For the transferred table, if I first serialize into bytes then recreate an arrow table using the bytes, it also works well, which I guess is because the newly created table from bytes is aligned. I tested on both 6.0.0 and 8.0.0.

@pitrou

Antoine Pitrou / @pitrou:
[~yyfyyf123123] Can you open a JIRA for the crash at arrow::util::CheckAlignment() ? I don't think it should crash.

@pitrou

Yifei Yang:
@pitrou Ticket (https://issues.apache.org/jira/browse/ARROW-17783) here, I'm still trying to reproduce it stably in a minimal test.

@pitrou

Yifei Yang:
@pitrou I'm now able to reproduce the issue stably, and I attached the source into that ticket.

@westonpace

Weston Pace / @westonpace:
I can look into CheckAlignment but I still think it would be nice to fix this in flight. Otherwise data received from flight that is then passed over the C data interface could fail if the receiving library doesn't support unaligned buffers. @marsupialtail just mentioned to me that they ran into this kind of situation trying to pass a buffer to polars (I believe arrow-rs expects buffers of T to be aligned to T).

@lidavidm

David Li / @lidavidm:
Unless I come up with a great idea, that's tantamount to forcing a copy on all data coming through flight - which may not be awful given that gRPC often forces a copy on us (right now), but still isn't ideal

@lidavidm

David Li / @lidavidm:
I've sort of wanted to explore this anyways in any case - can we match up the chunks that gRPC gives us to the buffers in the IPC payload, and avoid concatenating? I suppose that optimization would bring little benefit if we ended up having to align all of them anyways, though. And as mentioned, it might be possible to insert padding data to align the buffer on the wire, but then you can't count on the client being polite like that.

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

A number of users are running into this, perhaps in lieu of a proper solution we might be able to provide a workaround? Perhaps a kernel that properly aligns input buffers? Perhaps such a kernel already exists?

Whilst we could patch around this in arrow-rs FFI it undermines the zero-copy expectation of FFI, and of course doesn't help with all the other arrow implementations that require alignment.

@tustvold

@tustvold

We recently added such a utility #14758 and also recently started using it in the Acero source node since Acero relies on alignment.

Although the solution in #14758 is not zero-copy if the buffers are unaligned.

Thanks all; if a general fix becomes available that would be ideal, otherwise looks like it'll probably require an extra copy to get it working.

@alexander-beedie

@alexander-beedie Arrow only recommends alignment for internal processes, it requires it for IPC.

Implementations are recommended to allocate memory on aligned addresses (multiple of 8- or 64-bytes) and pad (overallocate) to a length that is a multiple of 8 or 64 bytes. When serializing Arrow data for interprocess communication, these alignment and padding requirements are enforced.

If Flight is handing out misaligned buffers to IPC it is non-compliant and must be fixed.

Arrow only recommends alignment for internal processes, it requires it for IPC.

That was also my understanding. Some clarification of the docs, the spec, or a fix would be great, otherwise all downstream consumers expecting aligned data are going to discover (and have to solve) the issue individually.

It was never the intention that flight data on the receiving side must be aligned. The intention some of us always had was if you need something aligned and it isn't you pay the copy then, not before. It'd be great for individual flight clients to have an option to ensure aligned with clear messaging about this possibly incurring a copy but forcing flight to always be aligned because some people rely on alignment just punishes those that don't. If presented as an option, this allows the individual client implementations to choose to optimize away copies now, later or never.

(As a note the documentation cited above says nothing about the receiving side afaict).

Wrt to unknown tag padding. We could simply add one more dummy field (I don't believe there is any requirement for tags to be in order in messages) or simply restate a field with the first one being arbitrary padding. If I recall correctly, protobuf spec is that the last tag value for a specific tag within a message wins and previous ones should be ignored. (comes from streaming where something can be overridden by being expressed a second time)

I still intend to look at the options here when I get a chance. I need to do some refactoring in C++ first to be able to pass down options to the deserializer.

Also, see #34485 where we may want to evaluate a new framing format for Flight to support this and also things like messages that don't fit within a single gRPC/Protobuf message, and #37900 where we may want to evaluate if we can get more control over memory usage from Flight/gRPC.

Stumbled upon this bug when inserting thousands of objects with lists of 60-ish strings inside into a delta lake table. Is there any idea for how to fix this?

@choucavalier

Hi @choucavalier, I don't think we have anything at the API level yet that would serve as a workaround but, if you're willing to do the work, you may be able to adapt the internal helper in #14758 to your use case.

@EnricoMi

Can we get another review on #44279? @EnricoMi has been working on this for a while but it seems none of the maintainers are willing to take a look (I would merge it but I don't want to add a new IPC option by fiat)

That would provide the option everyone has been asking for

GH-32276: [C++][FlightRPC] Add option to align RecordBatch buffers gi…

Issue resolved by pull request 44279
#44279

apacheGH-32276: [C++][FlightRPC] Add option to align RecordBatch buff…

Navigation Menu

Repository navigation

[C++][FlightRPC] Flight generates misaligned buffers #32276

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[C++][FlightRPC] Flight generates misaligned buffers #32276

Description

Activity

asfimport commented on Jul 1, 2022

asfimport commented on Jul 1, 2022

asfimport commented on Jul 1, 2022

asfimport commented on Jul 1, 2022

asfimport commented on Sep 20, 2022

asfimport commented on Sep 20, 2022

asfimport commented on Sep 20, 2022

asfimport commented on Sep 20, 2022

asfimport commented on Sep 23, 2022

asfimport commented on Sep 23, 2022

asfimport commented on Sep 23, 2022

asfimport commented on Dec 22, 2022

tustvold commented on Apr 19, 2023

westonpace commented on Apr 20, 2023

westonpace commented on Apr 20, 2023

7 remaining items

alexander-beedie commented on Aug 2, 2023

orlp commented on Aug 30, 2023

alexander-beedie commented on Sep 2, 2023

jacques-n commented on Sep 28, 2023

lidavidm commented on Sep 28, 2023

choucavalier commented on Mar 12, 2025

amoeba commented on Mar 17, 2025

lidavidm commented on Mar 27, 2025

lidavidm commented on Mar 27, 2025

lidavidm commented on May 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions