Use object_store:BufWriter instead of put_multipart

[tustvold]

Is your feature request related to a problem or challenge?

Currently in many places we use put_multipart for streaming writes. When writing files smaller than 10MiB this is wasteful, as it performs 3 requests when 1 would suffice.

Describe the solution you'd like

object_store 0.9.1 added https://docs.rs/object_store/latest/object_store/buffered/struct.BufWriter.html which can automatically switch between using Put and PutMultipart based on the amount of data that has been written

Describe alternatives you've considered

We could implement our own adaptive logic in the write path within DF

Additional context

A future version of object_store is likely to significantly change put_multipart, and using BufWriter will limit the impact of this - apache/arrow-rs#5500

[tustvold]

FYI @devinjdangelo @alamb

[alamb]

Related ticket for cleaning up parallel parquet writer is #9493

[yyy1000]

Can I take this to get familiar with datasource related code?
Currently my plan is:

1. Create BufWriter using given object_storeand path
2. Remove put_multipart method and call AsyncArrowWriter::try_new using the new BufWriter

That's my initial plan after investigation, hope to hear your feedback. :)

[alamb]

@yyy1000 good luck -- this ticket will require some API exploration / potential changes so it will likely be a bit trickey.

I think your suggested plan sounds good.

It will be interesting if you can also capture any experience / improvements that would make using BufWriter easier from the context of DataFusion

[tustvold]

Your plan sounds good and should be relatively straightforward