object_store returns an error when other compression features are enabled in reqwest #7142

Describe the bug

apache/arrow-rs-object-store#32 describes the problems that occur when the optional gzip feature is enabled in reqwest. The fix in #6843 addressed this by calling the reqwest::ClientBuilder.no_gzip() method.

Reqwest offers other features that automatically enable compression on a supported server too: in addition to gzip, there's also deflate, brotli, and zstd. When any of these features are enabled, reqwest will include the Accept-Encoding header with supported compression formats, and supporting HTTP servers will reply with Content-Encoding indicating the accepted compression format (if any). When this happens, the Content-Length header will be set to the compressed length of the body. That is, if it's set at all-- many servers will end up omitting the header when compression is enabled.

To Reproduce

1. Clone the example repo at this commit: kylewlacy/object_store_compression_repro@256c8c0
2. Run cargo test within the repo

The test fails with the following error:

thread 'tests::repro' panicked at src/lib.rs:32:14:
called `Result::unwrap()` on an `Err` value: Generic { store: "HTTP", source: Header { source: MissingContentLength } }

(Example repo based on the example from apache/arrow-rs-object-store#32, thanks @phillipleblanc for making an easy-to-use example for the previous bug!)

Expected behavior

The test should pass. It passes if the zstd feature is disabled in the reqwest crate.

Additional context

This example repo uses a local Axum server with gzip/deflate/brotli/zstd compression enabled, and with the zstd feature enabled on the reqwest crate. Therefore, the default reqwest client includes Accept-Encoding: zstd in the request, and the server responds with Content-Encoding: zstd. Because compression is enabled, reqwest does not (and cannot) return the decompressed content length of the response.