Skip to content

Releases: drivendataorg/cloudpathlib

v0.25.0

Choose a tag to compare

@pjbull pjbull released this 22 Aug 18:39
a55e549
  • Security fix (GHSA-r4f8-3xc4-c8vw): local path traversal via .. in cloud object keys.
    Cloud object keys are opaque strings and some backends (e.g. Google Cloud Storage) accept ..
    segments. cloudpathlib mapped keys onto local paths with plain path arithmetic and no containment
    check, so a key such as ../../evil could escape the local cache directory (on any
    read/write/open/os.fspath of a CloudPath) or a download destination
    (download_to/copytree/copy/move to a local path), writing attacker-controlled bytes
    outside the intended directory. cloudpathlib now confirms each computed local target stays within
    its base directory and raises CloudPathLocalPathTraversalError otherwise. This guards the cache
    mapping (CloudPath._local and its S3/HTTP overrides), both branches of CloudPath.download_to
    and CloudPath.copytree (including joins that can escape via \ or drive letters in keys on
    Windows), and the cloudpathlib.local mock's storage-side mapping. Reported by mohammad adnan
    (cystack.ps redteam).
  • Added a lazy keyword argument to CloudPath.walk. By default (lazy=False) the existing fast behavior is preserved: the whole subtree is fetched up front with a single recursive listing. Passing lazy=True lists each directory on demand so that, when top_down=True, callers can prune subdirectories by modifying dirnames in-place (à la os.walk / Path.walk) to skip fetching the contents of those subtrees entirely — dramatically reducing API calls for large, sparsely-traversed trees. (Issue #518, PR #574)
  • Added presigned URL support to the mock clients in cloudpathlib.local, so as_url(presign=True) on a LocalS3Path, LocalGSPath, or LocalAzureBlobPath returns a deterministic URL with expires and signature query parameters instead of raising NotImplementedError. This lets code that generates presigned URLs be exercised in tests that use the mock classes. (PR #572)
  • Fixed S3 copy and move operations to forward copy-specific extra args such as CopySourceSSECustomerKey, and added addressing_style="virtual" support for S3Client (Issues #500, #527, PR #576)
  • Changed S3Client._get_metadata to read object metadata with HeadObject instead of GetObject, so stat, etag, and size no longer open the object body. Also fixes a KeyError on ContentLength against S3-compatible gateways that drop Content-Length from GetObject responses. (Issue #564, PR #565)
  • Fixed KeyError on ETag against S3-compatible gateways that omit ETag from HeadObject responses. (Issue #582, PR #583)
  • Fixed mypy 2.x type errors in Client and CloudPath that caused CI lint failures (Issue #563, PR #566)
  • Added AGENTS.md with repository-specific guidance for coding agents covering contributor
    workflow, compatibility expectations, test rig and mock usage, live backend validation, and PR
    hygiene. (PR #573)

v0.24.0

Choose a tag to compare

@pjbull pjbull released this 30 Apr 00:52
1d65750
  • Added support for S3 Multi-Region Access Point (MRAP) URLs in S3Path (Issue #556, PR #557)
  • Added support for Pydantic serialization (Issue #537, PR #538)

v0.23.0

Choose a tag to compare

@pjbull pjbull released this 07 Oct 22:46
28f1d94
  • Added support for Python 3.14 (Issue #529, PR #530)
    • Changed CloudPath.copy to have the first parameter named target instead of destination and added new follow_symlinks and preserve_metadata kwargs. Breaking change for users that relied on the first parameter being named destination instead of target.
    • Added CloudPath.copy_into to copy a file or directory into another file or directory.
    • Added CloudPath.move to move a file or directory to another location.
    • Added CloudPath.move_into to move a file or directory into another file or directory.
    • Added CloudPathInfo and CloudPath.info to get information about a file or directory.
    • Added additional no-op kwargs to mkdir, touch, glob, rglob, stat to match pathlib.

v0.22.0

Choose a tag to compare

@pjbull pjbull released this 30 Aug 05:18
  • Fixed issue with GS credentials, using default auth enables a wider set of authentication methods in GS (Issue #390, PR #514, thanks @ljyanesm)
  • Added support for http(s) urls with HttpClient, HttpPath, HttpsClient, and HttpsPath. (Issue #455, PR #468)
  • Added experimental support for patching the builtins open, os, os.path, and glob to work with CloudPath objects. It is off by default; see the new "Compatibility" section in the docs for more information. (Issue #128, PR #322)
  • Added support for CloudPath(*parts) to create a CloudPath object from a list of parts (to match pathlib.Path). This is a potentially breaking change for users that relied on the second arg being the client instead of making it an explicit kwarg. (PR #322)

v0.21.1

Choose a tag to compare

@jayqi jayqi released this 15 May 02:30
b086100
  • Fixed rmtree fail on Azure with no hns and more than 256 blobs to drop (Issue #509, PR #508, thanks @alikefia)

v0.21.0

Choose a tag to compare

@pjbull pjbull released this 04 Mar 01:12
  • Removed support for deprecated env var that had a typo (CLOUPATHLIB_FILE_CACHE_MODE; you should use CLOUDPATHLIB_FILE_CACHE_MODE).
  • Added support for timeout and retry kwargs for GSClient. (Issue #484, PR #485, thanks @Mchristos)
  • Fixed CloudPath(...) / other to correctly attempt to fall back on other's __rtruediv__ implementation, in order to support classes that explicitly support the / with a CloudPath instance. Previously, this would always raise a TypeError if other were not a str or PurePosixPath. (PR #479)
  • Added md5 property to GSPath, updated LocalGSPath to include md5 property, updated mock_gs.MockBlob to include md5_hash property. (Issue #482, PR #483)
  • Fixed an uncaught exception on Azure Gen2 storage accounts with HNS enabled when used with DefaultAzureCredential. (Issue #486, PR #487)
  • Removed support for Python 3.8, which was EOL in October 2024. (PR #502)

v0.20.0

Choose a tag to compare

@pjbull pjbull released this 18 Oct 20:35
5f5e054

v0.19.0

Choose a tag to compare

@pjbull pjbull released this 29 Aug 17:39
40e205d
  • Fixed an error that occurred when loading and dumping CloudPath objects using pickle multiple times. (Issue #450, PR #454, thanks to @kujenga)
  • Fixed typo in FileCacheMode where values were being filled by environment variable CLOUPATHLIB_FILE_CACHE_MODE instead of CLOUDPATHLIB_FILE_CACHE_MODE. (PR #424, thanks to @mynameisfiber)
  • Fixed CloudPath cleanup via CloudPath.__del__ when Client encounters an exception during initialization and does not create a file_cache_mode attribute. (Issue #372, thanks to @bryanwweber)
  • Removed support for Python 3.7 and pinned minimal boto3 version to Python 3.8+ versions. (PR #407)
  • Changed GSClient to use the native exists() method from the Google Cloud Storage SDK. (PR #420, thanks to @bachya)
  • Changed default clients to be lazily instantiated (Issue #428, PR #432)
  • Fixed download_to to check for the existence of the cloud file (Issue #430, PR #433)
  • Added env vars CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUD and CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD. (Issue #393, PR #437)
  • Fixed glob for cloudpathlib.local.LocalPath and subclass implementations to match behavior of cloud versions for parity in testing. (Issue #415, PR #436)
  • Changed how cloudpathlib.local.LocalClient and subclass implementations track the default local storage directory (used to simulate the cloud) used when no local storage directory is explicitly provided. (PR #436, PR #462)
    • Changed LocalClient so that client instances using the default storage access the default local storage directory through the get_default_storage_dir rather than having an explicit reference to the path set at instantiation. This means that calling get_default_storage_dir will reset the local storage for all clients using the default local storage, whether the client has already been instantiated or is instantiated after resetting. This fixes unintuitive behavior where reset_local_storage did not reset local storage when using the default client. (Issue #414)
    • Added a new local_storage_dir property to LocalClient. This will return the current local storage directory used by that client instance.
      by reference through the `get_default_ rather than with an explicit.
  • Refined the return type annotations for CloudPath.open() to match the behavior of pathlib.Path.open(). The method now returns specific types (TextIOWrapper, FileIO, BufferedRandom, BufferedWriter, BufferedReader, BinaryIO, IO[Any]) based on the provided mode, buffering, and encoding arguments. (Issue #465, PR #464)
  • Added Azure Data Lake Storage Gen2 support (Issue #161, PR #450), thanks to @M0dEx for PR #447 and PR #449

v0.18.1

Choose a tag to compare

@pjbull pjbull released this 26 Feb 22:57
29a147a
  • Fixed import error due to incompatible google-cloud-storage by not using transfer_manager if it is not available. (Issue #408, PR #410)

Includes all changes from v0.18.0.

Note: This is the last planned Python 3.7 compatible release version.

v0.18.0 (yanked)

Choose a tag to compare

@jayqi jayqi released this 26 Feb 02:35
6d13512
  • Implement sliced downloads in GSClient. (Issue #387, PR #389)
  • Implement as_url with presigned parameter for all backends. (Issue #235, PR #236)
  • Stream to and from Azure Blob Storage. (PR #403)
  • Implement file: URI scheme support for AnyPath. (Issue #401, PR #404)

Note: This version was yanked due to incompatibility with google-cloud-storage <2.7.0 that causes an import error.